vb.net: rounding error -- vb.net

Re: rounding error

rowe_newsgroups
2/5/2008 4:23:55 AM

vb.net:

On Feb 5, 4:59 am, "Steve Lloyd" <steve-remo...@-spam-
[quoted text, click to view]

It all has to do with the accuracy of doubles and there isn't too much
you can do about it. You might do some searching for the exact "whys",
this topic has been covered numerous times on most programming
newsgroups.

Thanks,

RE: rounding error

guy
2/5/2008 4:49:03 AM

Use a Decimal rather than a Double for your result variable

Dim result As Decimal= cdec(textbox.text) / 100

hth

Guy

[quoted text, click to view]

rounding error

Steve Lloyd
2/5/2008 9:59:47 AM

Hi,

Can anyone please explain this rounding error on a windows forms project.
I take an input and need to convert from a percentage to a decimal so i do:
dim result as double = cdbl(textbox.text) / 100

when 9.85 is entered into the text box "result" is calcualted as
0.09849999999999999.

For debugging i changed it to cdbl(textbox.text) / 1000 and also /10. both
of these gave the accurate answers, 0.00985 and 0.985 respectively.

Any ideas?

Thanks for any help

Steve.

Got something to say? Click here to post a reply to this thread.

Don't see what you're looking for? Try a search.

View other messages in the vb.net group.