This project is read-only.


Javascript rounding problem


Hmm...the problem is actually in the EcmaScript library - the ScriptConvert.ToString method takes a double and does this:
            string ret = d.ToString ("g");
            // TODO: This is plain wrong, but as close as we can get
            // without converting DtoA to C#.
(note the comment!), which defaults to 15 digit precision, ( so your values of 16+ are being rounded.
I guess, since we already have a modified version of this library, we could fix this... purekrome, what are your thoughts?
Closed Dec 28, 2012 at 8:10 AM by freeranger
The problem is in the ECMAScript library, which PK maintains cannot be fixed in YUI directly.The interim solution is to have the bit of script with these large numbers defined in an uncompressed .js file. This can still be combined with the compressed files to benefit from reduced downloads, and only applies to the bit of code defining the numbers, not the bit manipulating them


freeranger wrote Jan 28, 2012 at 12:23 PM

Not as simple as it seemed at first - see discussion thread

wrote Jan 28, 2012 at 1:31 PM

Associated with changeset 74419.

freeranger wrote Jan 28, 2012 at 1:33 PM

Checked in change set improves accuracy, using G16 (as per the discussion thread), but does not eliminate the problem completely

wrote Mar 1, 2012 at 7:53 AM

wrote Dec 28, 2012 at 8:10 AM

wrote Feb 22, 2013 at 12:55 AM

wrote May 16, 2013 at 12:20 PM