1. 22 Dec, 2006 1 commit
    • cmiller@zippy.cornsilk.net's avatar
      Bug#22555: STDDEV yields positive result for groups with only one row · 50726b23
      cmiller@zippy.cornsilk.net authored
      When only one row was present, the subtraction of nearly the same number 
      resulted in catastropic cancellation, introducing an error in the 
      VARIANCE calculation near 1e-15.  That was sqrt()ed to get STDDEV, the 
      error was escallated to near 1e-8.  
      
      The simple fix of testing for a row count of 1 and forcing that to yield 
      0.0 is insufficient, as two rows of the same value should also have a
      variance of 0.0, yet the error would be about the same.
      
      So, this patch changes the formula that computes the VARIANCE to be one
      that is not subject to catastrophic cancellation.
      
      In addition, it now uses only (faster-than-decimal) floating point numbers
      to calculate, and renders that to other types on demand.
      50726b23
  2. 11 Nov, 2006 1 commit
  3. 10 Nov, 2006 1 commit
  4. 08 Nov, 2006 1 commit
    • cmiller@zippy.cornsilk.net's avatar
      Bug#10963: LEFT/RIGHT/SUBSTR/.. string functions returns wrong result \ · 6260e12b
      cmiller@zippy.cornsilk.net authored
        on large length
        
      Problem:  Most (all) of the numeric inputs were being coerced into
      int (32 bit) sized variables.  Works OK for sane inputs; any input
      larger than 2^32 (or 2^31 for signed vars) exihibited predictable
      wrapping behavior (up to about 10^18) and then started having really
      strange behaviour past that point (since the conversion to 64 bit int
      from the DECIMAL type can do weird things on out of range numbers).
      
      Solution: 1)  Add many tests.  2)  Convert input from (u)long type to
      (u)longlong.  3)  Do (sometimes multiple) sanity checks on input,
      keeping in mind that sometimes a negative longlong is not a negative
      longlong (if the unsigned_flag is set).  4) Emulate existing behavior
      w/rt negative and "small" out-of-bounds values.
      6260e12b
  5. 07 Nov, 2006 2 commits
  6. 06 Nov, 2006 3 commits
  7. 04 Nov, 2006 4 commits
  8. 03 Nov, 2006 12 commits
  9. 02 Nov, 2006 15 commits