Skip to content

Commit

Permalink
refine precision cutoffs for very large requests
Browse files Browse the repository at this point in the history
  • Loading branch information
mflatt committed Nov 5, 2024
1 parent f483697 commit 2d927ce
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions s/strnum.ss
Original file line number Diff line number Diff line change
Expand Up @@ -222,11 +222,11 @@ an exception.
;; bound p; we don't need a tight bound, and adding 2
;; extra bits over `double` precision to make sure
;; rounding will be right
(min p (+ (max a-bits b-bits) 53 2))]
(min p (+ a-bits b-bits 53 2))]
[(= b (bitwise-arithmetic-shift-left 1 (- b-bits 1)))
;; no need for extra precision if the
;; denominator is a power of 2
(min p (+ a-bits b-bits))]
(min p a-bits)]
[else p])])
(let*-values
([(a b)
Expand Down

0 comments on commit 2d927ce

Please sign in to comment.