Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decimalExpansion gives out unexpected values #72

Open
chokokatana opened this issue Mar 7, 2023 · 3 comments
Open

decimalExpansion gives out unexpected values #72

chokokatana opened this issue Mar 7, 2023 · 3 comments

Comments

@chokokatana
Copy link

I've tried to use the BDouble for storage but at some point I need to use decimalExpansion to format the value to end users, and I was getting really weird values. In a debug session I got the following:

(lldb) po percent
▿ 7450921770987/7450936270987
  - sign : false
  ▿ numerator : 1 element
    - 0 : 7450921770987
  ▿ denominator : 1 element
    - 0 : 7450936270987
  - magnitude : 0.0
  - _precision : 4

(lldb) po percent.decimalDescription
"0.99910"

(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 2)
"0.910"

(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 0)
"0.0"

(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 7)
"0.9999981"

I'm pretty certain a value nearly 1 should not round to 0 when displayed to end users. Is decimalExpansion not the function I'm expecting and should I be using something else? I had expected the value to round to either 1.0 or 0.9.

@mkrd
Copy link
Owner

mkrd commented Mar 7, 2023

Can you set the "rounded" parameter to false and check again? I think the problem might have something to do with rounding at a given precision level.

But still that doesn’t make sense why for example the case with 2 doesn’t give at least two 9's

@mkrd
Copy link
Owner

mkrd commented Mar 7, 2023

Okay I got it, in the case of precision=2, it computes 0.999 so since the third 9 is >= 5, the second 9 is rounded up, but there is a bug so it is replaced with 10 and the number becomes 0.910

@mkrd
Copy link
Owner

mkrd commented Mar 7, 2023

Could be solved easily by checking the last digit of the derived BInt before converting to a string, if .divmod(10).remainder >= 5 then take .divmod(10).quotient + 1, else .divmod(10).quotient. Only after that take the Bint.description and prune 0's from the end if needed (so if 999 is rounded to 1000, then 1.000 has to remove the last zero

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants