Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct format gG precision handling #91

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

R3G3N3R4T0R
Copy link

@R3G3N3R4T0R R3G3N3R4T0R commented Mar 3, 2025

The current implementation does not adhere to the Python documentation and forward the precision directly to __pfmt_f() and __pfmt_e(), this means the only time it is correct is for numbers in the range of 0.1 <= x < 1.0.
Additionally the Python spec makes sure the printed number will never have trailing zeros by switching to scientific notation when number of integer digits exceeds precision. The current implementation simply hard code it to x < 1E6.

The exact wording is as follows

The precise rules are as follows: suppose that the result formatted with presentation type 'e' and precision p-1 would have exponent exp. Then, if m <= exp < p, where m is -4 for floats and -6 for Decimals, the number is formatted with presentation type 'f' and precision p-1-exp.

Since I am new here I don't know if taking log directly will affect performance, and if floating point imprecision will make it 1 less than the actual exponent, but it currently has the correct behavior.

This will resolve #88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

g format differs from Python
1 participant