Skip to content
This repository was archived by the owner on Mar 1, 2024. It is now read-only.

Returns dramatically wrong results #108

Open
goerz opened this issue Feb 2, 2023 · 2 comments
Open

Returns dramatically wrong results #108

goerz opened this issue Feb 2, 2023 · 2 comments

Comments

@goerz
Copy link

goerz commented Feb 2, 2023

julia> using Formatting
julia> Formatting.fmt(".1e", 0.0003)
"2.10e-04"
@Seelengrab
Copy link
Contributor

For comparison, the Printf stdlib returns the correct result:

julia> using Printf

julia> @sprintf "%.1e" 0.0003
"3.0e-04"

@danielwe
Copy link

danielwe commented Feb 21, 2023

Another example:

julia> using Formatting
julia> Formatting.fmt(".1e", 0.0006)
"5.10e-04"

I've only seen this for these two particular numbers. Same error with 3e-8 and 6e-8. Probably an unfortunate interaction between the truncation algorithm and certain inexact floating point representations.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants