Add json decoder

[tiehuis]

Everyone needs a Json decoder so here is one.

This is blocked on parsing floats #375. I figured I'd push for now however since it is otherwise largely complete.

Streaming Decoder

This implements a streaming json decoder which conforms to RFC8259 (the latest spec). This decoder returns tokens indicating an event that occurs on the json stream. This is pretty much exactly the same as any other streaming parser but I've opted for the token approach versus callbacks since nobody likes callbacks.

I've attempted to keep the streaming states largely branchless (although there are still a lot) where possible just because. This is also marginally more complex than normal since we decode without any lookahead. This is also the reason we have a slightly odd interface which returns possibly two tokens and not just one.

This decoder doesn't allocate any memory and should be used to build more friendly decoders.

Value/Tree Decoder

I've also implemented a more standard tree decoder on top of the streaming version. This allocates nodes as needed and is useful for parsing unknown data formats.

Tests

Tests are taken from here: https://github.com/nst/JSONTestSuite
Recommended accompanying reading: http://seriot.ch/parsing_json.php

Other Things to do

• Decode Json into a struct based on it's fields. This should actually be relatively easy as the streaming parser does the majority of the work.
• Benchmark to get a relative idea of the performance would be nice.
• An accompanying encoder of course needs to be written. This is fairly straight-forward in comparison.

[tiehuis]

I figure we'll probably move all the encoding/decoding formats into a subfolder at some point.

[tiehuis]

We also cannot zero-copy if we encounter a standard escape code (e.g. \n) which needs to be transformed.

[tiehuis]

This was a mildly confusing thing I came across when I had the specific value variants (ObjectSeparator, ValueEnd) not at the start of the enum. Printing the state names gave unexpected results.

Would indexing based on enumeration value make more sense?

[andrewrk]

Ah right. @tagName was around before enum members could have explicit values. I'll open an issue. #976

[tiehuis]

#489 may be nice here (more so on the stack overflow checks). Second transition is only required on numbers followed directly by }, ].

[tiehuis]

Want a has_escape flag set here as in the u unicode case.

[tiehuis]

Forgot these were still std.debug.warn outputs. Custom formatting functions would be useful here instead.

[andrewrk]

If this function is for debugging - there's a pattern of using foo.dump() as a method that returns void and prints a bunch of stuff to stderr about foo to aid in debugging.

[tiehuis]

What happens here if we pass an allocator derived from an ArenaAllocator and the use that as an input to our own ArenaAllocator? Are there other wrapping allocators which don't handle this correctly?

[andrewrk]

The worst that can happen is inefficient space usage. Allocators that wrap other allocators respect the API that Allocator provides and so everything will work correctly.

[tiehuis]

This should be a separate function such as mp.end(&mt1). The only token that is possible here would be to terminate a number value. We can never get multiple.

[tiehuis]

We could expose some helper methods on Value to make accessing nested objects slightly easier. Non-essential for the moment.

[tiehuis]

May be more idiomatic/nicer to return a double-token struct here.

[andrewrk]

I wonder if we could make it like the zig tokenizer, and make it always return exactly 1 token? Here's the signature:

    pub fn next(self: &Tokenizer) Token {

Here's what's nice about it:

• It can't fail. But it can return Token.Invalid. The invalid token has start and end of the invalid character(s), and then normal tokens can follow.
• It always returns 1 token.
  • If you get to the end it will always return Token.Eof.
  • If it needs to emit 2 tokens, it saves the one token and feeds it to the caller for next time, making the API simpler for the caller.

[andrewrk]

I just realized, the difference between the zig tokenizer and this json one is that the json one is streaming. So that makes sense.

If the caller already has to handle possibly greater than 1 token though, then would it make more sense for the feed function to accept a buf: []const u8 instead of c: u8, and then return N tokens?

...oh then it would need to be a callback, because if we needed to return > N tokens, and the return token slice was too small, we would lose data.

Alright, you're 2 steps ahead of me :-)

[tiehuis]

You could still change this to accept a slice. You instead could take a slice of tokens for output and fill until the input slice runs out or you run out of usable token space. If you run out of token space, return an error.NeedMoreTokens and save the current position in the buffer until the caller can use filled tokens and call again.

More complicated with the only real benefit being a saved function call cost. Technically you could perform faster matching of quoted-strings, ends of numbers etc by skipping some state changes but that would be special-casing longer buffer sizes and may have the reverse effect in other cases.

[BraedonWooding]

On the note of your implementation: I think it would be beneficial to allow for comments :). A simple // to ignore the rest of the line would be trivial to implement and is really nice for JSON systems to have, this would break our '100% compatibility with latest spec' but that statement isn't worth the removal of comments.

On the note of whether or not this should be in the actual std library is an interesting question; I think the problem with having JSON in the std is should we also have XML and TOML? And maybe YAML? Where do we stop? I personally think that this is probably more of a 'external' library thing? Though I quite like your library (I took a very different approach, would be interested in seeing which one is faster; mine is a lot shorter since I just took a less 'structured' approach due to wanting to test out Zig).

Maybe this indicates that the std should also have a series of 'external' libraries that are also downloaded with your install of Zig (or maybe we have a easy homebrew/apt-get/whatever installation thing to do them) and when we get a package manager be handled by that? These could include things like a GUI library, a science library and well JSON/TOML/... and so on, these will prevent the std becoming bloated?

[alexnask]

I'm not too sure where I stand on this being an std package or not.
I think @BraedonWooding's argument is valid, however I think JSON is now so commonly used that it may deserve being in the std even when XML, TOML, YAML and friends are absent.

I think a cool layer on top of this library would be the ability to parse JSON into arbitrary types.
Using @typeinfo on the type requested, we can essentially build a schema and fill an object of the type requested, or return some error (at comptime for fields that cannot be parsed from JSON, at runtime if the JSON we are parsing does not fit the schema in some way).

[andrewrk]

wow, nice work!

I have some suggestions, but this is definitely a welcome addition to the standard library.

Would you mind if I took a pass at it and played around with the API and implementation?

[andrewrk]

If this function is for debugging - there's a pattern of using foo.dump() as a method that returns void and prints a bunch of stuff to stderr about foo to aid in debugging.

[andrewrk]

Ah right. @tagName was around before enum members could have explicit values. I'll open an issue. #976

[andrewrk]

I wonder if we could make it like the zig tokenizer, and make it always return exactly 1 token? Here's the signature:

    pub fn next(self: &Tokenizer) Token {

Here's what's nice about it:

• It can't fail. But it can return Token.Invalid. The invalid token has start and end of the invalid character(s), and then normal tokens can follow.
• It always returns 1 token.
  • If you get to the end it will always return Token.Eof.
  • If it needs to emit 2 tokens, it saves the one token and feeds it to the caller for next time, making the API simpler for the caller.

[andrewrk]

max_stack_size should probably be a runtime parameter to init and then we would allocate stack on the heap. Because we may want to decide how much nesting to allow in the json data at runtime - or at least in the API of std.json

[tiehuis]

I was thinking if it was worthwhile to make the max_stack_size parameter in the streaming parser a comptime variable for this specific case. For the value decoder, I'd actually change this to just use an ArrayList and not be dependent on the streaming parsers stack size. The rationale being that the common case will likely not be nested so reduce memory usage for a few extra dynamic allocations/copies in the worst case.

[andrewrk]

The worst that can happen is inefficient space usage. Allocators that wrap other allocators respect the API that Allocator provides and so everything will work correctly.

[andrewrk]

you could build this with "[" ** 100000

[andrewrk]

you could build this with "[{\"\":" ** 100000

[andrewrk]

you could build this with ("[" ** 500) ++ ("]" ** 500)

[andrewrk]

is the u5 meant to save space? zig will use the minimum tag size if you do not specify. This feature is more meant for packed structs where you want to control the bit layout.

[andrewrk]

it would be idiomatic to rename TokenId to Id and nest it inside Token. Then it can be referred to with Token.Id

[tiehuis]

@andrewrk

Nope, go for it.

@BraedonWooding

Regarding comments I'm personally not convinced. I realize that it is useful if using Json as a configuration specification of sorts, but that feels like an anti-pattern to me. If we allowed comments then there also is the question of how lenient we should be in terms of extensions. Trailing commas, hex literals are kind of nice, but also aren't to spec for example.

Regarding what goes in the stdlib. I don't disagree that we shouldn't throw every under the sun in it. However, I also don't think we should keep it minimal and lacking in some very common areas just for the sake of size, especially for small components like this. We have lazy-parsing in zig and we also don't link unused code from std in to binaries so the main costs are the maintenance burden, and the built distribution size.

Json is fairly ubiquitous these days as well, so regardless of the above thoughts I'd argue it should exist in std. If you check nuget or crates.io the Json packages these are some of the most imported packages.

@alexnask

I was thinking the exact same thing and was waiting on typeInfo for that!

[andrewrk]

@tiehuis as far as I'm concerned this is ready for merge. After I realized the streaming ability I didn't have any ideas for improvement.

[tiehuis]

I'll follow up with the struct serialization/deserialization and float parsing soon in another pr.

[komuw]

@andrewrk hi, First time zig user here.
I'm trying to work with Json using zig and two things have come up;

1. https://ziglang.org/documentation/master does not have any documentaion on the json package
2. I have been able to deserialize json into zig types but I havent been able to serialize zig types into json. Is there an example of serialization that I can look at? The std only has a deserialization example:

   zig/std/json.zig

   Line 1350 in d40c4e7

   test "json parser dynamic" {

PS: I can send a patch to add documentation for json if you show me where that should go.

[emekoi]

@komuw the entirety of the standard library is currently undocumented. documentation is planned but it's not one of the goals for the 0.4.0 release.

[komuw]

@emekoi thanks.
you wouldn't happen to know; Is there an example of serialization that I can look at?

[emekoi]

i don't think that automatic serialization is possible but it's could be wrong. the best i can think of is accessing each field from the json directly assigning them manually.

[komuw]

right now I'm doing;
https://github.com/komuw/komu.engineer/blob/2f5678bdae7c606f683dc0508e1aad13442fbe8d/blogs/lambda-shim/main.zig#L35-L59

which feels totally wrong

[andrewrk]

documentation is planned but it's not one of the goals for the 0.4.0 release.

It actually is: #21

[emekoi]

for required_buf_length, have you tried making it a const?

[komuw]

for required_buf_length, have you tried making it a const?

@emekoi if I do make it a const

/home/lambda-shim/main.zig:65:24: error: unable to evaluate constant expression
    var all_together: [required_buf_length]u8 = undefined;

[ghost]

I'll follow up with the struct serialization/deserialization

Did this ever happen / is someone on it?

I saw json.parseInto appear in a code snippet here #2930 but I wasn't sure if it was just hypothetical. @daurnimator

[daurnimator]

I saw json.parseInto appear in a code snippet here #2930 but I wasn't sure if it was just hypothetical. @daurnimator

Hypothetical.