ConsolidateMetadata2

[norlandrhagen]

ConsolidateMetadata doesn't seem to be creating a .zmetadata in the reference recipe tests.

Based on Martin Durants comments here: fsspec/kerchunk#240 (comment), this PR updates how the .zmetadata is being created.

Some open questions:

• It seemed like the previous tests of zarr.open_consolidated were working...without a .zmetadata file?
• This method path.fs.save_json(ref_path) calls save_json`. How do we modify the parquet stores?

cc @abarciauskas-bgse @ranchodeluxe

[abarciauskas-bgse]

Thanks for looking at this @norlandrhagen ! I was just reading about the parquet implementation and noticed it says:

For now, kerchunk.combine.MultiZarrToZarr only operates on JSON/dict input. Therefore, refs_to_dataframe can only be used on the final output reference set.

Since consolidate_metadata also seems to assume JSON/dict input I would not expect to be able to pass geoparquet to consolidate_metadata either.

Did you consolidate metadata before converting to geoparquet for the CMIP6 dataset you worked on?

[norlandrhagen]

Update:

Thanks @abarciauskas-bgse & @ranchodeluxe for talking through this!

According to discussion in this PR: #675. It seems like there isn't really a point to consolidate the metadata for a Kerchunk reference.

• We should probably remove any Kerchunk specific logic for consolidating references.
• Consolidate_Metadata works on Kerchunk json references (Although it's probably not needed at all)
• ConsolidateMetadata fails on Kerchunk parquet references.

[ranchodeluxe]

According to discussion in this PR: #675. It seems like there isn't really a point to consolidate the metadata for a Kerchunk reference.

* We should probably remove any Kerchunk specific logic for consolidating references.

* Consolidate_Metadata works on Kerchunk json references (Although it's probably not needed at all)

* ConsolidateMetadata fails on Kerchunk parquet references.

(just adding these bullets also)

• Include the fix for write_combined_reference regarding parquet files

• Make sure ConsolidateMetadata takes target_root and passes along fsspec kwargs

[norlandrhagen]

• Consolidating references for Kerchunk recipes have been removed.
• Fixed a major bug where we were writing json files with a .parquet suffix.
• Updated the LazyReferenceMapper.create method to kwargs.

[ranchodeluxe]

Thanks for doing this @norlandrhagen 🥳 Can't wait to get this merged so I can double check if my assumption about fsspec auth args is correct or not

[norlandrhagen]

@cisaacstern would you mind taking a look at this when you get a chance?