Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add minio to Integration Tests #638

Merged
merged 28 commits into from
Nov 6, 2023
Merged

Add minio to Integration Tests #638

merged 28 commits into from
Nov 6, 2023

Conversation

norlandrhagen
Copy link
Contributor

@norlandrhagen norlandrhagen commented Oct 13, 2023 •
edited
Loading

Adds minio to integration tests to allow the integration tests to be ran against an s3 bucket.

@norlandrhagen norlandrhagen added the test-integration Apply this label to run integration tests on a PR. label Oct 13, 2023
@norlandrhagen
Copy link
Contributor Author

minio tests are failing locally (proc.returncode == 1). Example modified s3 json config:

{
    "Bake": {
        "bakery_class": "pangeo_forge_runner.bakery.local.LocalDirectBakery"
    },
    "TargetStorage": {
        "fsspec_class": "s3fs.S3FileSystem",
        "fsspec_args": {
            "key": "780364be353640797d913ef225d9b437",
            "secret": "8d91ffe53d0c9f1469e07774737020dd",
            "client_kwargs": {
                "endpoint_url": "http://10.0.0.191:19555"
            }
        },
        "root_path": "/private/var/folders/mb/7d7yq_4j2qgdfm_j3j4tsyl40000gn/T/pytest-of-nrhagen/pytest-103/tmp0/target"
    },
    "InputCacheStorage": {
        "fsspec_class": "s3fs.S3FileSystem",
        "fsspec_args": {
            "key": "780364be353640797d913ef225d9b437",
            "secret": "8d91ffe53d0c9f1469e07774737020dd",
            "client_kwargs": {
                "endpoint_url": "http://10.0.0.191:19555"
            }
        },
        "root_path": "/private/var/folders/mb/7d7yq_4j2qgdfm_j3j4tsyl40000gn/T/pytest-of-nrhagen/pytest-103/tmp0/cache"
    }
}

@norlandrhagen
Copy link
Contributor Author

The minio integration tests are failing locally for the nasa-oisst recipe.

pytest tests/test_integration.py -k 'test_integration[minio_confpath-noaa-oisst]' --run-integration

Traceback:


CompletedProcess(args=['sh', '/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/examples/runner-commands/bake.sh'], returncode=1, stdout=b'Target Storage is FSSpecTarget(S3FileSystem(key=<str>, secret=<str>, client_kwargs=<dict>, root_path="/private/var/folders/mb/7d7yq_4j2qgdfm_j3j4tsyl40000gn/T/pytest-of-nrhagen/pytest-112/tmp0/target")\n\nInput Cache Storage is CacheFSSpecTarget(S3FileSystem(key=<str>, secret=<str>, client_kwargs=<dict>, root_path="/private/var/folders/mb/7d7yq_4j2qgdfm_j3j4tsyl40000gn/T/pytest-of-nrhagen/pytest-112/tmp0/cache")\n\nMetadata Cache Storage is MetadataTarget(AbstractFileSystem(, root_path="")\n\n{\'TARGET_STORAGE\': FSSpecTarget(fs=<s3fs.core.S3FileSystem object at 0x139fc8af0>, root_path=\'/private/var/folders/mb/7d7yq_4j2qgdfm_j3j4tsyl40000gn/T/pytest-of-nrhagen/pytest-112/tmp0/target\'), \'INPUT_CACHE_STORAGE\': CacheFSSpecTarget(fs=<s3fs.core.S3FileSystem object at 0x139fc8af0>, root_path=\'/private/var/folders/mb/7d7yq_4j2qgdfm_j3j4tsyl40000gn/T/pytest-of-nrhagen/pytest-112/tmp0/cache\')}\n{\'StoreToZarr\': {\'target_root\': \'TARGET_STORAGE\'}, \'WriteCombinedReference\': {\'target_root\': \'TARGET_STORAGE\'}, \'OpenURLWithFSSpec\': {\'cache\': \'INPUT_CACHE_STORAGE\'}}\nParsing recipes...\nBaking only recipe_id=\'noaa-oisst\'\nConverting string literal type hint to Any: "beam.PCollection[Tuple[Index, xr.Dataset]]"\nConverting string literal type hint to Any: "beam.PCollection[zarr.storage.FSStore]"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "Indexed[T]"\nConverting string literal type hint to Any: "XarraySchema"\nConverting string literal type hint to Any: "Indexed[T]"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "XarraySchema"\nConverting string literal type hint to Any: "zarr.storage.FSStore"\nConverting string literal type hint to Any: "Optional[Dict[str, int]]"\nConverting string literal type hint to Any: "Optional[Dict[str, str]]"\nConverting string literal type hint to Any: "zarr.storage.FSStore"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection"\nConverting string literal type hint to Any: "beam.PCollection[Tuple[Index, xr.Dataset]]"\nConverting string literal type hint to Any: "beam.PCollection[zarr.storage.FSStore]"\nRunning job for recipe noaa-oisst\n\nDefault Python SDK image for environment is apache/beam_python3.10_sdk:2.42.0\nDefault Python SDK image for environment is apache/beam_python3.10_sdk:2.42.0\n==================== <function annotate_downstream_side_inputs at 0x14c1eec20> ====================\n==================== <function fix_side_input_pcoll_coders at 0x14c1eed40> ====================\n==================== <function pack_combiners at 0x14c1ef250> ====================\n==================== <function lift_combiners at 0x14c1ef2e0> ====================\n==================== <function expand_sdf at 0x14c1ef490> ====================\n==================== <function expand_gbk at 0x14c1ef520> ====================\n==================== <function sink_flattens at 0x14c1ef640> ====================\n==================== <function greedily_fuse at 0x14c1ef6d0> ====================\n==================== <function read_to_impulse at 0x14c1ef760> ====================\n==================== <function impulse_to_input at 0x14c1ef7f0> ====================\n==================== <function sort_stages at 0x14c1efa30> ====================\n==================== <function add_impulse_to_dangling_transforms at 0x14c1efb50> ====================\n==================== <function setup_timer_mapping at 0x14c1ef9a0> ====================\n==================== <function populate_data_channel_coders at 0x14c1efac0> ====================\nstarting control server on port 59588\nstarting data server on port 59589\nstarting state server on port 59590\nstarting logging server on port 59591\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c4f90c0> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c4f9990> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c4fa200> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c4faaa0> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c4fb340> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c4fbb80> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c518430> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nCreated Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.SubprocessSdkWorkerHandler object at 0x14c518ca0> for environment ref_Environment_default_environment_1 (beam:env:harness_subprocess_python:v1, b\'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/python3.10 -m apache_beam.runners.worker.sdk_worker_main\')\nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 366772890 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 373854875 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 375899791 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 379344940 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 379706859 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 405642032 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 406009197 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 405750989 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 411961078 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 413661956 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 411810159 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 413588047 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 412060022 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 413765907 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 416978836 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 417413949 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 416764974 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 417089939 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 428899049 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 429812908 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 433959007 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 435131788 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 416442155 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 433811902 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 440031051 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 435342073 } message: "semi_persistent_directory: None" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:110" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 440607070 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 433887004 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 416731834 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 441687107 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 442451000 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 444014072 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 442355871 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 455060005 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 455202102 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: WARN timestamp {   seconds: 1698082466   nanos: 439848899 } message: "Discarding unparseable args: [\\\'--pipeline_type_check\\\', \\\'--direct_runner_use_stacked_bundle\\\']" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/options/pipeline_options.py:339" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 444555997 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 455106973 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 455693006 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 445686101 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 455730915 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 441323041 } message: "Pipeline_options: {\\\'runner\\\': \\\'DirectRunner\\\', \\\'direct_num_workers\\\': 0, \\\'direct_running_mode\\\': \\\'multi_processing\\\', \\\'pickle_library\\\': \\\'cloudpickle\\\', \\\'save_main_session\\\': True, \\\'sdk_worker_parallelism\\\': \\\'1\\\', \\\'environment_cache_millis\\\': \\\'0\\\'}" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:126" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 444854021 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 445989131 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 458930015 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 458842992 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 444308042 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 444600105 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 455970048 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 458756208 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 473907947 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_4" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 474178075 } message: "State channel established." instruction_id: "bundle_4" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 447117090 } message: "Creating state cache with size 0" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/statecache.py:172" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 447418212 } message: "Creating insecure control channel for localhost:59588." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:181" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 479802131 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 479781150 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 480629920 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 480663061 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 481863021 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 483614921 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 482540130 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_4" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 482641935 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 484976053 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 482892990 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 481898069 } message: "Control channel established." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:189" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 482478141 } message: "Initializing SDKHarness with unbounded number of workers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:232" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 485156059 } message: "Python sdk harness starting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:182" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 492549896 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_3" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 492783069 } message: "State channel established." instruction_id: "bundle_3" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 501120090 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_3" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 503802776 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_6" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 504024982 } message: "State channel established." instruction_id: "bundle_6" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 502621889 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_5" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 502842187 } message: "State channel established." instruction_id: "bundle_5" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 511868953 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_6" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 517362833 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_5" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 520880222 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_7" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 521145105 } message: "State channel established." instruction_id: "bundle_7" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 532382011 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_8" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 535109996 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_1" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 531966924 } message: "Creating insecure state channel for localhost:59590." instruction_id: "bundle_2" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:858" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 533601999 } message: "State channel established." instruction_id: "bundle_8" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 535346031 } message: "State channel established." instruction_id: "bundle_1" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 532220840 } message: "State channel established." instruction_id: "bundle_2" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:865" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 542032957 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_7" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 561668157 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_8" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 561428070 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_1" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082466   nanos: 566703081 } message: "Creating client data channel for localhost:59589" instruction_id: "bundle_2" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:772" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082467   nanos: 877338171 } message: "Caching file \\\'https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810902.nc\\\'" instruction_id: "bundle_11" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/OpenURLWithFSSpec/MapWithConcurrencyLimit/open_url" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/storage.py:179" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082467   nanos: 917903900 } message: "Caching file \\\'https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc\\\'" instruction_id: "bundle_9" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/OpenURLWithFSSpec/MapWithConcurrencyLimit/open_url" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/storage.py:179" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082468   nanos: 50256967 } message: "Copying remote file \\\'https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810902.nc\\\' to cache" instruction_id: "bundle_11" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/OpenURLWithFSSpec/MapWithConcurrencyLimit/open_url" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/storage.py:190" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082468   nanos: 57458877 } message: "Copying remote file \\\'https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc\\\' to cache" instruction_id: "bundle_9" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/OpenURLWithFSSpec/MapWithConcurrencyLimit/open_url" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/storage.py:190" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082473   nanos: 583986997 } message: "Splitting fragment = ({Dimension(name=\\\'time\\\', operation=<CombineOp.CONCAT: 2>): IndexedPosition(value=1, indexed=True, dimsize=2)}, <xarray.Dataset>\\nDimensions:  (time: 1, zlev: 1, lat: 720, lon: 1440)\\nCoordinates:\\n  * lat      (lat) float32 -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\\n  * lon      (lon) float32 0.125 0.375 0.625 0.875 ... 359.1 359.4 359.6 359.9\\n  * time     (time) datetime64[ns] 1981-09-02T12:00:00\\n  * zlev     (zlev) float32 0.0\\nData variables:\\n    anom     (time, zlev, lat, lon) float32 ...\\n    err      (time, zlev, lat, lon) float32 ...\\n    ice      (time, zlev, lat, lon) float32 ...\\n    sst      (time, zlev, lat, lon) float32 ...\\nAttributes: (12/37)\\n    title:                      NOAA/NCEI 1/4 Degree Daily Optimum Interpolat...\\n    source:                     ICOADS, NCEP_GTS, GSFC_ICE, NCEP_ICE, Pathfin...\\n    id:                         oisst-avhrr-v02r01.19810902.nc\\n    naming_authority:           gov.noaa.ncei\\n    summary:                    NOAAs 1/4-degree Daily Optimum Interpolation ...\\n    cdm_data_type:              Grid\\n    ...                         ...\\n    metadata_link:              https://doi.org/10.25921/RE9P-PT57\\n    ncei_template_version:      NCEI_NetCDF_Grid_Template_v2.0\\n    comment:                    Data was converted from NetCDF-3 to NetCDF-4 ...\\n    sensor:                     Thermometer, AVHRR\\n    Conventions:                CF-1.6, ACDD-1.3\\n    references:                 Reynolds, et al.(2007) Daily High-Resolution-...), with target_chunks = {} and schema = {\\\'attrs\\\': {\\\'sensor\\\': \\\'Thermometer, AVHRR\\\', \\\'date_created\\\': \\\'2020-05-08T19:05:13Z\\\', \\\'geospatial_lon_units\\\': \\\'degrees_east\\\', \\\'geospatial_lat_units\\\': \\\'degrees_north\\\', \\\'metadata_link\\\': \\\'https://doi.org/10.25921/RE9P-PT57\\\', \\\'naming_authority\\\': \\\'gov.noaa.ncei\\\', \\\'comment\\\': \\\'Data was converted from NetCDF-3 to NetCDF-4 format with metadata updates in November 2017.\\\', \\\'keywords_vocabulary\\\': \\\'Global Change Master Directory (GCMD) Earth Science Keywords\\\', \\\'creator_url\\\': \\\'https://www.ncei.noaa.gov/\\\', \\\'cdm_data_type\\\': \\\'Grid\\\', \\\'instrument\\\': \\\'Earth Remote Sensing Instruments > Passive Remote Sensing > Spectrometers/Radiometers > Imaging Spectrometers/Radiometers > AVHRR > Advanced Very High Resolution Radiometer\\\', \\\'instrument_vocabulary\\\': \\\'Global Change Master Directory (GCMD) Instrument Keywords\\\', \\\'standard_name_vocabulary\\\': \\\'CF Standard Name Table (v40, 25 January 2017)\\\', \\\'processing_level\\\': \\\'NOAA Level 4\\\', \\\'summary\\\': \\\'NOAAs 1/4-degree Daily Optimum Interpolation Sea Surface Temperature (OISST) (sometimes referred to as Reynolds SST, which however also refers to earlier products at different resolution), currently available as version v02r01, is created by interpolating and extrapolating SST observations from different sources, resulting in a smoothed complete field. The sources of data are satellite (AVHRR) and in situ platforms (i.e., ships and buoys), and the specific datasets employed may change over time. At the marginal ice zone, sea ice concentrations are used to generate proxy SSTs.  A preliminary version of this file is produced in near-real time (1-day latency), and then replaced with a final version after 2 weeks. Note that this is the AVHRR-ONLY DOISST, available from Oct 1981, but there is a companion DOISST product that includes microwave satellite data, available from June 2002\\\', \\\'references\\\': \\\'Reynolds, et al.(2007) Daily High-Resolution-Blended Analyses for Sea Surface Temperature (available at https://doi.org/10.1175/2007JCLI1824.1). Banzon, et al.(2016) A long-term record of blended satellite and in situ sea-surface temperature for climate monitoring, modeling and environmental studies (available at https://doi.org/10.5194/essd-8-165-2016). Huang et al. (2020) Improvements of the Daily Optimum Interpolation Sea Surface Temperature (DOISST) Version v02r01, submitted.Climatology is based on 1971-2000 OI.v2 SST. Satellite data: Pathfinder AVHRR SST and Navy AVHRR SST. Ice data: NCEP Ice and GSFC Ice.\\\', \\\'institution\\\': \\\'NOAA/National Centers for Environmental Information\\\', \\\'geospatial_lat_max\\\': 90.0, \\\'geospatial_lon_max\\\': 360.0, \\\'geospatial_lat_resolution\\\': 0.25, \\\'geospatial_lat_min\\\': -90.0, \\\'title\\\': \\\'NOAA/NCEI 1/4 Degree Daily Optimum Interpolation Sea Surface Temperature (OISST) Analysis, Version 2.1 - Final\\\', \\\'product_version\\\': \\\'Version v02r01\\\', \\\'creator_email\\\': \\\'[email protected]\\\', \\\'Conventions\\\': \\\'CF-1.6, ACDD-1.3\\\', \\\'keywords\\\': \\\'Earth Science > Oceans > Ocean Temperature > Sea Surface Temperature\\\', \\\'ncei_template_version\\\': \\\'NCEI_NetCDF_Grid_Template_v2.0\\\', \\\'history\\\': \\\'Final file created using preliminary as first guess, and 3 days of AVHRR data. Preliminary uses only 1 day of AVHRR data.\\\', \\\'geospatial_lon_min\\\': 0.0, \\\'platform\\\': \\\'Ships, buoys, Argo floats, MetOp-A, MetOp-B\\\', \\\'platform_vocabulary\\\': \\\'Global Change Master Directory (GCMD) Platform Keywords\\\', \\\'geospatial_lon_resolution\\\': 0.25, \\\'source\\\': \\\'ICOADS, NCEP_GTS, GSFC_ICE, NCEP_ICE, Pathfinder_AVHRR, Navy_AVHRR\\\', \\\'date_modified\\\': \\\'2020-05-08T19:05:13Z\\\'}, \\\'coords\\\': {\\\'lon\\\': {\\\'dims\\\': (\\\'lon\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Longitude\\\', \\\'units\\\': \\\'degrees_east\\\', \\\'grids\\\': \\\'Uniform grid from 0.125 to 359.875 by 0.25\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (1440,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'original_shape\\\': (1440,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1440,), \\\'shuffle\\\': True}}, \\\'lat\\\': {\\\'dims\\\': (\\\'lat\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Latitude\\\', \\\'units\\\': \\\'degrees_north\\\', \\\'grids\\\': \\\'Uniform grid from -89.875 to 89.875 by 0.25\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (720,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'original_shape\\\': (720,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (720,), \\\'shuffle\\\': True}}, \\\'zlev\\\': {\\\'dims\\\': (\\\'zlev\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Sea surface height\\\', \\\'actual_range\\\': \\\'0, 0\\\', \\\'positive\\\': \\\'down\\\', \\\'units\\\': \\\'meters\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (1,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'original_shape\\\': (1,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1,), \\\'shuffle\\\': True}}, \\\'time\\\': {\\\'dims\\\': (\\\'time\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Center time of the day\\\'}, \\\'dtype\\\': \\\'datetime64[ns]\\\', \\\'shape\\\': (2,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'units\\\': \\\'days since 1978-01-01 12:00:00\\\', \\\'original_shape\\\': (1,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1,), \\\'shuffle\\\': True}}}, \\\'data_vars\\\': {\\\'err\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Estimated error standard deviation of analysed_sst\\\', \\\'valid_min\\\': 0, \\\'valid_max\\\': 1000, \\\'units\\\': \\\'Celsius\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}, \\\'ice\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Sea ice concentration\\\', \\\'valid_min\\\': 0, \\\'valid_max\\\': 100, \\\'units\\\': \\\'%\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}, \\\'sst\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Daily sea surface temperature\\\', \\\'valid_min\\\': -300, \\\'valid_max\\\': 4500, \\\'units\\\': \\\'Celsius\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}, \\\'anom\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Daily sea surface temperature anomalies\\\', \\\'valid_min\\\': -1200, \\\'valid_max\\\': 1200, \\\'units\\\': \\\'Celsius\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}}, \\\'dims\\\': {\\\'lon\\\': 1440, \\\'lat\\\': 720, \\\'zlev\\\': 1, \\\'time\\\': 2}, \\\'chunks\\\': {\\\'time\\\': {1: 1, 0: 1}}}" instruction_id: "bundle_39" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/Rechunk/FlatMap(split_fragment)" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py:34" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082473   nanos: 636862039 } message: "Splitting fragment = ({Dimension(name=\\\'time\\\', operation=<CombineOp.CONCAT: 2>): IndexedPosition(value=0, indexed=True, dimsize=2)}, <xarray.Dataset>\\nDimensions:  (time: 1, zlev: 1, lat: 720, lon: 1440)\\nCoordinates:\\n  * lat      (lat) float32 -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\\n  * lon      (lon) float32 0.125 0.375 0.625 0.875 ... 359.1 359.4 359.6 359.9\\n  * time     (time) datetime64[ns] 1981-09-01T12:00:00\\n  * zlev     (zlev) float32 0.0\\nData variables:\\n    anom     (time, zlev, lat, lon) float32 ...\\n    err      (time, zlev, lat, lon) float32 ...\\n    ice      (time, zlev, lat, lon) float32 ...\\n    sst      (time, zlev, lat, lon) float32 ...\\nAttributes: (12/37)\\n    title:                      NOAA/NCEI 1/4 Degree Daily Optimum Interpolat...\\n    source:                     ICOADS, NCEP_GTS, GSFC_ICE, NCEP_ICE, Pathfin...\\n    id:                         oisst-avhrr-v02r01.19810901.nc\\n    naming_authority:           gov.noaa.ncei\\n    summary:                    NOAAs 1/4-degree Daily Optimum Interpolation ...\\n    cdm_data_type:              Grid\\n    ...                         ...\\n    metadata_link:              https://doi.org/10.25921/RE9P-PT57\\n    ncei_template_version:      NCEI_NetCDF_Grid_Template_v2.0\\n    comment:                    Data was converted from NetCDF-3 to NetCDF-4 ...\\n    sensor:                     Thermometer, AVHRR\\n    Conventions:                CF-1.6, ACDD-1.3\\n    references:                 Reynolds, et al.(2007) Daily High-Resolution-...), with target_chunks = {} and schema = {\\\'attrs\\\': {\\\'sensor\\\': \\\'Thermometer, AVHRR\\\', \\\'date_created\\\': \\\'2020-05-08T19:05:13Z\\\', \\\'geospatial_lon_units\\\': \\\'degrees_east\\\', \\\'geospatial_lat_units\\\': \\\'degrees_north\\\', \\\'metadata_link\\\': \\\'https://doi.org/10.25921/RE9P-PT57\\\', \\\'naming_authority\\\': \\\'gov.noaa.ncei\\\', \\\'comment\\\': \\\'Data was converted from NetCDF-3 to NetCDF-4 format with metadata updates in November 2017.\\\', \\\'keywords_vocabulary\\\': \\\'Global Change Master Directory (GCMD) Earth Science Keywords\\\', \\\'creator_url\\\': \\\'https://www.ncei.noaa.gov/\\\', \\\'cdm_data_type\\\': \\\'Grid\\\', \\\'instrument\\\': \\\'Earth Remote Sensing Instruments > Passive Remote Sensing > Spectrometers/Radiometers > Imaging Spectrometers/Radiometers > AVHRR > Advanced Very High Resolution Radiometer\\\', \\\'instrument_vocabulary\\\': \\\'Global Change Master Directory (GCMD) Instrument Keywords\\\', \\\'standard_name_vocabulary\\\': \\\'CF Standard Name Table (v40, 25 January 2017)\\\', \\\'processing_level\\\': \\\'NOAA Level 4\\\', \\\'summary\\\': \\\'NOAAs 1/4-degree Daily Optimum Interpolation Sea Surface Temperature (OISST) (sometimes referred to as Reynolds SST, which however also refers to earlier products at different resolution), currently available as version v02r01, is created by interpolating and extrapolating SST observations from different sources, resulting in a smoothed complete field. The sources of data are satellite (AVHRR) and in situ platforms (i.e., ships and buoys), and the specific datasets employed may change over time. At the marginal ice zone, sea ice concentrations are used to generate proxy SSTs.  A preliminary version of this file is produced in near-real time (1-day latency), and then replaced with a final version after 2 weeks. Note that this is the AVHRR-ONLY DOISST, available from Oct 1981, but there is a companion DOISST product that includes microwave satellite data, available from June 2002\\\', \\\'references\\\': \\\'Reynolds, et al.(2007) Daily High-Resolution-Blended Analyses for Sea Surface Temperature (available at https://doi.org/10.1175/2007JCLI1824.1). Banzon, et al.(2016) A long-term record of blended satellite and in situ sea-surface temperature for climate monitoring, modeling and environmental studies (available at https://doi.org/10.5194/essd-8-165-2016). Huang et al. (2020) Improvements of the Daily Optimum Interpolation Sea Surface Temperature (DOISST) Version v02r01, submitted.Climatology is based on 1971-2000 OI.v2 SST. Satellite data: Pathfinder AVHRR SST and Navy AVHRR SST. Ice data: NCEP Ice and GSFC Ice.\\\', \\\'institution\\\': \\\'NOAA/National Centers for Environmental Information\\\', \\\'geospatial_lat_max\\\': 90.0, \\\'geospatial_lon_max\\\': 360.0, \\\'geospatial_lat_resolution\\\': 0.25, \\\'geospatial_lat_min\\\': -90.0, \\\'title\\\': \\\'NOAA/NCEI 1/4 Degree Daily Optimum Interpolation Sea Surface Temperature (OISST) Analysis, Version 2.1 - Final\\\', \\\'product_version\\\': \\\'Version v02r01\\\', \\\'creator_email\\\': \\\'[email protected]\\\', \\\'Conventions\\\': \\\'CF-1.6, ACDD-1.3\\\', \\\'keywords\\\': \\\'Earth Science > Oceans > Ocean Temperature > Sea Surface Temperature\\\', \\\'ncei_template_version\\\': \\\'NCEI_NetCDF_Grid_Template_v2.0\\\', \\\'history\\\': \\\'Final file created using preliminary as first guess, and 3 days of AVHRR data. Preliminary uses only 1 day of AVHRR data.\\\', \\\'geospatial_lon_min\\\': 0.0, \\\'platform\\\': \\\'Ships, buoys, Argo floats, MetOp-A, MetOp-B\\\', \\\'platform_vocabulary\\\': \\\'Global Change Master Directory (GCMD) Platform Keywords\\\', \\\'geospatial_lon_resolution\\\': 0.25, \\\'source\\\': \\\'ICOADS, NCEP_GTS, GSFC_ICE, NCEP_ICE, Pathfinder_AVHRR, Navy_AVHRR\\\', \\\'date_modified\\\': \\\'2020-05-08T19:05:13Z\\\'}, \\\'coords\\\': {\\\'lon\\\': {\\\'dims\\\': (\\\'lon\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Longitude\\\', \\\'units\\\': \\\'degrees_east\\\', \\\'grids\\\': \\\'Uniform grid from 0.125 to 359.875 by 0.25\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (1440,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'original_shape\\\': (1440,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1440,), \\\'shuffle\\\': True}}, \\\'lat\\\': {\\\'dims\\\': (\\\'lat\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Latitude\\\', \\\'units\\\': \\\'degrees_north\\\', \\\'grids\\\': \\\'Uniform grid from -89.875 to 89.875 by 0.25\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (720,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'original_shape\\\': (720,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (720,), \\\'shuffle\\\': True}}, \\\'zlev\\\': {\\\'dims\\\': (\\\'zlev\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Sea surface height\\\', \\\'actual_range\\\': \\\'0, 0\\\', \\\'positive\\\': \\\'down\\\', \\\'units\\\': \\\'meters\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (1,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'original_shape\\\': (1,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1,), \\\'shuffle\\\': True}}, \\\'time\\\': {\\\'dims\\\': (\\\'time\\\',), \\\'attrs\\\': {\\\'long_name\\\': \\\'Center time of the day\\\'}, \\\'dtype\\\': \\\'datetime64[ns]\\\', \\\'shape\\\': (2,), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'zlib\\\': True, \\\'units\\\': \\\'days since 1978-01-01 12:00:00\\\', \\\'original_shape\\\': (1,), \\\'dtype\\\': dtype(\\\'float32\\\'), \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1,), \\\'shuffle\\\': True}}}, \\\'data_vars\\\': {\\\'err\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Estimated error standard deviation of analysed_sst\\\', \\\'valid_min\\\': 0, \\\'valid_max\\\': 1000, \\\'units\\\': \\\'Celsius\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}, \\\'ice\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Sea ice concentration\\\', \\\'valid_min\\\': 0, \\\'valid_max\\\': 100, \\\'units\\\': \\\'%\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}, \\\'sst\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Daily sea surface temperature\\\', \\\'valid_min\\\': -300, \\\'valid_max\\\': 4500, \\\'units\\\': \\\'Celsius\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}, \\\'anom\\\': {\\\'dims\\\': (\\\'time\\\', \\\'zlev\\\', \\\'lat\\\', \\\'lon\\\'), \\\'attrs\\\': {\\\'long_name\\\': \\\'Daily sea surface temperature anomalies\\\', \\\'valid_min\\\': -1200, \\\'valid_max\\\': 1200, \\\'units\\\': \\\'Celsius\\\'}, \\\'dtype\\\': \\\'float32\\\', \\\'shape\\\': (2, 1, 720, 1440), \\\'encoding\\\': {\\\'complevel\\\': 4, \\\'scale_factor\\\': 0.01, \\\'zlib\\\': True, \\\'original_shape\\\': (1, 1, 720, 1440), \\\'dtype\\\': dtype(\\\'int16\\\'), \\\'add_offset\\\': 0.0, \\\'fletcher32\\\': False, \\\'chunksizes\\\': (1, 1, 720, 1440), \\\'shuffle\\\': True}}}, \\\'dims\\\': {\\\'lon\\\': 1440, \\\'lat\\\': 720, \\\'zlev\\\': 1, \\\'time\\\': 2}, \\\'chunks\\\': {\\\'time\\\': {1: 1, 0: 1}}}" instruction_id: "bundle_34" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/Rechunk/FlatMap(split_fragment)" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py:34" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082473   nanos: 703398942 } message: "Combining group = ((\\\'time\\\', 0),), containing fragments = [({Dimension(name=\\\'time\\\', operation=<CombineOp.CONCAT: 2>): IndexedPosition(value=0, indexed=True, dimsize=2)}, <xarray.Dataset>\\nDimensions:  (time: 1, zlev: 1, lat: 720, lon: 1440)\\nCoordinates:\\n  * lat      (lat) float32 -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\\n  * lon      (lon) float32 0.125 0.375 0.625 0.875 ... 359.1 359.4 359.6 359.9\\n  * time     (time) datetime64[ns] 1981-09-01T12:00:00\\n  * zlev     (zlev) float32 0.0\\nData variables:\\n    anom     (time, zlev, lat, lon) float32 ...\\n    err      (time, zlev, lat, lon) float32 ...\\n    ice      (time, zlev, lat, lon) float32 ...\\n    sst      (time, zlev, lat, lon) float32 ...\\nAttributes: (12/37)\\n    title:                      NOAA/NCEI 1/4 Degree Daily Optimum Interpolat...\\n    source:                     ICOADS, NCEP_GTS, GSFC_ICE, NCEP_ICE, Pathfin...\\n    id:                         oisst-avhrr-v02r01.19810901.nc\\n    naming_authority:           gov.noaa.ncei\\n    summary:                    NOAAs 1/4-degree Daily Optimum Interpolation ...\\n    cdm_data_type:              Grid\\n    ...                         ...\\n    metadata_link:              https://doi.org/10.25921/RE9P-PT57\\n    ncei_template_version:      NCEI_NetCDF_Grid_Template_v2.0\\n    comment:                    Data was converted from NetCDF-3 to NetCDF-4 ...\\n    sensor:                     Thermometer, AVHRR\\n    Conventions:                CF-1.6, ACDD-1.3\\n    references:                 Reynolds, et al.(2007) Daily High-Resolution-...)]" instruction_id: "bundle_42" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/Rechunk/MapTuple(combine_fragments)" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py:169" thread: "Thread-13" \nWorker: severity: INFO timestamp {   seconds: 1698082473   nanos: 740993976 } message: "Combining group = ((\\\'time\\\', 1),), containing fragments = [({Dimension(name=\\\'time\\\', operation=<CombineOp.CONCAT: 2>): IndexedPosition(value=1, indexed=True, dimsize=2)}, <xarray.Dataset>\\nDimensions:  (time: 1, zlev: 1, lat: 720, lon: 1440)\\nCoordinates:\\n  * lat      (lat) float32 -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\\n  * lon      (lon) float32 0.125 0.375 0.625 0.875 ... 359.1 359.4 359.6 359.9\\n  * time     (time) datetime64[ns] 1981-09-02T12:00:00\\n  * zlev     (zlev) float32 0.0\\nData variables:\\n    anom     (time, zlev, lat, lon) float32 ...\\n    err      (time, zlev, lat, lon) float32 ...\\n    ice      (time, zlev, lat, lon) float32 ...\\n    sst      (time, zlev, lat, lon) float32 ...\\nAttributes: (12/37)\\n    title:                      NOAA/NCEI 1/4 Degree Daily Optimum Interpolat...\\n    source:                     ICOADS, NCEP_GTS, GSFC_ICE, NCEP_ICE, Pathfin...\\n    id:                         oisst-avhrr-v02r01.19810902.nc\\n    naming_authority:           gov.noaa.ncei\\n    summary:                    NOAAs 1/4-degree Daily Optimum Interpolation ...\\n    cdm_data_type:              Grid\\n    ...                         ...\\n    metadata_link:              https://doi.org/10.25921/RE9P-PT57\\n    ncei_template_version:      NCEI_NetCDF_Grid_Template_v2.0\\n    comment:                    Data was converted from NetCDF-3 to NetCDF-4 ...\\n    sensor:                     Thermometer, AVHRR\\n    Conventions:                CF-1.6, ACDD-1.3\\n    references:                 Reynolds, et al.(2007) Daily High-Resolution-...)]" instruction_id: "bundle_41" transform_id: "Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/Rechunk/MapTuple(combine_fragments)" log_location: "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py:169" thread: "Thread-14" \nWorker: severity: ERROR timestamp {   seconds: 1698082473   nanos: 883427143 } message: "Error processing instruction bundle_41. Original traceback is\\nTraceback (most recent call last):\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1417, in process\\n    return self.do_fn_invoker.invoke_process(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 837, in invoke_process\\n    self._invoke_process_per_window(\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 983, in _invoke_process_per_window\\n    self.process_method(*args_for_process, **kwargs_for_process),\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/transforms/core.py\\", line -1, in <lambda>\\n  File \\"/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py\\", line 89, in store_dataset_fragment\\n    _store_data(vname, da.variable, index, zgroup)\\n  File \\"/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py\\", line 32, in _store_data\\n    zarr_array = zgroup[vname]\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/zarr/hierarchy.py\\", line 500, in __getitem__\\n    raise KeyError(item)\\nKeyError: \\\'time\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py\\", line 284, in _execute\\n    response = task()\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py\\", line 357, in <lambda>\\n    lambda: self.create_worker().do_instruction(request), request)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py\\", line 597, in do_instruction\\n    return getattr(self, request_type)(\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py\\", line 635, in process_bundle\\n    bundle_processor.process_bundle(instruction_id))\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py\\", line 1003, in process_bundle\\n    input_op_by_transform_id[element.transform_id].process_encoded(\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py\\", line 227, in process_encoded\\n    self.output(decoded_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py\\", line 528, in output\\n    _cast_to_receiver(self.receivers[output_index]).receive(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py\\", line 240, in receive\\n    self.consumer.process(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py\\", line 908, in process\\n    delayed_applications = self.dofn_runner.process(o)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1419, in process\\n    self._reraise_augmented(exn)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1491, in _reraise_augmented\\n    raise exn\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1417, in process\\n    return self.do_fn_invoker.invoke_process(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 623, in invoke_process\\n    self.output_handler.handle_process_outputs(\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1581, in handle_process_outputs\\n    self._write_value_to_tag(tag, windowed_value, watermark_estimator)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1694, in _write_value_to_tag\\n    self.main_receivers.receive(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py\\", line 240, in receive\\n    self.consumer.process(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py\\", line 908, in process\\n    delayed_applications = self.dofn_runner.process(o)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1419, in process\\n    self._reraise_augmented(exn)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1507, in _reraise_augmented\\n    raise new_exn.with_traceback(tb)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 1417, in process\\n    return self.do_fn_invoker.invoke_process(windowed_value)\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 837, in invoke_process\\n    self._invoke_process_per_window(\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py\\", line 983, in _invoke_process_per_window\\n    self.process_method(*args_for_process, **kwargs_for_process),\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/transforms/core.py\\", line -1, in <lambda>\\n  File \\"/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py\\", line 89, in store_dataset_fragment\\n    _store_data(vname, da.variable, index, zgroup)\\n  File \\"/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py\\", line 32, in _store_data\\n    zarr_array = zgroup[vname]\\n  File \\"/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/zarr/hierarchy.py\\", line 500, in __getitem__\\n    raise KeyError(item)\\nKeyError: \\"time [while running \\\'Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/StoreDatasetFragments/Map(store_dataset_fragment)\\\']\\"\\n\\n" instruction_id: "bundle_41" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:288" thread: "Thread-14" \nWorker: severity: INFO timestamp {   seconds: 1698082474   nanos: 669620990 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082474   nanos: 670053005 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082474   nanos: 670234918 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082474   nanos: 670457124 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082474   nanos: 681385040 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082474   nanos: 681679964 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 68678855 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 68814039 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 68865060 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 68907022 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 69225072 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 69274187 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 391753911 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 391888856 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 391937971 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 391978979 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 393482923 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 393569946 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 752373933 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 752500057 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 752549171 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 752590179 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 753604173 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082475   nanos: 753678083 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 86351871 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 86504936 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 86554765 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 86595058 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 86934089 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 87002992 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 399111986 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 399282932 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 399332046 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 399379968 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 400151014 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 400223016 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 712072849 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 712205171 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 712254047 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 712296009 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 712883949 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082476   nanos: 712954998 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082477   nanos: 64671039 } message: "No more requests from control plane" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:261" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082477   nanos: 64780950 } message: "SDK Harness waiting for in-flight requests to complete" log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:262" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082477   nanos: 64831972 } message: "Closing all cached grpc data channels." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/data_plane.py:805" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082477   nanos: 64873933 } message: "Closing all cached gRPC state handlers." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:877" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082477   nanos: 65729141 } message: "Done consuming work." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py:274" thread: "MainThread" \nWorker: severity: INFO timestamp {   seconds: 1698082477   nanos: 65786123 } message: "Python sdk harness exiting." log_location: "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker_main.py:184" thread: "MainThread" \nTraceback (most recent call last):\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/bin/pangeo-forge-runner", line 8, in <module>\n    sys.exit(main())\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-runner/pangeo_forge_runner/cli.py", line 28, in main\n    app.start()\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-runner/pangeo_forge_runner/cli.py", line 23, in start\n    super().start()\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/traitlets/config/application.py", line 481, in start\n    return self.subapp.start()\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-runner/pangeo_forge_runner/commands/bake.py", line 277, in start\n    pipeline.run()\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/pipeline.py", line 574, in run\n    return self.runner.run_pipeline(self, self._options)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline\n    return runner.run_pipeline(pipeline, options)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 199, in run_pipeline\n    self._latest_run_result = self.run_via_runner_api(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 212, in run_via_runner_api\n    return self.run_stages(stage_context, stages)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 442, in run_stages\n    bundle_results = self._execute_bundle(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _execute_bundle\n    self._run_bundle(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 999, in _run_bundle\n    result, splits = bundle_manager.process_bundle(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1415, in process_bundle\n    for result, split_result in executor.map(execute, zip(part_inputs,  # pylint: disable=bad-option-value\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator\n    yield _result_or_cancel(fs.pop())\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel\n    return fut.result(timeout)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/concurrent/futures/_base.py", line 458, in result\n    return self.__get_result()\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n    raise self._exception\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/utils/thread_pool_executor.py", line 37, in run\n    self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1407, in execute\n    return bundle_manager.process_bundle(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1348, in process_bundle\n    raise RuntimeError(result.error)\nRuntimeError: Traceback (most recent call last):\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1417, in process\n    return self.do_fn_invoker.invoke_process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 837, in invoke_process\n    self._invoke_process_per_window(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 983, in _invoke_process_per_window\n    self.process_method(*args_for_process, **kwargs_for_process),\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/transforms/core.py", line -1, in <lambda>\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 89, in store_dataset_fragment\n    _store_data(vname, da.variable, index, zgroup)\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 32, in _store_data\n    zarr_array = zgroup[vname]\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/zarr/hierarchy.py", line 500, in __getitem__\n    raise KeyError(item)\nKeyError: \'time\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute\n    response = task()\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>\n    lambda: self.create_worker().do_instruction(request), request)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction\n    return getattr(self, request_type)(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle\n    bundle_processor.process_bundle(instruction_id))\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle\n    input_op_by_transform_id[element.transform_id].process_encoded(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded\n    self.output(decoded_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 528, in output\n    _cast_to_receiver(self.receivers[output_index]).receive(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 240, in receive\n    self.consumer.process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 908, in process\n    delayed_applications = self.dofn_runner.process(o)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1419, in process\n    self._reraise_augmented(exn)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1491, in _reraise_augmented\n    raise exn\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1417, in process\n    return self.do_fn_invoker.invoke_process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 623, in invoke_process\n    self.output_handler.handle_process_outputs(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1581, in handle_process_outputs\n    self._write_value_to_tag(tag, windowed_value, watermark_estimator)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1694, in _write_value_to_tag\n    self.main_receivers.receive(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 240, in receive\n    self.consumer.process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 908, in process\n    delayed_applications = self.dofn_runner.process(o)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1419, in process\n    self._reraise_augmented(exn)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1507, in _reraise_augmented\n    raise new_exn.with_traceback(tb)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1417, in process\n    return self.do_fn_invoker.invoke_process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 837, in invoke_process\n    self._invoke_process_per_window(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 983, in _invoke_process_per_window\n    self.process_method(*args_for_process, **kwargs_for_process),\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/transforms/core.py", line -1, in <lambda>\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 89, in store_dataset_fragment\n    _store_data(vname, da.variable, index, zgroup)\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 32, in _store_data\n    zarr_array = zgroup[vname]\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/zarr/hierarchy.py", line 500, in __getitem__\n    raise KeyError(item)\nKeyError: "time [while running \'Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/StoreDatasetFragments/Map(store_dataset_fragment)\']"\n\n', stderr=b'/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.10. You may encounter bugs or missing features.\n  warnings.warn(\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/xarray/core/dataset.py:2461: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs\n  return to_zarr(  # type: ignore[call-overload,misc]\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/xarray/core/dataset.py:2461: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs\n  return to_zarr(  # type: ignore[call-overload,misc]\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/xarray/core/dataset.py:2461: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs\n  return to_zarr(  # type: ignore[call-overload,misc]\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/xarray/core/dataset.py:2461: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs\n  return to_zarr(  # type: ignore[call-overload,misc]\nTraceback (most recent call last):\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1417, in process\n    return self.do_fn_invoker.invoke_process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 837, in invoke_process\n    self._invoke_process_per_window(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 983, in _invoke_process_per_window\n    self.process_method(*args_for_process, **kwargs_for_process),\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/transforms/core.py", line -1, in <lambda>\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 89, in store_dataset_fragment\n    _store_data(vname, da.variable, index, zgroup)\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 32, in _store_data\n    zarr_array = zgroup[vname]\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/zarr/hierarchy.py", line 500, in __getitem__\n    raise KeyError(item)\nKeyError: \'time\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute\n    response = task()\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>\n    lambda: self.create_worker().do_instruction(request), request)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction\n    return getattr(self, request_type)(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle\n    bundle_processor.process_bundle(instruction_id))\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle\n    input_op_by_transform_id[element.transform_id].process_encoded(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded\n    self.output(decoded_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 528, in output\n    _cast_to_receiver(self.receivers[output_index]).receive(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 240, in receive\n    self.consumer.process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 908, in process\n    delayed_applications = self.dofn_runner.process(o)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1419, in process\n    self._reraise_augmented(exn)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1491, in _reraise_augmented\n    raise exn\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1417, in process\n    return self.do_fn_invoker.invoke_process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 623, in invoke_process\n    self.output_handler.handle_process_outputs(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1581, in handle_process_outputs\n    self._write_value_to_tag(tag, windowed_value, watermark_estimator)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1694, in _write_value_to_tag\n    self.main_receivers.receive(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 240, in receive\n    self.consumer.process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py", line 908, in process\n    delayed_applications = self.dofn_runner.process(o)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1419, in process\n    self._reraise_augmented(exn)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1507, in _reraise_augmented\n    raise new_exn.with_traceback(tb)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 1417, in process\n    return self.do_fn_invoker.invoke_process(windowed_value)\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 837, in invoke_process\n    self._invoke_process_per_window(\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/common.py", line 983, in _invoke_process_per_window\n    self.process_method(*args_for_process, **kwargs_for_process),\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/transforms/core.py", line -1, in <lambda>\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 89, in store_dataset_fragment\n    _store_data(vname, da.variable, index, zgroup)\n  File "/Users/nrhagen/Documents/carbonplan/pangeo_forge/pangeo-forge-recipes/pangeo_forge_recipes/writers.py", line 32, in _store_data\n    zarr_array = zgroup[vname]\n  File "/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/zarr/hierarchy.py", line 500, in __getitem__\n    raise KeyError(item)\nKeyError: "time [while running \'Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/StoreDatasetFragments/Map(store_dataset_fragment)\']"\n\n/Users/nrhagen/micromamba/envs/pangeo-forge-recipes/lib/python3.10/site-packages/apache_beam/runners/worker/operations.py:908: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs\n  delayed_applications = self.dofn_runner.process(o)\n')

@norlandrhagen
Copy link
Contributor Author

norlandrhagen commented Oct 27, 2023 •
edited
Loading

Getting back to debugging this.
Some notes:

  • Failing tests in CI are now identical to failing local tests.
  • Integration tests fails for minio pytest tests/test_integration.py -k 'test_integration[noaa-oisst]' --run-integration.
  • Test fails in WriteToZarr.
  • ** Test fails in test_ds~**
  • raise KeyError(item)\nKeyError: "time [while running \'Create|OpenURLWithFSSpec|OpenWithXarray|StoreToZarr|Test dataset/StoreToZarr/StoreDatasetFragments/Map(store_dataset_fragment)\']
  • OpenWithXarray section returns proccode=0
  • Test is caching files against minio!
image
  • Cached files are valid xarray datasets:
image
  • test_dataset is not successfully reading the written zarr stores from minio!

@norlandrhagen
Copy link
Contributor Author

The minio integration tests are failing/passing inconsistently. When the integration tests are ran back-to-back. The results are inconsistent.

total
partial
pass

@norlandrhagen norlandrhagen marked this pull request as ready for review October 31, 2023 20:08
@norlandrhagen norlandrhagen changed the title [WIP] Add minio to Integration Tests Add minio to Integration Tests Oct 31, 2023
@norlandrhagen
Copy link
Contributor Author

Note: The inconsistent minio failures were remedied with a fix from @ranchodeluxe!

ranchodeluxe
ranchodeluxe approved these changes Oct 31, 2023
View reviewed changes
Copy link
Contributor

@ranchodeluxe ranchodeluxe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Just some small golfing comments

@norlandrhagen
Copy link
Contributor Author

Thanks for the review @ranchodeluxe!

cisaacstern
cisaacstern reviewed Nov 6, 2023
View reviewed changes
@norlandrhagen @cisaacstern
Update pyproject.toml
86479a5 
Co-authored-by: Charles Stern <[email protected]>
@norlandrhagen norlandrhagen merged commit a73d120 into pangeo-forge:main Nov 6, 2023
@norlandrhagen norlandrhagen deleted the minio branch November 6, 2023 22:54
@cisaacstern
Copy link
Member

cisaacstern commented Nov 6, 2023 •
edited
Loading

@norlandrhagen sorry I was just going to say, I think we need a follow-on to this that restores the

assert proc.returncode == 0

because it looks like at least one of the integration tests was failing silently here:

https://github.com/pangeo-forge/pangeo-forge-recipes/actions/runs/6777276247/job/18420542826#step:7:32

@cisaacstern
Copy link
Member

Actually both of the minio test cases are failing, here's the other one:

https://github.com/pangeo-forge/pangeo-forge-recipes/actions/runs/6777276247/job/18420542826#step:7:317

but without that assert they are registering to pytest as passes.

@cisaacstern cisaacstern mentioned this pull request Nov 6, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cisaacstern cisaacstern cisaacstern left review comments

@ranchodeluxe ranchodeluxe ranchodeluxe approved these changes

Assignees
No one assigned
Labels
test-integration Apply this label to run integration tests on a PR.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@norlandrhagen @cisaacstern @ranchodeluxe