index.org

Tools for analysis of geospatial time series data.

Background

• Remote sensing of soil moisture
• Time series based retrieval algorithm
• Temporal dynamics are often more important for analysis than spatial distribution
• Grew out of soil moisture validation

file:http://www.esa-soilmoisture-cci.org/sites/default/files/images/soilmoisture_logo_small.png

Data preparation for time series analysis

digraph G {
                resolution=100
                rankdir=LR
                bgcolor=Transparent
                fontcolor=White
                
                node [style="filled", color=White, fontname="Hack" ,shape=box] //All nodes will this shape and colour
                edge [color=Gray] //All the lines look like this
                
                data [label="RS data" shape=Mrecord]
                resamp [label="Resampling"]
                reproj [label="Reprojection"]
                ts [label="Temporal resampling", color="#888888"]
                store [label="Storage as DatasetTs" shape=Mrecord]

                data->{resamp reproj}->ts->store
                {rank=same; ts store}
}

Typical workflow for time series analysis

digraph G {
                resolution=100
                rankdir=UD
                bgcolor=Transparent
                fontcolor=White
                
                node [style="filled", color=White, fontname="Hack" ,shape=box] //All nodes will this shape and colour
                edge [color=Gray] //All the lines look like this
                
                data [label="DatasetTs" shape=Mrecord]
                data1 [label="DatasetTs" shape=Mrecord]
                spatial [label="Spatial matching"]
                dataprep [label="Data preparation"]
                algo [label="Algorithm"]
                store [label="Storage of results" shape=Mrecord]

                {data1}->spatial 
                {data}->spatial 
                spatial->dataprep 
                dataprep->algo 
                algo->store 
                {rank=same; spatial algo}
}

Spatial referencing

• Grid systems
  • UTM system
  • Equi7 system [cite:bauer2014optimisation]
• Discrete Global grids
  • SMOS grid - Icosahedron Snyder Equal Area (ISEA)
  • Gaussian Grids - used by ECMWF

pygeogrids

• is a package for managing discrete (global) grids
• has a notion of subdivision that can correspond to I/O units (e.g. netCDF files)

How to store time series

• Resample to data cube
• Store as compressed sparse arrays
• Climate and Forecast (CF) Metadata conventions http://cfconventions.org/

CF - Orthogonal multidimensional

file:./img/CF-DSG-orthogonal-multidim.svg

CF - Incomplete multidimensional

file:./img/CF-DSG-incomplete-multidim.svg

CF - Contiguous ragged

file:./img/CF-DSG-contigous-ragged.svg

CF - Indexed ragged

file:./img/CF-DSG-indexed-ragged.svg

Interface for data I/O

defines abstract base classes for a common interface

read_ts(gpi)
read_ts(lon, lat)
iter_ts()
read_img(datetime)
iter_img(startdate, enddate)

Implemenation for netCDF CF

• implements the interface for netCDF files according to the CF conventions
• Works but writing of data can be optimized
• integration with general purpose packages (xray)

Dataset that knows itself

• Can traverse itself and return a time series per grid point
• Optimized I/O because dataset object knows how data is stored

digraph structs {
                fixedsize=true;
                resolution=150
                bgcolor=Transparent
                
                node [style="filled" fillcolor=White, color=Black, fontname="Hack" ,shape=box] //All nodes will this shape and colour
                edge [color=Gray] //All the lines look like this
                
                data [label="{DatasetTs|.grid\n.read_ts()\n.iter_ts()}" shape=Mrecord]
}

Algorithms and processing

• <img src=”pytesmo file:./img/GitHub-Mark-Light-64px.png” alt=”pytesmo file:./img/GitHub-Mark-Light-64px.png” /> implements temporal matching, filtering, and algorithms specifically needed for soil moisture validation e.g. triple collocation.
• Framework for time series based computations parallelized via ipyparallel (formerly IPython parallel)
• I/O for soil moisture RS and in-situ datasets

Example - global validation

ecmwf_reader = ECMWF.ERA_Interim()
ascat_reader = AscatH25_SSM()

datasets = {'ASCAT': {'class': ascat_reader,
                      'columns': ['sm'],
                      'type': 'reference'},
            'ERAINT': {'class': ecmwf_reader,
                       'columns': ['soilm-l1'],
                       'type': 'other',
                       'kwargs': {'period': period},
                       'grids_compatible': False,
                       'use_lut': True, 'lut_max_dist': 100000}}

process = Validation(datasets=datasets,
                     temporal_matcher=BasicTemporalMatching(),
                     scaling='lin_cdf_match', 
                     metrics_calculator=BasicMetrics(),
                     period=period)
jobs = process.get_processing_jobs()
return process, jobs

Where do the packages fit in

digraph G {
                resolution=100
                rankdir=UD
                bgcolor=Transparent
                fontcolor=White
                nodesep=1.0
                
                node [style="filled", color=White, fontname="Hack" ,shape=box] //All nodes will this shape and colour
                edge [color=Gray fontname="Hack" fontcolor="#42affa"] //All the lines look like this
                
                data [label="DatasetTs" shape=Mrecord]
                data1 [label="DatasetTs" shape=Mrecord]
                spatial [label="Spatial matching"]
                dataprep [label="Data preparation"]
                algo [label="Algorithm"]
                store [label="Storage of results" shape=Mrecord]

                {data1}->spatial [label=""]
                {data}->spatial [label="pynetCF"]
                spatial->dataprep [label="pygeogrids"]
                dataprep->algo [label="pytesmo"]
                algo->store [label="pynetCF"]
                {rank=same; spatial algo}
}

Summary

• Suite of Python packages for time series optimized processing
• <img src=”pytesmo file:./img/GitHub-Mark-Light-64px.png” alt=”pytesmo file:./img/GitHub-Mark-Light-64px.png” />
• <img src=”pygeogrids file:./img/GitHub-Mark-Light-64px.png” alt=”pygeogrids file:./img/GitHub-Mark-Light-64px.png” />
• <img src=”pygeobase file:./img/GitHub-Mark-Light-64px.png” alt=”pygeobase file:./img/GitHub-Mark-Light-64px.png” />
• <img src=”pynetCF file:./img/GitHub-Mark-Light-64px.png” alt=”pynetCF file:./img/GitHub-Mark-Light-64px.png” />
• BSD licensed

Thank you, Questions?

• Get in touch
  • You can find this presentation at http://cpaulik.github.io/2015-eoscience2.0
  • [email protected]
• Thanks to the Python open source community.

References

bibliographystyle:plain bibliography:/home/cp/Dropbox/Arbeit/Papers/bibliography.bib