Storage growth mitigation through data analysis ready climate datasets using HDF5 Virtual Datasets

<p>Climate datasets are usually provided in separate files that facilitate dataset management in climate data distribution systems. In ESGF¹ (Earth System Grid Federation) a time series of a variable is split into smaller pieces of data in order to reduce file size. Although this enhances usability for data management in the ESGF distribution system (i.e. file publishing, download, &#8230;), this demotes usability for data analysis. Usually, workflows need to pre-process and rearrange multiple files as a single data source, in order to obtain a data analysis dataset, involving data rewriting and duplication with the corresponding storage growth.</p><p>The mitigation of storage growth can be achieved by creating virtual views, allowing a number of actual datasets to be multidimensionally mapped together into a single multidimensional dataset that does not require rewriting data nor to consume additional storage. Due to the increasing interest in offering to climate researchers appropriate single data analysis datasets, some mechanisms have been or are being developed to tackle this issue, such as NcML (netCDF Markup Language), xarray/netCDF-4 Multiple File datasets and H5VDS. HDF5 Virtual Datasets³ (H5VDS) provide researchers with different views of interest of a compound dataset, without the cost of duplicating information, facilitating data analysis in an easy and transparent way.</p><p>In the climate community and in ESGF, netCDF is the standard data model and format for climate data exchange. netCDF-4 default storage format is HDF5, introducing into the netCDF library features from HDF5. This includes chunking², compression², virtual datasets and many other capabilities. H5VDS introduces a new dataset storage type that allows a number of multiple HDF5 (and netCDF-4) datasets to be mapped together into a single sliceable dataset via an interface layer. The datasets can be mixed in arbitrary combinations, based on range selection mapping to range selection on sources. This mapping allows mapping between different data types and to add, remove or modify existing metadata (i.e. datasets attributes), which usually it&#8217;s a common issue to access the data.&#160;</p><p>In this work, examples of applications of H5VDS features are applied to CMIP6 climate simulations datasets from ESGF, in order to provide data analysis ready virtual datasets. Examples of common tools/libraries (i.e. netcdf-c, xarray, nco, cdo, &#8230;) illustrate the convenience of the proposed approach. Using H5VDS facilitates data analysis workflows by enabling climate researchers to focus on data analysis rather than data engineering tasks. Also, since the H5VDS is created at the storage layer, these datasets are transparent to the netCDF-4 library and existing applications can benefit from this feature.</p><p><strong>References</strong></p><div> <div> <div>[1] L. Cinquini <em>et al.</em>, &#8220;The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data,&#8221; <em>Future Generation Computer Systems</em>, vol. 36, pp. 400&#8211;417, 2014, doi: 10.1016/j.future.2013.07.002. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0167739X13001477. [Accessed: 16-Jan-2020]</div> <div>[2] The HDF Group, &#8220;Chunking in HDF5&#8221;, 11-Feb.-2019. [Online]. Available: https://portal.hdfgroup.org/display/HDF5/Chunking+in+HDF5. [Accessed: 12-Jan.-2022]</div> </div> </div><p>[3] The HDF Group, &#8220;Virtual Dataset VDS&#8221;, 06-Apr.-2018. [Online]. Available: https://portal.hdfgroup.org/display/HDF5/Virtual+Dataset++-+VDS. [Accessed: 12-Jan.-2022]</p><p><strong>Acknowledgements</strong></p><p>This work it&#8217;s been developed under support from IS-ENES3 which is funded by the European Union&#8217;s Horizon 2020 research and innovation programme under grant agreement No 824084.</p>