hFits: From Storing Metadata to Publishing ESO Data

The ESO Archive holds ca. 20 million FITS files: raw observations taken at the La Silla Paranal Observatory in Chile, data from APEX and UKIRT(WFCAM) telescopes, pipeline-processed data generated by the Quality Control and Data Processing Group(1) in ESO Garching and, since recently, reduced data delivered by the PI's through the ESO Phase 3 infrastructure(2). A metadata repository has been developed at the ESO Archive(3) (Dobrzycki et al. 2007), (Vera et al. 2011), to hold all the FITS file headers with up-to-date information using data warehouse technology. Presently, the repository contains more that 10 billion keywords from headers of all ESO FITS files. We have added to the repository a mechanism for keeping track of header ingestion and modification, allowing to build incremental applications on top of it. The aim is to provide a framework allowing for creation of fast and good quality metadata query services. We present hFits, a tool for data publishing allowing for metadata enhancement. The tool reads from the metadata repository and inserts the metadata into the conventional relational database systems using a simple configuration framework. It utilises the metadata repository tracking mechanism to incrementally refresh the services and transparently propagate any metadata updates. It supports the use of user defined functions where, for example, WCS coordinates can be calculated or related metadata can be extracted from other information systems, and for each new header file it provides to the archived file the access attributes following ESO data access policy, publishing the data to the community.