Data Management At the UKIRT and JCMT

Abstract

For more than a decade the Joint Astronomy Centre has been developing software tools to simplify observing and make it possible to use the telescopes in many different operational modes. In order to support remote operations the data handling systems need to be in place to allow observation preparation, flexible queue scheduling, data quality pipelines and science archives all to be connected in a data-driven environment. We discuss the history of these developments at UKIRT and JCMT and how the decision to combine software development at both telescopes led each to get features that they could not have justified if they were treated independently.

An overview of the planned CCAT software system

“…The JCMT-OT is the sub-millimeter ground-based telescope version of the generic JAC-OT written by the Joint Astronomy Centre that also supports the United Kingdom Infrared Telescope (UKIRT). [26][27][28] It supports the HARP heterodyne receiver 29 and the SCUBA-2 bolometer array, 30 which are similar to CHAI and SWCam.…”

ORAC-DR: A generic data reduction pipeline infrastructure

“…All information required to reduce the data correctly must be available in the metadata of the input data files. This requires a systems engineering approach to observatory operations where the metadata are treated as equals to the science pixel data (see e.g., Jenness and Economou, 2011, for an overview of the JCMT and UKIRT approach) and all observing modes are designed with observation preparation and data reduction in mind. An overview of the pipeline process is show in Fig.…”

Learning from 25 years of the extensible N-Dimensional Data Format

“…In 2009, it was decided that the inconvenience of this "single line of descent" approach to history was no longer justified by the savings in disk space and processing time, and so an alternative system was provided that enables each NDF to retain full information about all ancestor NDFs, and the processing that was used to create them (Jenness et al, 2009;Jenness and Economou, 2011). Thus each NDF may now contain a full "family tree" that goes back as far as any NDFs that have no recorded parents, or which have been marked explicitly as "root" NDFs.…”