Informing Observatory Operations with Accessible Telemetry and Performance Metrics

The operation of robotic telescopes at remote locations without supervision by local staff is a specific challenge for Las Cumbres Observatory's (LCOGT) global network, where more than 25 telescopes are operated at seven globally distributed sites. LCOGT personnel typically visits each site every 18 months for general maintenance and upgrade tasks, whereas minor failures are handled by local site support staff. LCOGT is making efforts to quantitatively inform decisions about scheduling preventative maintenance tasks and to provide tools to assist with failure diagnostic and response. For example, all night-time observations from all telescopes are analyzed to quantify the telescopes' throughput; these data, updated daily and looking back more than four years, are used to inform the cadence of telescope mirror washing and recoating. Further examples of utilizing every day's science data are presented, including monitoring the readnoise of more than 25 CCD cameras which has enabled the early detection of a failing CCD controller. Telemetry from all installations is collected in a no-SQL database (Opensearch)* and presented to various stakeholders via the visualization tools Kibana and Grafana(dagger). These utilities are used to diagnose problems in preprogrammed views, e.g., to detect acute issues or long-term degradation in the fleet of Cryotiger cooling systems. Telemetry data and performance metrics have always been collected at LCOGT. Making those data consistent, accessible, and easy to use for all stakeholders at LCOGT made the deluge of information usable in the daily observatory operation routine.