Experience with Shifter Assistant - an intelligent tool to help operations of ATLAS TDAQ system in LHC Run 2

The Trigger and DAQ (TDAQ) system of the ATLAS experiment is a complex distributed computing system, composed of O(10, 000) of applications running on more than 2,500 computers. The system is operated by a crew of operators on shift. An important aspect of operations is to minimize the downtime of the system caused by runtime failures, such as human errors, unawareness or miscommunication. The paper describes recent developments in one of the intelligent TDAQ frameworks, the Shifter Assistant (SA) and summarizes the experience of it's use in operations of ATLAS during LHC Run 2. SA is a framework whose main aim is to automatize routine system checks, error detection and diagnosis, events correlations etc. in order to help the operators react to runtime problems promptly and effectively. The tool is based on CEP (Complex Event Processing) technology. It constantly processes this stream of operational events (O(100 kHz)) over a set of "directives" (or rules) in the knowledge base, producing human-oriented alerts and making shifters aware of the operational issues. More than 200 directives were developed by TDAQ and ATLAS detector experts for different domains. In this paper we also describe different types of directives which were developed in course of Run 2, and present few examples of most interesting and challenging ones, demonstrating the power of CEP for this type of applications.