Online Reconstruction of Structural Information from Datacenter Logs.
EuroSys(2017)
摘要
Well-run datacenter application architectures are heavily instrumented to provide detailed traces of messages and remote invocations. Reconstructing user sessions, call graphs, transaction trees, and other structural information from these messages, a process known as sessionization, is the foundation for a variety of diagnostic, profiling, and monitoring tasks essential to the operation of the datacenter. We present the design and implementation of a system which processes log streams at gigabits per second and reconstructs user sessions comprising millions of transactions per second in real time with modest compute resources, while dealing with clock skew, message loss, and other real-world phenomena that make such a task challenging. Our system is based on the Timely Dataflow framework for low latency, data-parallel computation, and we demonstrate its utility with a number of use-cases and traces from a large, operational, mission-critical enterprise data center.
更多查看译文
关键词
Sessionization, Trace Trees, Resource Attribution, Streaming Log Analytics, Data Parallelism
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络