FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems
CoRR(2024)
摘要
Postmortem analysis is essential in the management of incidents within cloud
systems, which provides valuable insights to improve system's reliability and
robustness. At CloudA, fault pattern profiling is performed during the
postmortem phase, which involves the classification of incidents' faults into
unique categories, referred to as fault pattern. By aggregating and analyzing
these fault patterns, engineers can discern common faults, vulnerable
components and emerging fault trends. However, this process is currently
conducted by manual labeling, which has inherent drawbacks. On the one hand,
the sheer volume of incidents means only the most severe ones are analyzed,
causing a skewed overview of fault patterns. On the other hand, the complexity
of the task demands extensive domain knowledge, which leads to errors and
inconsistencies. To address these limitations, we propose an automated
approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets.
It leverages hierarchy-guided contrastive learning to train a hierarchy-aware
incident encoder and predicts fault patterns with enhanced incident
representations. We evaluate FaultProfIT using the production incidents from
CloudA. The results demonstrate that FaultProfIT outperforms state-of-the-art
methods. Our ablation study and analysis also verify the effectiveness of
hierarchy-guided contrastive learning. Additionally, we have deployed
FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+
incidents from 30+ cloud services, successfully revealing several fault trends
that have informed system improvements.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要