Learning Risk Factors from App Reviews: A Large Language Model Approach for Risk Matrix Construction

Abstract Context. Analyzing mobile app reviews is essential for identifying trends and issue patterns that impact user experience and app reputation in app stores. A risk matrix provides a simple and intuitive way to prioritize software maintenance actions to reduce negative ratings. However, the manual construction of a risk matrix is time-consuming, and stakeholders work to understand the context of risks due to varied descriptions and review volume. Objective. There is a need for machine learning-based methods to extract risks and classify their priority. Existing studies have automated risk matrix generation in software development but have not explored app reviews or utilized Large Language Models (LLMs). Method. To address this gap, we propose using recent LLMs, specifically the OPT model, to automatically construct a risk matrix by extracting information from app reviews, such as features and bugs. We conduct experimental evaluations using reviews from eight mobile apps, generating risk matrices and comparing them with annotated reference matrices. Results. Results demonstrate that OPT models generate competitive risk matrices with proper prompt optimization. Conclusions. Our contributions include a dynamic and automatic prompt generation approach for customized instructions, allowing accurate and automated review analysis. We also develop instructions to identify risk severity using zero-shot learning. Additionally, we evaluate how OPT models compare to proprietary language models like GPT, showing the feasibility of LLMs in resource-constrained and sensitive contexts. This study represents a significant step toward improving software maintenance and feature prioritization. Mathematics Subject Classification (2020) MSC 68T07 · MSC 68T50 · MSC 68N01 · MSC 68T35