An exploratory study of software artifacts on GitHub from the lens of documentation

Context: The abundance of software artifacts in open-source repositories has been analyzed by researchers from many perspectives, to address challenges in downstream tasks such as bug localization, code clone detection and so on. However, there is limited exploration of artifacts such as pull-requests and issues from a documentation perspective. Objective: We aim to explore the presence of information useful for documentation in different sources within the software projects. We present an exploratory analysis of 1.38M artifacts extracted from 950 GitHub repositories that analyses the content present in multiple software artifacts from a documentation perspective. Method: We arrive at a list of documentation types and sources through card-sorting and a developer survey. We apply topic modeling on the data extracted from 1.38M software artifacts based on these lists and study the extent of documentation-related information present in the software artifacts. The exploratory analysis of the artifacts listed is consolidated into the 'DocMine' dataset that comprises 50.63M textual sentences spanning across repositories written in four different programming languages. Results: We observe that about 28.1% of content extracted from the artifacts contains information related to features and modifications of the project at a higher level, and that pull-requests and issues comprise 18.26% and 17.85% of the extracted information. Conclusion: The presence of information about the projects in pull-requests and issues indicates immense scope in analyzing and processing multiple software artifacts for the purposes of generating software documentation and beyond. We envision that this study could open up a new line of research in software documentation.