Not Surveying Developers and Code About Languages

We present cross-sectional analyses of programming language use and reflect upon our experience in doing so. In particular, we directly survey 1500-13000 developers and indirectly do so by mining 200000 repositories. Our analysis reveals programming language adoption phenomena surrounding developer age, birth year, work place, opinions, gender, and choice of software repository. We find that survey methods are increasingly accessible and relevant, but there are widespread problems in examining developers and code in particular. Prominently, we show that analyzing software repositories suffers from the same sample bias expected from directly polling developers. These problems question, for example, the scope of results of the increasingly common research practice of mining software. We conclude by describing concrete practices and opportunities for amplifying results of developer and code surveys.