On Recruiting Experienced GitHub Contributors for Interviews and Surveys on Prolific

Abstract

Software engineering researchers have been using general purpose online tools for crowd-sourcing for quite some time. Those tools can be useful to recruit participants for research studies as they are paid for their time. However, those tools should be used carefully. In this paper, we have described the issues we faced when recruiting participants on Prolific. We used Prolific to recruit open-source developers with experience in submitting and reviewing pull requests (PRs). However, we did not succeed in obtaining valid participants for either the interview or the survey, which led us to change the approach of our study and not use Prolific anymore. For example, a major issue we faced on Prolific was that even using the specific filter related to participants with software development knowledge, the screening study revealed the majority of them did not have any development knowledge. Based on this experience, we formulate recommendations for future research when using such crowd-sourcing tools.