

\section{Introduction}
Constructing open source software (OSS) depends on contributors with a diverse set of skills. Each skill is vital to OSS: problem-solving skills allow software developers to build new features to address issues; organizational skills help software maintainers manage the moving parts of an OSS project; and communication skills enable writers to generate clear, concise documentation and facilitate collaboration. Unlike in software engineering, where programmers primarily write code, many OSS contributors provide equally valuable non-code contributions \cite{allcontributors2022, trinkenreich2020hidden}. While OSS-related skills include a subset of software engineering skills, contributors also work in contexts unique to OSS (e.g., wrangling contributors, identifying funding, consistently collaborating in a distributed form).

Despite the importance of skills related to OSS development, to our knowledge, there are no tools that currently exist which detect such skills. 
Related work in software engineering has developed techniques to detect specific software engineering skills, for example, Java programming skills \cite{bergersen2014construction} and general programming experience \cite{siegmund2014measuring}.
Meanwhile, other work detects programming-related skills by pulling data from version control systems such as GitHub\xspace \cite{papoutsoglou2019extracting, greene2016cvexplorer, montandon2019identifying, mockus2002icse, montandon2021mining, hauff2015matching}. Montandon et al. showed that this data could help identify experts in OSS communities \cite{montandon2019identifying} and predict technical roles of GitHub\xspace users \cite{montandon2021mining}. Other work has demonstrated that GitHub\xspace data can be used to extract skills for job recommendations \cite{greene2016cvexplorer, hauff2015matching}. %

These studies have significantly advanced the field, but focus largely on a single topic: mining technical programming skills. Thus, a gap still remains in mining OSS expertise, which encapsulates both soft and hard skills \cite{vadlamani2020studying} across many roles aside from software engineering. Rather than simply focusing on technical software development skills, our work focuses on mining GitHub\xspace data to detect both soft skills and hard skills, which are vital to OSS development.

In this paper, we introduce a method to detect OSS skills and implement it in a tool called \ADDED{\toolname (\textbf{D}etect\textbf{I}ng \textbf{SK}ills in \textbf{O}SS)}, with promising results. Our approach relies on identifying accurate \emph{signals}, which are measurable activities or cues associated with a skill used to identify the presence of having that skill. For example, having proficiency in a programming language could be measured using a certain number of lines of code written in that language \cite{bergersen2014construction}. The notion of signals follows prior work, such as Marlow et al., who identified cues that GitHub\xspace users utilized to judge a contributor's coding ability, project-relevant skills, and personality \cite{marlow2013activity}. 

We discuss how we identify relevant signals (Section \ref{sec:SkillsSignalsModel}) and outline \toolname's features (Section \ref{sec:ImplementingTool}). We then present an evaluation of \toolname and its results (Section \ref{sec:Evaluation}). Finally, we discuss the implications (Section \ref{sec:Discussion}) and next steps for this research (Section \ref{sec:FuturePlans}). 





\section{Identifying Signals for Skills}
\label{sec:SkillsSignalsModel}
\input{tables/skills-signals-table}
To develop \toolname, OSS skills and signals need to be identified. The first author extracted relevant \textbf{OSS skills} by reading prior literature on software engineering expertise \cite{baltes2018towards, ahmed2012evaluating, li2020distinguishes, papoutsoglou2017mining} and social factors of OSS \cite{dias2021makes, gousios2015work, marlow2013activity, steinmacher2015social, tsay2014influence}. These papers were selected to include a broad set of roles and contribution types. The first author read each manuscript and identified key skills. %

Next, we performed additional literature review to identify \textbf{signals} for each skill. In this work, \emph{signals are expressed as true or false statements} (i.e., the signal is either present or not). Potential signals were elicited through the nine papers to identify OSS skills \cite{dias2021makes, tsay2014influence, gousios2015work, steinmacher2015social, baltes2018towards, ahmed2012evaluating, li2020distinguishes, marlow2013activity, papoutsoglou2017mining} and relevant literature on mining OSS activity \cite{bergersen2014construction, hata2021github, chouchen2021anti, chatterjee2021aid, zhou2012make, lee2017one, eluri2021predicting, bao2019large, raman2020stress}. We read through each manuscript, identified potential signals for as many OSS skills as possible, and recorded the source of each signal. We finalized the signals from literature by converting each one as a true or false statement by defining thresholds (e.g., an activity happens at least $N$ times, an activity occurs with a frequency at least at the $N$th percentile). We then generated signals which seemed viable to compute and were representative of the skill. 
We designed our own signals for \skill{Is familiar with OSS practices} because it was cited often in literature \cite{ahmed2012evaluating, gousios2015work, steinmacher2015social}, but prior work did not define clear signals for this skill.


During this process, authors consulted three OSS community experts who made significant contributions to OSS and regularly work with OSS stakeholders for their profession. They provided insight on important skills and signals. Based on the experts' feedback and the frequency of the skills cited in literature, we reduced the final model to four skills, which is shown in Table \ref{tab:SkillsSignalsTable}.


\section{Automatically Detecting Skills}
\label{sec:ImplementingTool}
After developing a model of OSS skills and their associated signals (see Section \ref{sec:SkillsSignalsModel}), the first author implemented \toolname in Python to detect a contributor's OSS skills from the model based on GitHub\xspace user data. We use the GitHub\xspace GraphQL API \cite{github2021graphql}, GitHub\xspace REST API \cite{github2021rest}, and GHTorrent \cite{gousios2012ghtorrent} as sources of GitHub\xspace data. 


Our tool rates skills on a zero to five scale, where zero represents no proficiency and five represents a high level of proficiency. %
We use the function $rate(N, M)$ to determine skill level, where $N$ represents the number of signals present and $M$ represents the total number of signals a skill has. This function weights each signal equally.

\[ 
    rate(N, M) =
    \begin{cases} 
        0 & \tfrac{N}{M} = 0 \\
        1 & 0 < \tfrac{N}{M} \leq 0.2 \\
        2 & 0.2 < \tfrac{N}{M} \leq 0.4 \\
        3 & 0.4 < \tfrac{N}{M} \leq 0.6 \\
        4 & 0.6 < \tfrac{N}{M} \leq 0.8 \\
        5 & 0.8 < \tfrac{N}{M} \leq 1.0 \\
   \end{cases}
\]


\subsubsection*{Selecting programming languages} 
Programming languages that the tool detected were selected based on if they were present in both the 2020 State of the Octoverse \cite{github2020state} and the 2020 Stack Overflow Developer Survey \cite{stackoverflow2020developer}. This resulted in 9 programming languages: C, C\#, Java, JavaScript, PHP, Python, Ruby, Shell, and TypeScript.

\subsubsection*{\ADDEDZ{Excluding third-party libraries as} user contributions} 
Contributors often included code from third-party libraries in their commits, which has also been shown in prior work \cite{lopes2017dejavu}. \ADDED{Code from third-party libraries thus should be excluded from contributors' mined contributions.} To that end, we identify popular package managers for each programming language from a Wikipedia article on popular package managers \cite{wikipedia2021list}. We compile a list of installation folder names across all the package managers. Next, we analyze the top folder names for each programming language. We use a list of OSS for social good (OSS4SG) projects (i.e., OSS projects which address a societal issue and target a specific community) from Huang et al. \cite{huang2021leaving} We download the contents of each repository in this list and record its file tree. From this, we identify files written in the language based on file extensions and extract the path from the repository folder. We then retrieve the top 50 folder names associated with each programming language. For each language, we exclude the entire file tree under a folder name from the analysis when: 1) the folder name corresponds to an installation location of the programming language's package manager(s), or 2) the folder name is in the top 50 folder names for the programming language and the list of installation folder names across all package managers.

\subsubsection{Computing distributions} 
Some signals rely on the user activity being at a certain percentile and thus depends on an underlying distribution to compare to. To compute this, we generate a list of OSS contributors by recording the top 30 contributors \ADDEDZ{per project} from Huang et al.'s list of 437 OSS4SG projects~\cite{huang2021leaving}. We randomly select 500 GitHub\xspace users from this list. Each distribution is computed based on these 500 users' relevant activity for the distribution.

\section{Evaluation}
\label{sec:Evaluation}
\subsection{Design}
We designed a two-part survey to validate \toolname (see Section \ref{sec:ImplementingTool}):
\begin{enumerate}
    \item a Qualtrics survey where participants submitted anonymized responses and  
    \item a Microsoft Forms survey where participants submitted personal identifying information (PII). \ADDEDZ{This survey was displayed after the completion of the Qualtrics survey.}
\end{enumerate} \ADDEDZ{The survey was implemented in two parts so PII was linked only to a small number of questions.} Participants could choose to only take the Qualtrics survey \ADDEDZ{and not provide any PII}.
Topics in the Qualtrics survey included the importance of our tool's OSS skills and the participants' willingness to display their skills and ratings on GitHub\xspace. Topics in the Microsoft Forms survey included GitHub\xspace usernames and self-assessments of the skills from \toolname. 
Some survey questions are shown in Figure~\ref{fig:survey} and the full survey instrument is available as supplemental material~\cite{supplemental-materials}. 
We sent the survey to a subset of contributors who authored commits, opened issues and pull requests, or commented on others’ issues and pull requests from Huang et al.'s list of 1,079 OSS and OSS4SG projects \cite{huang2021leaving}. \ADDED{Our survey was sent to a total of 9,095 OSS contributors with a response rate of 5\%.} The Qualtrics survey received 455 responses. The Microsoft Forms survey received 386 responses, resulting in 316 valid usernames with public code contributions from merged pull requests. 
After completing the survey, participants could join a sweepstakes to win one of four \$100 Amazon.com gift cards.

\ADDEDZ{The GitHub\xspace usernames allowed us to compare the skill self-assessments with the skills scores computed by \toolname.}
To evaluate \toolname, we focus on its precision---that is, when the participant is confident they have a skill, does \toolname agree? We used two measurements of precision: 1) the precision of users who had self-evaluated skill levels greater than 0 (i.e., detecting the presence of a skill) and 2) the precision of users who had self-evaluated skill levels greater than 3 (i.e., detecting moderate to high skill proficiency).

For the analysis of the survey, we only look at close-ended questions and use standard statistical analysis techniques. We report percentages on how frequently participants agreed or strongly agreed with a statement and how frequently participants said a skill was important or very important. This follows Kitchenham's and Pfleeger's best practices to analyze survey data~\cite{survey-guidelines}.

\begin{figure}
    \begin{tcolorbox}[left=-14pt,right=2pt,top=2pt,bottom=2pt]
    {\sc \hspace{16pt}Part 1: Qualtrics Survey Questions (anonymous)}
    \begin{itemize}
        \item \questiontext{Q16}
        \item \questiontext{Q34}
        \item \questiontext{Q35}
    \end{itemize}
    \end{tcolorbox}
    \vspace{0.5\baselineskip}
    \begin{tcolorbox}[left=-14pt,right=2pt,top=2pt,bottom=2pt]
    {\sc \hspace{16pt}Part 2: Microsoft Forms Survey Questions}
    \begin{itemize}
        \item \ADDEDZ{\questiontext{MF5}}
        \item \questiontext{MF6}
        \item \questiontext{MF7}
        \vspace{0.5\baselineskip}
    \end{itemize}
    \vspace{-0.5\baselineskip}
    \end{tcolorbox}
    \caption{A subset of the survey questions. The complete survey instrument is in the supplemental materials~\cite{supplemental-materials}. }
    \label{fig:survey}
\end{figure}



\subsection{Preliminary Results}
\input{tables/skills-importance}
\input{tables/merged}


\subsubsection{Skill importance} All the detected skills were important to participants; the results are shown in Table \ref{tab:SkillsImportanceTable}. A majority of participants found the soft skills (\skill{Teaches others to be involved in the OSS project}, \skill{Shows commitment towards the OSS project}) to be important. Notably, participants found soft skills more important than hard skills. \skill{Is familiar with OSS practices} was also rated as important by a majority of participants, but was less important than the soft skills.


\subsubsection{Displaying skills} All detected skills would be displayed by a majority of participants (see Table \ref{tab:MergedTable}). Participants were most willing to share \skill{Is familiar with OSS practices}, \skill{JavaScript}, and \skill{Python}. \skill{PHP} was the least popular skill to share. Overall, there was variation in how willing participants were to share certain skills. In the free response, some participants were excited by sharing skills and suggested new languages to detect (e.g., Golang, Rust) or expressed encouragement. Others expressed concerns about the impact of \toolname, questioning whether it should be deployed.

\subsubsection{Tool accuracy} \toolname's performance is displayed in Table \ref{tab:MergedTable}. We find that \toolname identifies the presence of OSS skills with impressive performance, with precision scores ranging from 77\% (\skill{Ruby}) to 97\% (\skill{Is familiar with OSS practices}, \skill{Teaches others to be involved in the OSS project}). However, the overall performance of \toolname drops while detecting  moderately skilled to expert users. It is also notable that the rating distributions for \skill{Teaches others to be involved in the OSS project}, \skill{Shows commitment towards the OSS project}, and \skill{Is familiar with OSS practices} are skewed positively in the survey.

\section{Discussion}
\label{sec:Discussion}

Our evaluation shows encouraging results for \toolname. Participants are most excited to display programming language-related skills, which our tool detects with reasonable confidence. However, the high importance placed on soft skills by our participants should not be overlooked. This is especially relevant with emerging neural models that generate code with high quality, such as GitHub\xspace Copilot \cite{github2021copilot}. Given its importance, future versions of \toolname could be enhanced to more accurately detect soft skills.

\subsubsection*{Potential applications.} \ADDED{Our results indicate a potential future for skill detection approaches within OSS development and software engineering.} While skills underlie these activities, they are not widely supported in tooling. Skills detection methods such as ours could be applied in practice to transform existing experiences. For example, \toolname could assist project maintainers with OSS team formation by supporting a search engine for potential collaborators based on skills or automatically recommending contributors for particular OSS roles. Furthermore, \toolname's skill ratings could be used to identify a contributor's potential areas of growth or recommend mentors with the expertise for the contributor's professional goals. Skills could also be displayed publicly to GitHub\xspace profiles---users could self-select skills they were proficient in, while the platform could display a verified badge for detected skills.



\subsubsection*{Limitations.} One limitation of our evaluation is that self-rated skills is a biased measure, as participants may systematically overestimate or underestimate their skills. Additionally, limitations in describing skills in a survey may cause mismatched expectations of a skill. These may contribute to lower evaluation scores. For example, the signals for \skill{Is familiar with OSS practices} was designed to be simple to compute and beginner friendly, but participants rated themselves more harshly on the skill. Thus, when designing experiences with automatic skills detection, transparency in how the skill rating is computed is paramount and should be communicated clearly.




\section{Future Plans}
\label{sec:FuturePlans}
Our preliminary results indicate that there are some promising directions for \toolname. However, additional steps are required to improve upon our current approach, which we outline below.

\smallskip
\emph{Identifying additional signals from practitioners.} We hope to interview OSS contributors to understand how they evaluate their peers' expertise for the skills our tool detects. This could generate new signals to improve the accuracy of our tool. 258 participants have agreed to be interviewed for this work.

\smallskip
\emph{Defining weights of the signals.} In practice, each signal is not equally weighted in predicting a contributor's skill. Since our current approach does not support this, one next step could use a linear regression model to examine the relationship between signals and participants' self-evaluated skill level and then use the resulting model weights to weigh each signal in our tool. 

\smallskip
\emph{Evaluating the tool with other measures of skill.} We hope to run an additional evaluation using data sources less impacted by personal bias. For example, soft skills may be evaluated using peer evaluations, while hard skills may be evaluated using skill tests. Previous work has administered skill tests to determine Java expertise \cite{bergersen2014construction}.



\section{Conclusion}
We present \toolname, a tool to detect OSS-related skills which identifies \emph{signals} (i.e., measurable activities or cues associated with the skill) and then computes them from GitHub\xspace data. \toolname detects the following skills: \skill{Teaches others to be involved in the OSS project}, \skill{Shows commitment towards the OSS project}, \skill{Has knowledge in specific programming languages}, and \skill{Is familiar with OSS practices}. We demonstrate this approach yields positive results, as \toolname detects the presence of OSS skills with precision scores between 77\% to 97\%. Additionally, a near majority of participants find the detected skills important to OSS. We expect to improve the tool and perform more rigorous evaluations in the future. Future work could design tools to augment existing OSS experiences or improve upon our current approach. Our supplemental materials are publicly available at \cite{supplemental-materials}.

\begin{acks}
We thank our survey participants for their insight and Christian Bird, Mala Kumar, Victor Grau Serrat, and Lucy Harris for their feedback. Jenny T. Liang conducted this work for an internship at Microsoft Research's Software Analysis and Intelligence Group.
\end{acks}

\bibliographystyle{ACM-Reference-Format}




