Summer 2025: Evaluating the Performance, Usage, and Challenges of Coding Agents in Open Source Software Development

Affiliations: College of Engineering and Computer Science
Team Leader:
Atish Kumar Dipongkor
at308343@ucf.edu
Computer Science PhD
Faculty Mentor:
Kevin Moran, PhD
Team Size:
4
Open Spots: 4
Team Member Qualifications:
Python, VS code, GitHub, GitLab
Description:
The rise of AI-powered coding agents, such as GitHub Copilot, OpenAI Codex, and Devstral, has transformed software development by automating tasks like code generation, bug fixing, and documentation. These agents, integrated into development environments like IDEs and version control systems, aim to enhance developer productivity by handling repetitive or well-defined tasks. However, their real-world performance, the scope of tasks they are used for, and their limitations remain underexplored, particularly in open source repositories where diverse, complex, and collaborative development occurs. This project aims to systematically collect and analyze pull requests (PRs) and commits assisted by coding agents in open source repositories to evaluate their performance, identify the tasks developers delegate to these agents, and uncover current challenges. The findings will provide insights into the efficacy of coding agents and guide future improvements in agentic software development.