Study Finds Auto-Fix Tool Gets More Programmers to Upgrade Code

Failure to make necessary upgrades to software code can have dire consequences, such as the major data breach at Equifax. A recent study finds that auto-fix tools are effective ways to get programmers to make the relevant upgrades – if programmers opt to use them.

“Most software programs rely, in part, on code in external ‘libraries’ to perform some of their functions,” says Chris Parnin, an assistant professor of computer science at North Carolina State University and senior author of a paper on the work. “If those external libraries are modified to address flaws, programmers need to update their internal code to account for the changes. This is called ‘upgrading an out-of-date dependency.’ However, for various reasons, many programmers procrastinate, putting off the needed upgrades.

“This is what happened at Equifax,” Parnin says. “An external library they relied on had made public that it contained a security flaw. And while the external library was patched, Equifax never got around to updating its internal code. So months after the problem was identified, Equifax was still vulnerable and got hacked.

“Our goal with this project was to assess tools designed to get more programmers to upgrade their out-of-date dependencies. Could they help prevent another Equifax?”

For this study, the researchers looked at thousands of open-source projects on GitHub, an online programming community that fosters collaboration on open-source software projects. Specifically, the researchers looked at different means projects used to incentivize or facilitate upgrades and whether those incentives made any difference.

One group consisted of 2,578 projects that utilized automated pull requests, which notified project owners of needed upgrades to out-of-date dependencies, proposed potential code changes, and ran a small battery of tests to determine if the replacement code was viable. These project owners were still required to approve the changes or modify updated code if it failed initial viability tests.

A second group consisted of 1,273 projects that did not utilize incentives to upgrade out-of-date dependencies.

The researchers found that projects with automated pull requests made 60 percent more of the necessary upgrades than projects that didn’t use incentives.

“We also found that the majority of automated pull request projects were using the most up-to-date versions of dependent software, whereas the unincentivized projects were all over the map,” Parnin says. “The take-home message here is that we have automated tools that can help programmers keep up with upgrades. These tools can’t replace good programmers, but they can make a significant difference. However, it’s still up to programmers to put these tools in place and make use of them.”

The paper, “Can Automated Pull Requests Encourage Software Developers to Upgrade Out-of-Date Dependencies?”, will be presented at the IEEE/ACM International Conference on Automated Software Engineering, Oct. 30-Nov. 3 at the University of Illinois at Urbana-Champaign, Ill. Lead author of the paper is Samim Mirhosseini, an undergraduate at NC State. Mirhosseini’s work on the project was supported by a Research Experience for Undergraduates grant from the National Science Foundation.

-shipman-

Note to Editors: The study abstract follows.

“Can Automated Pull Requests Encourage Software Developers to Upgrade Out-of-Date Dependencies?”

Authors: Samim Mirhosseini and Chris Parnin, North Carolina State University

Presented: IEEE/ACM International Conference on Automated Software Engineering, Oct. 30-Nov. 3 at the University of Illinois at Urbana-Champaign, Ill.

Abstract: Developers neglect to update legacy software dependencies, resulting in buggy and insecure software. One explanation for this neglect is the difficulty of constantly checking for the availability of new software updates, verifying their safety, and addressing any migration efforts needed when upgrading a dependency. Emerging tools attempt to address this problem by introducing automated pull requests and project badges to inform the developer of stale dependencies. To understand whether these tools actually help developers, we analyzed 7,470 GitHub projects that used these notification mechanisms to identify any change in upgrade behavior. Our results find that, on average, projects that use pull request notifications upgraded 1.6x as often as projects that did not use any tools. Badge notifications were slightly less effective: users upgraded 1.4x more frequently. Unfortunately, although pull request notifications are useful, developers are often overwhelmed by notifications: only a third of pull requests were actually merged. Through a survey, 62 developers indicated that their most significant concerns are breaking changes, understanding the implications of changes, and migration effort. The implications of our work suggests ways in which notifications can be improved to better align with developers’ expectations and the need for new mechanisms to reduce notification fatigue and improve confidence in automated pull requests.