Measuring ChatGPT’s Effect on Global Software Development

Researchers are using the GitHub Innovation Graph to quantify how AI tools like ChatGPT are changing software development. The longitudinal, country-level metrics in the dataset have enabled a new causal analysis of the technology’s impact on development velocity, presented at the Munich Summer Institute in May.

The work comes from Alexander Quispe, a junior researcher at the World Bank’s Digital Development Global Practice and a lecturer at PUCP, and Rodrigo Grijalba, a data scientist specializing in causal inference and AI-integrated systems. Their paper exploits the staggered availability of ChatGPT across countries — where it launched early versus where it was legally restricted — as a natural experiment.

Key Findings: Velocity, Not Volume

The analysis of Innovation Graph data at the country level finds that the arrival of ChatGPT significantly increased the number of Git pushes per 100,000 inhabitants. The effects on the raw number of repositories and unique developers were positive but not statistically significant.

The impact was uneven across programming languages, according to the researchers. High-level, popular languages such as Python and JavaScript showed significant increases in the count of unique developers. Domain-specific languages like HTML and SQL showed more varied or smaller effects.

Quispe interprets the overall pattern as an acceleration of existing workflows rather than an expansion of the developer base. If AI’s main contribution was to lower the barrier to entry, you would expect a pronounced bump in new repositories and new contributors. Instead, the signal points to faster iteration within established projects.

The Synthetic Difference-in-Differences Approach

The researchers relied on a synthetic difference-in-differences (SDID) method, as described by Arkhangelsky, Athey, Imbens, and Wager (2021). They also reported results from traditional difference-in-differences (DiD) and synthetic control (SC) methods, but preferred SDID for technical reasons.

Grijalba notes that the Innovation Graph’s aggregated, country-and-language structure fit well with these methods. It allowed a clean split between treatment and control groups, defined as countries where ChatGPT’s terms of service permitted use versus those where regulators blocked it. The data can be disaggregated further to explore other dimensions, but language was the most obvious way to look at heterogeneous adoption effects.

DiD typically relies on the assumption of parallel pre-treatment trends, which in this dataset did not hold between the treated and untreated groups. SC methods demand enough pre-treatment periods to match the untreated group precisely — here, only eleven quarters were available, which limits the robustness of that approach. Quispe chose SDID because it combines the strengths of both: it constructs a synthetic control while also weighting pre-treatment periods to account for the differences in their predictive power for post-treatment outcomes.

In the plots produced by the study, the treated and control groups are nearly indistinguishable in the quarters leading up to ChatGPT’s launch. They visibly diverge immediately afterward, and the divergence is interpreted as the treatment effect.

Event Studies and Counterfactual Comparisons

There is a conceptual distinction between the researchers’ country-comparison design and the event study format found in GitHub’s own transparency data. In an event study, one tracks a single group’s before-and-after trend versus the counterfactual projection of no event.

As an example, GitHub’s DMCA transparency data shows a 15x jump in circumvention claims coinciding with a change to the takedown form. In that specific case, the observed path after the event naturally extended the prior trendline, making the post-event data an accidental visualization of the counterfactual.

Quispe’s setup instead compares a treated group against an untreated group. This is useful when the counterfactual is not a simple extrapolation of the pre-treatment trend of the treated group, and it underpins the choice of DiD, SC, and SDID as comparison frameworks. In his words, it is the method for estimating whether the average country meets the criteria for being able to claim, after accounting for the parallel paths of a control group, that ChatGPT “caused Y to increase by X percent.”

Limitations and Open Questions

A methodological critique raised at the Munich workshop concerned the use of legally restricted countries as a control. Since VPN access could circumvent ChatGPT bans, treatment may have leaked into the control group. In his rebuttal, Quispe points to other research — Kreitmeir and Raschky (2023) and del Rio-Chanona et al. (2023) — that acknowledges such workarounds exist but insists that legal barriers still seriously slow adoption and, therefore, remain a valid identifying source.

For future work, the researchers want to go beyond country aggregates. Quispe hopes to analyze software developer-level administrative data, comparing productivity gains of Copilot subscribers versus non-users. He also sees value in user-experience metrics that would reveal heterogeneous outcomes among developers already using Copilot. He expects the integration of AI tools in development pipelines to deepen; whether and how policymakers deregulate access will help determine whether the gains documented in aggregate Git metrics translate into sustained broader productivity.

From Economics to Open-Source Causal ML

Both Alexander and Rodrigo came to causal machine learning through economics, but their paths illustrate different ways into the field. Alexander's starting point was a research associate role at MIT in early 2020, when he took a causal ML course taught by Victor Chernozhukov. Chernozhukov was looking for someone to translate tutorials from R to Python—a task that predated large language models. That translation work grew into a collaboration building Python and Julia software for causal ML methods.

The motivation was practical. Most advanced econometric techniques were written in Stata or R, which limited their adoption in industry. Microsoft's EconML was still early, and companies like Uber, Meta, and Amazon would only later develop their own Python packages. In 2022, Alexander moved to Stanford and worked with Professor Susan Athey on translating her Causal Inference and Machine Learning bookdown into Python. That process taught him about heterogeneous treatment effects, causal trees and forests, and the intersection of reinforcement learning and causal inference.

Rodrigo's path started in an undergraduate economics program, where a variable topic course on ML methods for social sciences caught his attention. The course was taught by Pavel Coronado, head of the AI laboratory for Social Sciences at the faculty. Coronado mentioned an upcoming graduate diploma program on ML and AI for Social Sciences, and Rodrigo joined before finishing his undergrad. The program covered object-oriented programming, neural networks, and big data—topics taught in part by Alexander. When Alexander later needed assistants with software development skills, Rodrigo joined the team.

Day-to-Day Work

Alexander splits his time between two roles. At the World Bank, he does econometric analysis on administrative data—particularly digital transfer data from South Asian countries like Pakistan, Bangladesh, and India—to analyze quasi-experiments. The other half of his time goes to leading a team called Dive Into Causal Machine Learning (D2CML), which Rodrigo is also part of. The team builds free causal ML packages in Python for industry and government use.

Current collaborations include work with Nathan Kallus (Cornell Tech/Netflix) on reinforcement learning and causal inference, with Damian Clarke (U. Chile) on the SDID algorithm, and with Pedro Sant'Anna on a difference-in-differences package for multiple time periods.

Rodrigo's work focuses mostly on D2CML projects, spanning teaching materials for undergraduates and graduates, AI applications, and academic research. He also does related freelance work and maintains some open source passion projects.

Generative AI's Impact

Generative AI tools have changed how both researchers approach their work. Before LLMs existed, Alexander's translation work from Stata and R to Python could take two to three months per package. With ChatGPT, that translation happens roughly twice as fast.

It works better for Python than for Julia's translation, Alexander notes. What he has learned is how to ask good questions: much of the work with panel datasets involves preprocessing large data, which requires being a precise prompt engineer.

He would like to try GitHub Copilot, believing it could optimize his algorithms more effectively than reviewing scripts manually with ChatGPT, which he finds tedious.

Rodrigo uses generative AI differently—as a starting point rather than a finishing tool. He prompts it with a topic and guidelines to get an overall structure, then works from there, even when the final result differs completely from the initial AI output. He also points to a generation of students now using these tools for assignments. Rather than banning them, he argues for familiarity: being a good prompt engineer, he says, will be an extremely useful skill in the near future.

Advice for Beginners

Both researchers have concrete recommendations for those starting out. Alexander emphasizes fundamentals:

  • Take an algorithms course, preferably in C++, since it provides a foundation for learning any language and is crucial for data processing and econometric implementation.
  • Master linear algebra—matrix decomposition and optimization methods make building packages much easier.
  • Learn Git and GitHub for version control and teamwork.
  • Take causal inference and machine learning courses, which are now accessible to undergraduates.
  • For graduate school, study reinforcement learning and its integration with causal inference, as tech companies increasingly use these techniques.

Rodrigo addresses the psychological side of starting out. The field can be overwhelming because of what you don't know, and people often fall into "optimization paralysis" trying to find the optimal path. His advice: sometimes it's enough to immerse yourself in the knowledge without forcing understanding—it comes naturally, like learning a language. And against analysis paralysis, he recommends a degree of impulsiveness: build an intuition and trust it.

For those interested in causal ML, both researchers recommend a specific set of books and tutorials:

  • The Book of Why by Judea Pearl
  • Causal Inference: The Mixtape by Scott Cunningham
  • The Effect by Nick Huntington-Klein
  • Applied Causal Inference Powered by ML and AI by Victor Chernozhukov et al.
  • Machine Learning and Causal Inference: A Short Tutorial by Guido W. Imbens and Susan Athey
  • Awesome Causal Inference, a curated list by Matteo Courthoud
  • Dive into Causal Machine Learning (the team's own site)

Rodrigo's current favorite is Applied Causal Inference Powered by ML and AI, and he also keeps an introductory econometrics textbook on hand—Jeffrey M. Wooldridge's Introductory Econometrics: A Modern Approach is his recommendation there.