Changes were arriving faster than CI could verify them. They also tended to be larger, and several could be ready during the roughly 20 minutes it took CI to verify one of them. Once one landed, the others often had to be rebased and run through CI again. Waiting created overlapping work, overlapping work created rebases, and those rebases created more CI work.

One engineer with coding agents was enough to make CI a bottleneck I had previously associated with much larger engineering organizations.

I addressed it by reducing how much CI each change triggered. By selecting only the tests a change could affect, many changes went from occupying the runner for around 20 minutes to taking a couple of minutes or skipping the suite entirely.

Why more CI capacity wasn’t the answer

At first, I paid for more CI capacity. I burned through GitHub Actions’ free minutes quickly, and the bill went from about $10 to $20 to $50 a month. It was still growing. I didn’t want CI costs to scale with the amount of work the agents produced.

I moved the workload to a dedicated runner on a generic VM provider and deliberately limited myself to one machine. One VM had enough total compute. A 20-minute suite was manageable overnight, when agents could take multiple turns without me waiting for each result. The problem was the 20-minute feedback loop during active development.

I could have added more runners, but I’d seen that approach at much greater scale at Shopify. More runners increase throughput, but every change still triggers the same amount of work. As the rate of changes grows, the required capacity grows with it.

I decided to reduce the work each change triggered instead.

Selecting only the tests a change can affect

The approach was one we had also used at Shopify: instead of running the entire test suite for every change, figure out which tests the change could affect and run only those.

I combine two pieces of information. Full test runs record which parts of the application each test uses. When a change enters CI, a TypeScript dependency graph shows which parts of the application the changed files can affect. CI combines those two sets of information to select the relevant tests.

Most of that work does not happen on every CI run. Scheduled full-suite runs collect the runtime information, while the TypeScript dependency graph takes only a few seconds to rebuild. A localized change may select only a handful of tests. A broadly shared change can still select most or all of the suite.

At Shopify, much of the codebase was Ruby, so we relied more heavily on runtime tracing to understand what each test touched. At that scale, collecting and processing those traces was expensive enough that we spent significant effort making it faster. TypeScript makes part of this much cheaper because the compiler can tell me how source files depend on one another without running the application.

The selection is deliberately conservative. If CI sees a changed source file that it cannot confidently connect to existing test coverage, it records that uncertainty instead of assuming the change is safe.

I still run the complete suite every night. Those runs also refresh the runtime information used for future test selections. This keeps the development loop fast while the nightly run provides broader coverage without blocking me.

Test selection does not guarantee that a skipped test could never fail. It avoids spending 20 minutes rerunning unrelated tests when CI has enough information to select a much smaller set.

Making test selection useful

Test selection only works well if the codebase has useful boundaries. If every part of the application depends on everything else, an accurate selector will still conclude that a small change can affect most of the test suite.

In this application, each page has a route file, and each group of backend operations has an API file. Larger parts of the application are also separated into packages. These boundaries give the selector useful connection points between a source change and the tests that cover it.

Some changes naturally cross those boundaries. Shared test setup, global styles, build configuration, and dependency changes can affect much of the application, so those changes still select most or all of the suite.

The selector also exposed weak boundaries in the codebase. If a small change consistently selected a large part of the test suite, too much code usually depended on the area being changed. Tightening those boundaries reduced the number of selected tests. It also helped with concurrent agent work because changes in separate parts of the system were less likely to overlap and require rebasing.

I also found work that looked parallel in the CI configuration but was serial in practice. Two E2E jobs competed for the same runner, so they repeated setup without reducing wall time. I combined them into one job that builds the application and prepares a local database once. Within that job, state-changing tests remain serial, while isolated visual tests can run at the same time.

Now the order of operations is: skip tests that cannot be relevant, make the remaining work cheaper, improve code boundaries when small changes still select too much, and only then consider adding runners.

The aim is for the cost of verification to match the scope of the change.

What changed in practice

The changes helped even in the worst case, when the selector still chose the full pull-request suite.

Metric                       Before    After     Reduction
---------------------------------------------------------
Total CI wall time           ~21 min   ~13 min   ~40%
E2E lane                     ~12 min   ~4 min    ~65%
Browser-test execution       ~10 min   <3 min    ~70%
Critical-path contention     ~14 min   ~6 min    ~60%

The selector itself adds about 15 seconds.

The slowest runs improved much more. Across successful pull-request runs that started E2E, p95 E2E time fell from about 23 minutes to 6 minutes, a 74 percent reduction. Including queue time, p95 total CI time fell from about 7 hours 35 minutes to 35 minutes, a 92 percent reduction.

The 92 percent reduction mostly came from queueing, not faster tests. Some pre-change runs spent hours waiting for the single runner. Near capacity, queueing is nonlinear: a modest reduction in the work each change requires can free enough runner capacity to reduce wait times by much more than the test-time improvement itself.

These are historical before-and-after measurements, not a controlled benchmark. They show two different effects: less unnecessary work improved typical runs, while less runner contention had a much larger effect on the slowest runs.

What I’d do next

At this point, one runner is enough again. Most pull requests only pay for the tests they can plausibly affect, while the full suite runs overnight.

If the workload eventually exceeds one machine, I can add runners after removing the work that does not need to happen. Otherwise I would be spending more money to run tests that are unrelated to the change.

The larger lesson is about arrival rate rather than headcount. One engineer working with coding agents produced changes fast enough to recreate a CI scaling problem I had previously encountered on a large engineering team. As agents increase the rate of change, verification has to become more selective or the infrastructure behind it has to grow at the same rate.

Subscribe now

You are doing great work, and there is a lot of value in sharing it, especially outside your team. Sharing can be daunting or feel like it is distracting from solving problems. At least, that’s how I used to feel about it. I only grasped the effects sharing has when I became a manager.

Isn’t my manager supposed to share the team’s work?

As a manager, I ensure my team is doing work that matches the organization’s priorities. This process involves conversations with other teams. I share what we work on, how it might affect them, and if there are any actions to take.

Sharing context and building relationships need a lot of work. When my reports post their work on slack, present, or write blog articles, the shared context travels through the organization. This results in less time explaining what we do and how it might affect them. Often, teams affected by my team’s work will reach out themselves, saving even more time.

Even though there are different roles in a team, everybody works towards the same outcome. It is a collaborative process, making lines of responsibility murky in places. You would pair, mentor, or pick up tasks from your team members to achieve team goals. Your manager is the same.

Why can’t my manager keep up with my work?

I used to believe my work should speak for itself. Instead of indulging in it, I could spend more time solving problems. But this isn’t realistic. Everybody relies on someone else to catch their attention. Your work needs to be approachable so people understand its value. Especially your manager.

As a manager, I need help to keep up with my team. If I have six reports, each working 40 hours a week, that’s 240 hours to stay on top of. The only way to keep up is for my reports to help me understand what they’re doing.

What if sharing my work feels sleazy?

I struggled to share my work because I thought I was selling it. It felt sleazy. But I realized that it is more like celebrating an achievement. Overcoming a challenge is worth feeling good about and celebrating. The work doesn’t have to be finished either. Hitting a milestone, de-risking a project with a prototype, or learning something new are all worth sharing.

Sometimes it can feel awkward to be in the limelight, especially when it is a team effort. But when you share the team's work, you lift everyone up. It creates more awareness of their achievements, and if you share publicly, your team members can reference it in their resume.

What if I only want to code?

The more important question is how much impact you want to have. Most problems worth solving are a team effort. You increase your impact by working through your team, which requires sharing your work. Other people can learn from you and take a similar approach, which is part of technical leadership.

You will also build a reputation for solving problems. And as a consequence, build trust to address more impactful problems. Which also makes it easier for me to promote you and raise your salary.

Another aspect to consider is avoiding micro-management. The most impactful teams I’ve worked on were self-organizing. We built trust with our manager by producing good work and advocating for it across the organization. They only stepped in when things weren't working as expected, and we needed coaching.

What if I'm not comfortable sharing?

It’s normal to feel awkward when learning something new. The trick is finding a way to talk yourself into it. Everybody is a bit different and needs their own narrative. The one that clicked for me was when a company I worked for went through a round of layoffs. I realized if I didn’t share what my team shipped, nobody would know why our work mattered, and we would be laid off. The pressure from big challenges and preventing downside risks motivate me. Some people get paralyzed by pressure and need a positive narrative. Others get motivated by how they affect other people.

I also like asking my team members for their thoughts when I’m unsure about something I want to share. Their feedback helps me make a decision. And they can help me through the creation process. I like sharing drafts and presentations early to rapidly iterate on them, which makes it much faster to produce and less daunting to share.

Who should I share with?

Generally, the broader the audience, the less detail is needed. I can share much more detail within my team than with people outside it. For them, more detail is better because your team members can apply it. But if I share my work with my department, I will share what I shipped, how it would affect them, and what they can learn from it.

Creating awareness of what goes into our work across the organization is a powerful tool. For example, it is valuable for the marketing team to know that an app will need downtime for maintenance. They don’t need to know the details. It is enough for them to be aware of the constraints affecting them. This helps coordinate when campaigns happen and when downtime is scheduled.

A great way to get your point across faster is to cut down on jargon. For example, someone with a background in marketing will understand "we want to improve our newsletter's CTR with a better CTA design" to mean "we want to increase the rate of how many people sign up for our newsletter, and we believe tweaking the signup form will help". But people outside their domain need the more explicit version. Likewise, people outside your team won't understand your jargon.

On the flip side, it helps to adopt your audience's language. I like reading through my audience's writing and presentations to integrate their concepts into my content. This approach makes it easier for my audience to understand me.

Wrapping up

Sharing your work creates awareness across the organization. It helps your manager coordinate with other teams and gives you more autonomy. You will be trusted to solve more impactful problems, which makes it easier for your manager to promote you. It can be uncomfortable to share. You can get help from your team to find interesting topics and progressively increase the audience. People are interested in your work, but you have to try to meet them where they are.

I would love to hear your thoughts on the topic. Feel free to message me on Twitter and if you want to hear more from me, make sure to follow.

Share Human Systems

Subscribe now

Subscribe now