Testing at AI speed: We built a drift detection capability, then used it on ourselves

Our development outpaced our ability to test it, as generative AI sped up how quickly we could ship code. Releases were shipping faster than our existing test suite could keep it honest. So, we looked in-house first. That’s how we landed on the drift detection capability inside SmartBear Swagger Contract Testing, and once we saw that it could catch bugs even before customers realized, we didn’t ease into it gently. We picked our biggest, most actively developed backend service, and once we saw that it could catch bugs even before customers realized, we leaned into it

Inside the service and the pipeline 

The repository we chose is a large, mature REST API with large operations, all of them now covered by drift tests. It’s been under active development for more than a decade and integrates with a wide range of external systems: GitHub, GitLab, and AWS and Azure gateways, among others. The OpenAPI YAML file serves as the single source of truth. Swagger Inflector reads it to wire HTTP paths to controller classes, and our Maven build runs Swagger Codegen against it to generate Jersey client stubs and Java model classes. Change the definition, and the clients follow. That’s as API-first as it gets. 

A common blind spot in API-first testing 

Before adding drift tests, we relied on a combination of unit, integration, and end-to-end tests. For a codebase of this size and age, that sounds reasonable, and for a long time, it was. But as the system grew, the tests and the definition started describing two slightly different APIs –  a common trade-off as any codebase of this size keeps evolving. 

That’s an easy thing to miss. We worked in a definition-first way: the DTOs are generated from the OpenAPI spec, so the assumption was that code and definition would stay in sync almost automatically. That covered the models, but not always the definition’s accuracy –  and there was no way to see the difference until drift detection told us. 

Swagger Contract Testing’s drift detection validates whether your running API actually behaves the way your definition says it should. Each test case targets a specific operation in the definition and defines the expected response: status code, schema, content type. When a drift check runs, it calls the real API and checks whether the response matches. Any mismatch between what the definition promises and what the code delivers shows up as drift. 

Was the effort of maintaining tests for every API definition worth it? Our honest take 

The reaction when we started implementing drift tests was mixed. I was excited, personally –  it was something new, it  uses AI to help write the tests, and I wanted to see what it would find. Others were less convinced. The main concern was simple: was the effort of generating and maintaining tests for every API definition actually  worth it? 

There was also a fair question about overlap. Our QA team was already building its own API functional test suite, and on the surface, the two efforts can look similar. Both test the API, after all. But we eventually saw them as complementary rather than competing. Drift checks are fast and lightweight; they check whether the running application matches the definition, with no need for any knowledge of the business logic behind it. The QA team’s functional tests go deeper. They require an understanding of the product, the workflows, and the edge cases. One doesn’t replace the other. 

What we planned versus what we did 

Our original plan was to run drift tests as a gate on every pull request before merging to main. Given the complexity of the service and its dependencies on other infrastructure, we decided to defer making it a PR gate and took a simpler path. Rather than let the perfect block the useful, we set up a nightly GitHub Actions workflow that runs the tests against our test environment. This kept the setup cost low and got us to real results fast. 

Covering every operation was a significant undertaking, but Swagger Contract Testing’s AI agent skills, working through Claude, did a strong job generating the test suite. Claude reads your API definition and produces a test file covering the operations and scenarios it describes: happy paths, error codes, auth failures, and more. We’re still working through fixing the mismatches it detected, but the first run alone gave us a clear picture of the state of the service. 

What we found: Three consistent patterns 

When drift detection ran for the first time, it surfaced several instances of API drift –  a reasonable number on an API with this much history and many integrations. Once we worked through them, about 80% turned out to be real drift worth acting on. Each one fell into one of three patterns. 

Looking back at a sample of that first run, most traced back to the first pattern below:   the definition hadn’t kept pace with the code. A much smaller share fell into the second pattern: genuine code bugs. The rest fell into the third: definition gaps ambiguous enough to trip up even the AI-generated tests. 

Pattern 1: The definition wasn’t updated after the code changed 

This surprised me most, precisely because we work API-first. How can the definition be wrong if it drives the code? The answer is that the definition drives the models, but not every subsequent code change flows back through it. One clear example is this: the definition still said a particular operation would return 422 for AsyncAPI and GraphQL format types, even though we’d already implemented AsyncAPI support. The DTOs were correct; the API definition wasn’t.

 For a developer who hasn’t worked in that area of the codebase, and I hadn’t, the definition is usually the first place you look to understand current behavior. An outdated definition  can send you in entirely the wrong direction. 

Pattern 2: The implementation didn’t match the definition 

I expected the drift tests to surface documentation problems; I didn’t expect actual bugs. But several operations were returning 500 where the definition and tests expected a success code. These were real defects in the running application, and they’d slipped past the existing test suite because the API definition hadn’t covered those paths in a live environment. 

One of these fixes had an outcome I still find satisfying. While I was working through the batch of mismatches, a colleague messaged me about a customer-reported bug: a notification call that was failing in production but that we couldn’t reproduce in the development environment. We couldn’t reproduce it because we’d already merged the fix. The drift check caught it first; we fixed it as part of clearing the mismatch list; and by the time the customer reported it, it was already resolved. Fixing a bug before it’s even been formally reported is a nice win. 

Pattern 3: The definition was ambiguous or incomplete 

Because test generation relies on Claude reading the definition and deriving test scenarios from it, the quality of what Claude produces reflects the quality of the definition itself. If something is unclear or missing, Claude has no way to fill in the gap, and the tests will reflect that incompleteness. 

In our case, several tests came back with 401 Unauthorized where a success code was expected. The reason: the definition didn’t describe which token type was required for those endpoints. Claude had no basis to choose the right authentication, so the tests were failing on auth rather than testing any actual logic. Our QA engineer had raised exactly the same problem with me that week. She needed that information documented and couldn’t find it anywhere. Drift detection surfaced the same gap independently, and now it’s documented. 

What changed after implementing drift detection 

Once we worked through the initial batch of mismatches, the definition became meaningfully more accurate. We fixed a handful of real bugs  before customers reported them. The nightly GitHub Actions run shifted from a diagnostic exercise into an early warning system: if anything new drifts, we find out the following morning rather than from a customer. 

We’re also rolling drift detection out more broadly across the team. The goal is to understand and maintain drift tests as a natural part of how every developer thinks about the relationship between definition and implementation, rather than something a small group monitors in the background. 

What I’d tell someone starting out 

The nightly run was the right starting point for us: low cost, low friction, and it delivered real value from the first run. The results alone –  genuine bugs caught, definition gaps documented, a customer issue resolved before it was even reported – made introducing drift detection capabilities into our workflows a no-brainer. 

What I didn’t anticipate was how much of the value came from what the tests revealed about the gaps in the definition. A definition that drives code generation isn’t automatically a definition that accurately describes behavior. Drift detection is what showed us the difference. “That’s really what we were after: measurable assurance that the API does what its definition promises, even as AI speeds up how fast we ship. That’s application integrity in practice, and drift detection is one concrete piece of it.”