The measurement gap: even a 1,400-engineer platform team can’t measure the drift it can feel
The strongest evidence for vendor drift isn’t a benchmark — it’s a practitioner with everything to gain from measuring it admitting he can’t. Heitor Lessa, Principal Engineer at Adyen (11 years at AWS before that; built Lambda Powertools), running a spec-driven agentic workflow for a ~1,400-engineer org, on the Beyond Coding podcast (July 2026, ~1:17):
“Every time there’s an outage at Anthropic, every time there’s a new model, every time something happens, I notice there’s a dip in quality. But it’s difficult to measure. It’s very difficult to do benchmarking. I barely trust [vendor] benchmarks nowadays. I have to have something that safeguards me from these deviations, these quality differences.”
Three admissions in one paragraph: the drift is real enough to notice, hard enough to measure that a principal engineer at a payments company hasn’t, and the vendor’s own numbers are not trusted by the vendor’s own customers. (Transcribed from audio; lightly cleaned.)
The cost math leadership is about to do
Same conversation, on token spend at enterprise scale (~0:58):
“When leadership start questioning — hang on a second, do I need an engineer plus $5,000 a month, $3,000 a month just for them to do their work? That math doesn’t add up. For one engineer, a few engineers, sure — pocket money, especially in an enterprise. But when you’re looking at the scale of, for instance, the 1,400 engineers that we have, that math starts to show very quickly.”
His response was model tiering — frontier for exploration, mid-tier for implementation, open-weight for review rounds — after a single front-end refactor burned ~200M tokens on a frontier model (“I need to stop using Opus”). Note what tiering does to the drift problem: it multiplies the number of served models a team depends on, each with its own silent-change surface.
The catch
If the teams with the most engineers, the best tooling, and the strongest vendor relationships cannot measure what they can feel, “we’ll notice if something changes” is not a strategy for anyone else either. The workable version is unglamorous: baseline the behaviors you depend on before you need them — a fixed probe suite, run on a cadence, against tasks your business actually cares about — so that a felt dip can be checked against your own numbers instead of the vendor’s. That is the second move in the playbook, and it is the one the industry keeps skipping.
The verification gap: your AI subscription won’t let you check what it’s serving
“The responses feel different” is not a measurement. After a session where a coding-model subscription behaved unexpectedly, we built a small probe harness to check whether the served model can actually be verified by the customer. The short answer: on the managed endpoint we tested, mostly it cannot — and the documentation confirms the served model can change without the served model ID changing.
Finding 1 — the verification knobs are refused
The managed coding endpoint (api.kimi.com/coding/v1, official third-party API-key path) rejects logprobs outright (HTTP 400, invalid parameter) and rejects temperature=0 (“only 1 is allowed for this model”). Those are the two strongest model-fingerprinting signals. A customer probing the served model is left with text-only comparisons at forced temperature 1 — the weakest form of evidence.
Finding 2 — the served model changes on a settings toggle
The vendor’s own documentation states that disabling “thinking” routes both K3 and K2.7 Code to K2.6, server-side (vendor docs). The model you are talking to changes with a switch; the model ID you requested does not. This is documented — but it is not surfaced anywhere in the product flow, and no notice is given per-request.
Finding 3 — style comparison cannot catch a swap
We ran an identical probe suite through two genuinely different models (K3 vs K2.6) and scored text similarity. Word-overlap bands overlapped: same-model 0.20–0.35, cross-model 0.19–0.29. One-shot style similarity has zero discriminating power. If your swap-detection plan is “the outputs look the same,” it is not a plan.
The contrast — what verification looks like when the knobs exist
Against locally hosted models (owned weights, Ollama), the same harness gets the full signals:
At temperature 0, a qwen3.5:4b is bit-exact across repeated runs — mean per-token logprob delta 0.0000 on all probes.
Two different models (qwen3.5:4b vs gemma3:4b) separate cleanly: |Δlogprob| 0.06–0.62, zero band overlap.
Both models once answered a probe with the identical string (“Au”) while their probability structures differed by 0.44 nats. Same words; different machines.
The conclusion writes itself: verification capacity follows ownership. Models you run, you can check. Models you rent, you largely cannot — “trust us” is the whole SLA. Until that changes: pin dated model IDs where possible, keep behavioral baselines of anything production-critical, and treat every alias as a scheduled surprise.