This is a follow-up to my earlier post where I laid out a framework for evaluating multi-agent architectures. In that post I made four predictions about what would happen when I tested an agentic system on Azure AI Foundry and NVIDIA NIM. Here is what actually happened: real numbers, some real surprises, and a few things I got wrong in ways I did not expect.
What I set out to validate
In the earlier post I laid out four failure modes: one model handling everything, no memory of repeats, invisible agent behavior, and no graceful degradation. I also sketched an architecture meant to fix each one, and made four specific predictions about what I would find. For this post I ran the full 81-query benchmark (51% FAQ, 35% retrieval, 15% reasoning) against two versions: a naive baseline that sends every query to a 49B reasoning model, and the optimized router with intent classification, cache, and telemetry-driven downshift.
Here is how the four predictions turned out:
| Prediction from the framework post | What actually happened |
| Cache deflects 25-40% of queries | 0%. Not just lower than expected — completely absent, in both runs. |
| Classification accuracy will be the bottleneck | Partly right, but I can’t fully prove it. The evidence points to latency, not accuracy. |
| Graceful degradation hardest to validate | Could not validate it at all. The mechanism can never trigger with this dataset, by design. |
| Observability needs custom instrumentation | Confirmed, and the gap was bigger than expected. |
The headline is this: the optimized router was 3.6x faster at p50 and 5.5x faster at p95. That is a real and substantial win. But it used slightly more total tokens than the naive version, not fewer. The architecture I designed to cut cost did not cut token count, it cut latency. Those are two different things, and mixing them up was the biggest mistake in my original framework.
The headline numbers:
| Metric | Naive (everything → 49B) | Optimized | Delta |
| Queries run | 81 | 81 | — |
| Wall time | 1,708.7s (~28.5 min) | 499.0s (~8.3 min) | 3.4x faster |
| p50 latency | 17,031 ms | 4,725 ms | 3.6x faster |
| p95 latency | 46,904 ms | 8,522 ms | 5.5x faster |
| p99 latency | 75,475 ms | 50,595 ms | 1.5x faster |
| Input tokens | 5,596 | 17,132 | 3.1x more |
| Output tokens | 35,445 | 25,486 | 28% fewer |
| Total tokens | 41,041 | 42,618 | ~4% more |
| Accuracy (FAQ + retrieval) | 68.1% (47/69) | 66.7% (46/69) | -1.4 pp |
| Cache hits | 0 | 0 | — |
| Downshifts | 0 | 0 | — |
One small correction about my own tooling: my bench.py script’s delta table printed “Total tokens: -4% fewer” for the optimized version. If you read it literally, the sign convention is technically consistent, but in plain English it reads backwards. The optimized version actually used more tokens, not fewer. I wrote this comparison table myself and still almost walked away with the wrong conclusion from my own output. That is now item one on the “what I would do differently” list.
So: a big win on latency, basically no change on raw token count, and accuracy that stayed roughly the same. None of these three results match what I predicted in the framework post.
The per-category breakdown
Naive baseline: every query goes to the 49B model.
| Category | Queries | Median latency | Input tokens | Output tokens | Total tokens |
| faq | 41 | 16,589 ms | 2,558 | 16,811 | 19,369 |
| retrieval | 28 | 11,519 ms | 1,860 | 10,439 | 12,299 |
| reasoning | 12 | 26,981 ms | 1,178 | 8,195 | 9,373 |
| TOTAL | 81 | — | 5,596 | 35,445 | 41,041 |
Optimized router: intent classification plus routing.
| Category | Queries | Median latency | Input tokens | Output tokens | Total tokens |
| faq | 41 | 4,707 ms | 5,778 | 11,522 | 17,300 |
| retrieval | 28 | 4,364 ms | 8,257 | 7,904 | 16,161 |
| reasoning | 12 | 7,481 ms | 3,097 | 6,060 | 9,157 |
| TOTAL | 81 | — | 17,132 | 25,486 | 42,618 |
The FAQ and retrieval numbers look close to what I hoped for in the framework post. FAQ went from 16.6s to 4.7s median, about 3.5x faster. Retrieval went from 11.5s to 4.4s, about 2.6x faster. That is the routing working as designed: small model, small prompt, fast answer.
The reasoning row is where things get interesting, and I think this is where the real story of this post is.
Prediction 1: cache deflection rate
What I predicted: 25-40%, probably closer to 30%. I even wrote that I would be surprised if the real number went above 30%
What actually happened: 0%, in both runs.
The reason is almost regrettable once you see it. The cache hashes the lowercased, stripped query string, and only returns a hit on an exact match. I deliberately added near-duplicate pairs to the eval set, like “What’s your return policy?” and “What is the return policy?”, just to give the cache something to catch. But those are different strings, character by character. They hash to different values. They never collide. Every one of the 81 queries in the eval set is a distinct string, so the cache never had a chance to fire, by design.
Here is what I think happened in my head when I wrote the framework. I was thinking about semantic de-duplication, the same underlying question asked in different words. But what I described and built was syntactic de-duplication, the exact same string, byte by byte. These are two different mechanisms with very different hit rates, and I did not notice I had swapped one for the other until the number came back as a flat zero.
For production, this means the “cache” failure mode and the “semantic similarity” failure mode are not two steps, where you build the simple version first and upgrade later. They are the same failure mode. A verbatim cache on a natural-language workload is not a weaker version of a semantic cache. It is closer to having no cache at all.
Prediction 1: classification accuracy
What I predicted: this would be the bottleneck
What actually happened: the evidence points in one direction but does not prove it, because I made a mistake in how I measured things. I never logged the router’s actual routing decision for each query, separately from the eval set’s ground-truth category. The per-category tables above are grouped by ground truth, not by where the router actually sent each query. So I can describe what I see, but I cannot give you a clean misclassification number.
Here is what I see. For the 12 queries labeled “reasoning” in the eval set, the naive version, all 12 sent to the 49B model, had a median latency of 26,981ms. The same 12 queries, run through the optimized router, had a median latency of only 7,481ms, less than a third of that. If the router had correctly classified all 12 as “reasoning” and sent them to the same 49B model with a similar token budget, I would expect something close to 26,981ms, not a 3.6x drop. The most likely explanation is that a meaningful number of the “reasoning” queries got classified as “faq” or “retrieval” and were answered by the fast 9B model instead.
And yet, overall accuracy on FAQ plus retrieval barely moved, from 68.1% to 66.7%, a drop of 1.4 percentage points. This is the part I did not expect. I assumed misclassification would either show up as a clear drop in accuracy, or would not be happening at all. Instead I got a result that does not fit either story cleanly: the headline accuracy number looks more or less fine, but the latency evidence says the routing is not doing what I designed it to do. Or, the other explanation, the 9B model is actually capable of answering some “reasoning” queries well enough when it has the order data as context, and my ground-truth labels were stricter than the task really needed.
I do not know which of these it is. That is not a satisfying answer, but it is the honest one, and it comes directly from a gap in how I built the measurement harness, one I did not notice until now.
Prediction 3: graceful degradation
What I predicted: this would be the hardest part to validate, and that the simplest possible heuristic, a rolling p95 over the last 50 reasoning-route latencies, triggering a downshift once it goes above 4000ms, might not survive contact with reality.
What actually happened: I never got to find out, because the mechanism can never trigger with this dataset, by design. The downshift only kicks in once len(_RECENT_REASONING_LATENCIES) >= 20, but the entire eval set has only 12 reasoning-routed queries, and based on Prediction 2, maybe even fewer than that actually reach the reasoning branch. The rolling window never fills up. downshifted was 0 in both runs, and it was guaranteed to be 0 before a single query even ran. This is not something the benchmark discovered. It is something I could have figured out just by reading my own code.
This is a different kind of mistake than Predictions 1 and 2. Those were cases of “I built the wrong thing and found out by running it.” This one is more like “I built a mechanism whose condition for activating is bigger than my whole test, and I never checked.” To actually test graceful degradation I would need either a much bigger eval set, so the reasoning bucket alone has more than 20 queries, or a lower window threshold sized to the dataset, or, more realistically, a separate load-generation script that hits the reasoning route over and over, regardless of what the eval set’s natural mix looks like. A POC benchmark that runs each query once, in order, will never produce “saturation.”
Prediction 4: observability
What I predicted: NAT plus App Insights would not be enough out of the box, and I would need more custom instrumentation than the “framework-agnostic, bolts on automatically” pitch suggests.
What actually happened: confirmed, and the gap was bigger than I expected. configure_azure_monitor() on its own only captured customMetric and performanceCounter entries, generic process-level telemetry like CPU and memory that says nothing about what the agent actually did. Getting the HTTP calls to NVIDIA’s endpoints to show up as queryable dependency traces took three separate fixes: installing opentelemetry-instrumentation-httpx explicitly (it is not bundled by default for the openai SDK’s transport layer), adding an atexit flush handler (short scripts exit before OpenTelemetry’s batch exporter runs on its own schedule), and then fixing a version conflict that the httpx instrumentation package created with the versions azure-monitor-opentelemetry had pinned internally.
Once all three pieces were in place, the result was genuinely good. A KQL query against the dependencies table returned 190 individual HTTP calls to integrate.api.nvidia.com, each one with a timestamp, target, and full request URL. That is real, per-call visibility into what the agent did and when. A one-line summarize query (summarize call_count = count() by name) gave a clean count of 190 calls to POST /v1/chat/completions.
Application Insights resource overview, showing the connection string and resource group.
KQL summarize query: 190 calls to POST /v1/chat/completions.
Individual dependency calls to integrate.api.nvidia.com, with timestamps and request targets.
So the end result, “agent behavior is queryable in KQL”, is real, and once it works it is genuinely useful. Getting there is just not zero-config. “Framework-agnostic instrumentation that bolts onto whatever agent code you have” is true in the sense that it is possible. But it understates, by a lot, how much of that bolting on you have to do yourself.
The thing that surprised me most
The cache going from “predicted to deflect 25-40%” to “deflected exactly 0%” is the most dramatic single number in this post, but I do not think it is the most important finding. The more important one is this: the architecture I built to reduce token use did not reduce token use.
Total tokens went from 41,041 (naive) to 42,618 (optimized), up, not down. The reason is obvious in hindsight: every query in the optimized version now makes two model calls instead of one, a classification call and then an answer call. The classification prompt alone carries around 140-150 tokens of instructions on every call. Across 81 queries, that adds up to about 11,500 extra input tokens before any of them produce a single token of actual answer. Output tokens did drop a lot, from 35,445 to 25,486, a 28% reduction, because the small model gives shorter, less rambling answers than the 49B model’s verbose reasoning style. But that was not enough to make up for the new input-token cost.
If your idea of “cost” is “total tokens,” this architecture is a wash, maybe even a small loss. If your idea of “cost” is “dollars,” it probably still wins, because tokens from a 9B model are usually priced well below tokens from a 49B model, often by 5-10x depending on the provider. But I did not measure that part, because I was tracking token counts, not tokens weighted by which model produced them.
The lesson I am taking from this: “tokens” is not a unit of cost. It is a unit of work, and the price per unit of work changes by an order of magnitude depending on which model did the work. A benchmark that reports “total tokens” without breaking down which model used which tokens is giving you a number that does not actually answer the cost question, and my own headline table did exactly that, right next to the sign-convention bug in the delta row. Two separate mistakes with the same root cause: I spent more care on building the architecture than on deciding what “better” even meant before I measured anything.
What I’d do differently if I were starting over
- Build the semantic cache from day one. The verbatim cache was not a simpler version of the real thing, it was a different thing that happened to share a variable name and a docstring. If a cache cannot deflect paraphrased duplicates, it is not really a cache for this kind of workload.
- Log the router’s actual routing decision for every query, separate from the eval set’s ground-truth category. Right now I have a benchmark that produces clean numbers, and a router whose real behavior I can only guess at from latency shapes. These should be two separate things I can both observe directly.
- Calculate cost using per-model token pricing, not raw token counts. “42,618 tokens” does not mean much on its own. “17,132 tokens at the 9B rate plus 25,486 tokens at the 49B rate” is the number that would actually tell me if this architecture saves money.
- Size the saturation-detection window to match the dataset, or test it separately with a load generator. A mechanism whose activation threshold is bigger than your whole test set has not really been tested just by running the test set through it once.
- Set up the httpx instrumentation, the atexit flush, and the dependency version pins before running anything at all, not as a string of fixes I discover halfway through, each one needing its own debugging session.
Updated framework
| Failure mode | Original framing | Updated framing |
| #1: One model for everything | “Route ~85% of queries to small model, cutting cost” | Routing works and gives a real 3.6-5.5x latency win. Whether it reduces cost depends entirely on per-model token pricing, which this benchmark did not capture. Total token count alone went up slightly. Latency and token cost are not the same thing. |
| #2: No memory of repeats | “Semantic cache deflects 25-40% of queries” | A verbatim or syntactic cache deflects close to 0% on any realistic natural-language query mix. “Cache” and “semantic similarity” are not two separate upgrades, one after the other. They are the same requirement. |
| #3: Invisible behavior | “NAT + App Insights makes agent behavior queryable in KQL” | True once it is set up, but “once set up” hides real, multi-step integration work: explicit instrumentation packages, manual flush handling for short-lived processes, and dependency version pinning. Plan for a debugging session here, not just a config change. |
| #4: No graceful degradation | “Router watches its own p95 and downshifts under saturation” | Untested. The mechanism’s minimum sample size was bigger than the entire test dataset. It cannot be validated as a side effect of a sequential single-pass benchmark. It needs either a much bigger eval set or a separate, concurrent load test. |
One more failure mode I would add that was not in the original framework:
#5: Routing decisions stay invisible unless you log them separately from the outcomes. I built a router, ran it against 81 queries, and got a full set of benchmark results, and I still cannot tell you with confidence what it routed where. If failure mode #3 is about agent behavior being invisible without instrumentation, this is the same problem one level up. The decisions that produce the behavior need their own log line, separate from whatever the benchmark harness happens to measure on the outside.
Closing
Three out of four predictions were wrong, and the fourth was right but in a more expensive way than I had guessed. None of this was really about Azure AI Foundry, NVIDIA NIM, or the Nemotron models themselves. Every mistake in this post is something I built, measured, or labeled wrong, not something the underlying platforms got wrong. The framework from my first post was still useful even where the details were off. It told me what to look for, and when I looked, I found something, just not what I had guessed.
Putting the two posts together, here is what I think the real takeaway is. The operational layer that demos do not show, caching, routing, observability, degradation, is exactly as hard as the “95% of pilots fail” number suggests. But the way it is hard is more specific and more boring than “agentic AI is still immature.” It is hash collisions that never happen, sample-size thresholds that nobody checked against the test data, a sign convention in a table that says the opposite of what it means, and a cost metric (tokens) that does not actually measure cost. None of these are exotic AI problems. They are the same kind of bugs you get in any system where someone designs the measurement after building the thing, instead of before.
I went into this never having built an agentic system before, with a framework I had put together from other people’s writing. I came out of it having built one, with a framework that my own data has corrected in three out of four places, and a clear idea of what I would build differently next time, even though I do not know yet if I will. That feels like a fair place to stop.
This post is a follow-up to the earlier framework post. The POC was built and benchmarked on a MacBook Pro (Apple Silicon, macOS), using NVIDIA’s serverless endpoints (Nemotron Nano 9B v2 and Llama 3.3 Nemotron Super 49B v1.5) and Azure Application Insights for tracing. The numbers above come from a single run on a single machine and should not be over-interpreted, especially given the measurement gaps described in Predictions 2 and 3.
Leave a Reply