I lead engineering, but I have not built an agentic AI system before. Right now I am learning this space, and this post is mainly how I am thinking about it — what can go wrong, what architecture I want to try, and what I expect before even running a POC.
Why this post exists
I have built and delivered many software systems over the years. But “Agentic AI in production” feels different to me. I have done POCs and some experimental coding, but not multi-agent systems with strong operational constraints. So I am in a similar position as many senior engineers today — I understand what may not work, but I am not fully sure which instincts will actually hold in this space.
When I started reading about Agentic AI, most of the content was not very helpful for this stage.
- Some are vendor demos — they show what is possible, but not what works in real production
- Others are success stories — but written after everything worked, so it looks obvious
But when you are starting, decisions are not obvious.
This does not help someone who is trying to design architecture today.
So before writing production code, I am doing what I usually do — create a framework to evaluate architectures against possible failure scenarios. Then I document it so I can validate it later. This post is that framework. A follow-up will share what actually happened after running the PoC.
The number that started my thinking
I read that around 95% of enterprise GenAI pilots do not create measurable business impact (MIT report). And BCG says around 70% of failures are because of operational or process issues, not model problems.
This made me think. The question is not “can we build an agent” — that is already proven. The real question is: what does the 5% do differently?
Most likely, the answer is in operational aspects that demos do not show — cost, latency, observability, and fallback behavior.
So my first step is not coding. It is creating a framework to evaluate architectures against failure modes, and then writing code based on that.
Four failure modes most tutorials skip
After reading multiple sources (MIT, BCG, NVIDIA docs, blogs), I came up with this framework. These are hypotheses, not proven results.
Failure mode 1: One model handling everything
Many tutorials use a single large reasoning model for all queries.
For a customer support use case (my POC domain), I assume traffic may look like this:
- ~50% simple FAQ queries (can be answered quickly with small model)
- ~35% need tool calls for data
- ~15% are complex multi-step queries
If this is roughly correct, then using a large model for all queries means high cost and high latency even when not needed.
Failure mode 2: No memory of repeats
Customer support has many repeated questions. Same question comes with small variations.
A basic agent calls the model every time.
Caching is not discussed much in tutorials, or it is treated as later optimization. My assumption is that by the time “later” comes, cost is already too high. So caching should be part of initial design.
Failure mode 3: Invisible behavior
Example scenario: agent tells a customer that refund is processed, but actually it is not.
Later this becomes an escalation, and no one can explain why it happened. Logs only show API success, but not agent reasoning — which tool was used, what parameters were passed.
This is risky, and also difficult to detect during the POC stage.
Failure mode 4: No graceful degradation
During peak traffic (like Black Friday), system load increases. Latency becomes very high.
A good system should reduce response complexity and respond faster instead of waiting for perfect answer.
I have not built this kind of routing before, so this is something I want to test carefully.
The architecture I’m planning to test
Mapping failure modes to solutions:
- One model for everything → multiple model sizes with routing
→ Expectation: ~85% queries handled by cheaper model - No memory of repeats → semantic caching
→ Expectation: 25–40% queries handled without model call - Invisible behavior → telemetry using NAT and App Insights
→ Expectation: agent behavior can be tracked and queried - No graceful degradation → router monitors latency (p95)
→ Expectation: system shifts to faster responses under load
All these numbers are assumptions. Actual results will come after POC.
Why this stack
Choosing stack is harder than writing code.
NeMo Agent Toolkit (NAT): It is framework-independent and supports OpenTelemetry. This helps keep observability separate from application logic.
Azure AI Foundry: I already work in Azure ecosystem. Foundry combines orchestration and observability. It also manages agent identity and state.
NVIDIA NIM: OpenAI-compatible API gives flexibility. Performance is optimized for Azure GPU. Nemotron models provide consistent range from small to large.
What the POC will measure
I’ve built a benchmark harness that compares two versions against the same dataset comparing:
- A naive baseline (all queries to large model)
- An optimized system (routing + caching + telemetry)
Dataset:
- 81 customer support queries
- 51% FAQ / 35% retrieval / 15% reasoning
- Includes duplicate-like queries for cache testing
- Mock order system with realistic cases
Metrics I will measure:
- Cost per 1K queries
- Latency (p50, p95, p99)
- Cache hit rate
- Accuracy difference
- Token usage
- Behavior under load
I will share all results, including where assumptions are wrong.
What’s coming next
Next post will include full POC results — actual numbers, what worked, what failed, and unexpected learnings.
I prefer to take time and share real data instead of assumptions.
If you are also evaluating agentic AI for first time, this framework may help even before results are available.
Disclosure: this post is written from the perspective of an experienced engineer evaluating a new domain, not a practitioner reporting on production deployment. Expected behavior ranges in the architecture section are hypotheses to be validated, not measurements. The follow-up will include numbers from the POC.
Leave a Reply