The API returned 200 OK.
The model responded. No exception. No alert. Every dashboard is green.
And the answer was still completely wrong.
That's the part traditional monitoring never had to deal with. Regular applications have logs, metrics, traces, error tracking, and dashboards. AI applications need all of that too. But they add a failure mode those tools can't see: the system succeeded, and the output is still bad.
Why Uptime Isn't Enough
If your service is down, you know. If your service is up and confidently telling users the wrong refund policy, you don't.
So AI observability has to go beyond infrastructure. It's not just "is the system up?" It's "why did the AI say that?"
The loop I use to think about it:
TRACE → SCORE → FLAG → DEBUG → FIX → TEST AGAIN
Let's walk through it.
Step 1: Capture the Request
Start by recording the metadata around every AI request:
- User question
- Session ID
- Timestamp
- Model
- Prompt version
- Temperature
- Environment
Skip this and debugging turns into guesswork. If answers get worse after a deployment, you need to know which model and which prompt version produced them. "It felt better last week" is not a debugging strategy.
Step 2: Trace the Whole Pipeline
An AI request is rarely a single model call. It usually looks more like this:
User Question ↓ Intent Router ↓ Retriever ↓ Retrieved Chunks ↓ LLM ↓ Tool Calls ↓ Final Response
Each step is a span. The complete journey is a trace.
Once you can see the full trace, the question changes. Instead of "the AI gave a bad answer," you get to ask better ones:
- Was the intent classification wrong?
- Did retrieval return the wrong chunks?
- Did the model ignore the context it was given?
- Did a tool return unexpected data?
Each of those points at a different fix. That's the whole value of tracing.
Step 3: Store the Context
For RAG applications, the trace needs to capture retrieval details:
- Documents retrieved
- Chunk IDs
- Similarity scores
- Source names
- Chunks actually passed to the model
The final answer is only half the story. You also need to know what the model saw. A wrong answer built on the right context is a model problem. A wrong answer built on the wrong context is a retrieval problem. Without the stored context, you can't tell which one you have.
Step 4: Track System Metrics
AI systems get the usual infrastructure metrics plus a few of their own:
- Latency
- Token usage
- Model cost
- Error rate
- Retry count
- Failed tool calls
A single request might look like this:
Latency: 2.34s Tokens: 2,153 Cost: $0.0041 Retries: 1 Tool failures: 0
Sudden shifts in these numbers often show up before users start complaining. A cost spike or a jump in retries is usually the first symptom of something quietly breaking.
Step 5: Score Answer Quality
A successful API call is not a successful response. So you evaluate the output itself.
Questions worth asking:
- Was the answer grounded in the retrieved documents?
- Did it answer the actual question?
- Did it hallucinate?
- Was it safe?
- Was it unnecessarily expensive?
- Did the user give negative feedback?
Some of this can be automated. Some needs human review. User feedback is another signal, and a cheap one to collect. You'll want a mix of all three.
Step 6: Flag the Bad Answers
Once you have traces and scores, you can write rules on top of them.
- No retrieval + policy answer → flag
- Retrieved document contradicts the answer → flag
- Sudden cost spike → flag
- New prompt version performs worse → flag
The goal isn't to collect logs. Nobody reads logs. The goal is to automatically surface the failures worth a human's time.
Step 7: Build a "Bad Answers" Dashboard
This is where observability starts paying for itself.
For every flagged response, show:
- User question
- Retrieved chunks
- Final answer
- Prompt version and model
- Cost and latency
- Quality score
- User feedback
- Failure reason
Now an engineer opens one page and investigates an actual failure instead of grepping through thousands of log lines. That's a very different Monday morning.
Step 8: Fix and Retest
Once you know where it broke, you fix that layer, not everything at once.
- Retriever is bad → improve retrieval
- Chunks are poor → change chunking (I wrote about why this quietly kills RAG in Chunking Strategies)
- Prompt is weak → refine the prompt
- Tool behavior is unreliable → fix the tool logic
- Model isn't right for the job → change it, or route to a different one
Then run the evaluation again. That last part matters. A fix you didn't re-score is just a guess with better intentions.
The Mindset Shift
AI observability isn't about knowing whether your system is up. It's about understanding why the AI behaved the way it did.
That's the difference between monitoring an AI application and actually being able to improve one.
The best AI systems aren't built once. They're measured, evaluated, debugged, and improved, over and over.
Trace → Score → Flag → Debug → Fix → Test Again.
— Cheers, NP