Beyond Alerts: Using AI to Continuously Improve Production Systems
By Jake Moses
Production systems usually receive a serious examination at one of two moments: something breaks and an engineer gets paged, or a monthly or quarterly review reveals that an API has become slower, infrastructure costs have crept upward, or a workflow is no longer performing as expected.
Both approaches are reactive. One waits for failure; the other waits for the calendar. In between, a system can quietly become slower, more expensive, or less reliable without crossing a threshold that attracts anyone’s attention.
AI agents create a third option.
Instead of waiting for an incident or the next scheduled review, an agent can continuously examine a system for opportunities to make it faster, cheaper, and more reliable. It can operate like a senior engineer with the time to keep asking: Why does this work this way? Does it still need to? Could it be better?
Connecting Signals to Their Purpose
This kind of analysis becomes possible when an agent can connect three layers that traditionally sit apart:
- Telemetry: What is happening?
- Code: What is actually running?
- Intent: Why does this behavior exist?
Traditional monitoring is very good at detecting conditions a team already knows to watch for. It can surface an elevated error rate, a slow trace, or a failed job. Identifying an optimization opportunity, however, requires more context.
A 200-millisecond increase in latency might not trigger an alert. Whether it matters depends on which API is affected, which customer interaction it powers, what the underlying code is doing, and why that code was written in the first place.
When an agent can connect a metric to the implementation and its purpose, it can move beyond reporting symptoms and begin investigating opportunities.
This does not replace observability. It expands what engineering teams can do with the information observability already provides.
Building a Continuous Optimization Workflow
I built this workflow using Claude Agent Skills and MCP integrations with GitHub, Sentry, and internal product documentation.
Together, those connections give the agent the context it needs:
- What each component is responsible for
- How to query and interpret telemetry
- What “healthy” means for the system
- Which constraints and tradeoffs shaped the code
I then coached the agent across repeated runs, persisting useful findings so that each investigation could build on previous ones.
Over several weeks, it stopped behaving like an isolated script and began operating more like a coworker—one focused entirely on finding optimization opportunities and capable of remembering what it discovered during earlier investigations.
From Continuous Evaluation to Confident Scale
The most important result of this workflow has not been any single optimization. It has been the confidence to scale.
The production agents supported by this process now participate in thousands of customer interactions every day. At that volume, waiting for a visible regression is not enough. A small decline in latency, reliability, or output quality can affect a significant number of interactions before it becomes an obvious incident.
Continuous evaluation makes it possible to detect those changes earlier. It provides greater confidence when expanding deployments and helps preserve that confidence as the underlying models, prompts, code, workflows, and integrations evolve.
That confidence does not come from assuming the system will remain stable. It comes from continuously verifying that the system is still behaving as intended.
The larger shift is one of expectations. Continuous optimization and validation once looked like luxuries reserved for organizations with enough headcount to dedicate engineers to them. AI agents are beginning to lower that barrier.
This workflow is just one of the projects I’ve been excited to build at Pedestal. It has helped give our team the confidence to scale production agents to thousands of customer interactions each day while continuing to trust the systems behind them as they evolve.
A system that evolves continuously needs a process for evaluating it continuously.