How to Implement AI Agents: From First Use Case to Production

Most agent projects do not fail on the model. They fail on the parts around it: a task scoped too broadly, tools with more access than the job needs, no way to tell whether a change made results better or worse, and a launch straight to full autonomy. Those parts are ordinary engineering work, and they can be planned.

A reliable way to implement AI agents is to move through fixed stages, from use case selection and tool design to evaluation and gradual rollout, and to let the agent advance to the next stage only when it meets agreed criteria on real cases.

This guide sets out seven implementation stages with deliverables and exit criteria, explains how to design tools and permissions, and shows how real deployments, including .wrk projects, handled each stage. It does not compare agent frameworks, and it is not a coding tutorial.

Key takeaways

• Start with one narrow, high-volume task where success is easy to check, rather than a general-purpose assistant.
• Give the agent the smallest set of tools and permissions the task needs, and separate actions that read, draft and change data.
• Build an evaluation set of real cases before building the agent, and use it to judge every change.
• Roll out in steps, from shadow mode to assisted use to supervised autonomy, and keep a human route open at every step.
• Log every model call and tool call, so failures can be traced and costs controlled.
• Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, mostly over cost, unclear value and weak risk controls, which are all things this process addresses.

The building blocks of an AI agent

Every agent, whatever framework it uses, is assembled from the same seven parts. Knowing them makes planning and estimating far easier.

Stage 1: choose a narrow use case

What you do. Pick one task that repeats often, has a clear owner and has an outcome you can check, such as a resolved request, a correct record update or an approved draft. Write down what the agent should achieve, which systems it touches and what it must never do.

Deliverable. A one-page brief with the goal, inputs, systems, success measure and the limits of the agent's authority.

Exit criterion. The business owner and the engineering lead agree on the success measure and on what counts as a failure.

In practice. For a regional telecom operator handling more than 12,000 inbound calls a day at peak, .wrk did not build a general voice assistant. The agent covered the 15 most common subscriber requests, such as balance checks, top-ups, tariff information and SIM blocking, and handed everything else to a live agent. Read the voice AI case study.

Stage 2: decide whether it needs to be an agent

Many tasks that look like agent work run better as a fixed workflow with AI steps, which is cheaper to run and easier to test. Anthropic's guide to building effective agents recommends finding the simplest solution that works and adding autonomy only when the task requires it.

What you do. Map the steps. If they are the same every time, build a workflow. If the path depends on what the agent finds along the way, an agent is justified.

Deliverable. A process map that marks which steps are fixed and which need the model's judgment.

Exit criterion. Each step has an owner: code, the model or a person.

In practice. For a US law firm, .wrk processed case bundles of about 250 pages through five specialized assistants in a fixed sequence, each handling one topic with its own examples and templates. A single open-ended agent could not handle the volume or keep topics apart. Read the legal document processing case study. Our comparison of AI workflows and AI agents covers this decision in detail.

Stage 3: design tools and permissions

Tools are where an agent touches your business, so they deserve the most design attention.

What you do. Define each tool with a clear name, a precise description and a strict input schema. Give the agent its own credentials or act with the signed-in user's permissions, never with an admin token. Sort tools by the level of risk they carry.

Deliverable. A tool catalog with schemas, permission levels and the systems each tool reaches.

Exit criterion. Security has reviewed every tool that writes data or reaches an external system.

In practice. In a CMS-based SEO tool, a model kept suggesting links to pages that did not exist. .wrk restricted the tool so the model could only choose from an approved list of URLs, which removed that whole class of error. Read the AI SEO tool case study. Standards such as the Model Context Protocol make tools reusable across agents, but each tool still needs its own permission review.

Stage 4: build the evaluation set first

What you do. Collect 50–200 real cases with the result you expect, including the awkward ones: incomplete requests, contradictory data, attempts to push the agent outside its role. Decide how each run is scored, by exact match, by rules or by a reviewer.

Deliverable. A versioned test set and a script that runs the agent against it and produces a score.

Exit criterion. The current manual process has been scored on the same set, so you have a baseline to beat.

In practice. On the legal document project, .wrk iterated prompts against attorney-approved sample opinions and first proved on real 250-page bundles that a single-prompt approach failed. That evidence shaped the five-assistant design before full development began.

Stage 5: build, then harden against failure

What you do. Build the agent loop with limits on steps, time and cost per task. Add validation after each tool call, retries with the error message when a step fails and a clear stop when the agent is stuck. Test it against the security risks specific to agents. The OWASP Top 10 for Agentic Applications lists them, including agent goal hijack, tool misuse, identity and privilege abuse and cascading failures.

Deliverable. An agent that passes the evaluation set at the agreed level, with documented limits.

Exit criterion. The agent beats the baseline on the evaluation set and fails safely on adversarial cases.

In practice. Two patterns from real projects stand out. In the legal project, assistants sometimes got stuck refusing corrections, so .wrk added an automatic session reset through the API. In a large code migration, Airbnb fed every test, linter and compiler error back to the model and retried until the file passed, which let about 97% of 3,500 test files migrate automatically.

Stage 6: roll out in steps

Autonomy should be earned. Move through four steps, and advance only when the numbers support it.

1. Shadow mode. The agent runs on real cases, but its output is only logged and compared with what people did.
2. Assisted. People see the agent's proposal and accept, edit or reject it.
3. Supervised autonomy. The agent acts alone on low-risk cases, and people review a sample and every exception.
4. Autonomy within limits. The agent handles its full scope, with escalation for anything outside it.

Deliverable. A rollout plan with the metric that unlocks each step.

Exit criterion. The agreed metrics hold for a set period at each step before the next one starts.

In practice. The telecom voice agent moved from shadow mode to limited traffic and then to full production on two pilot contact-center sites before any wider rollout. It had to respond within two seconds, ran on a high-availability cluster with automatic failover and transferred callers to a live agent when a request fell outside its 15 intents.

Stage 7: operate and improve

What you do. Log every run with its inputs, tool calls, outputs, cost and outcome. Review failures weekly, add them to the evaluation set and re-run the set before every prompt, model or tool change. Track cost per completed task, since agents make a variable number of model calls.

Deliverable. A dashboard with success rate, escalation rate, cost per task and the most common failure types.

Exit criterion. There is none, because this stage runs for as long as the agent does.

In practice. On the NEXUS threat intelligence platform, .wrk instrumented every pipeline stage, including language model latency, and the team used Claude as a coding agent with project-specific instructions, a security sandbox and automated code review in CI. Read the NEXUS case study.

Who you need on the team

A single-task agent usually takes four to eight weeks to build, and an agent with several integrations takes three to four months. Either way, it needs a small team:

• A business owner who defines success, accepts results and decides on each rollout step.
• An AI engineer who designs instructions, the agent loop and the evaluation set.
• A backend engineer who builds the tools and integrations with your systems.
• A QA or domain reviewer who scores results and collects failure cases.
• Security and DevOps support for permissions, hosting, logging and monitoring.

Budgets depend mostly on the number of integrations and the level of autonomy. Our guide to AI agent development cost breaks the figures down.

Why agent projects get canceled

Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls. Each cause maps to a stage above.

• Escalating costs come from open-ended loops and broad scope. Step limits, cost tracking and a narrow use case keep them in check.
• Unclear business value comes from launching without a baseline. An evaluation set scored against the current process shows the value in numbers.
• Inadequate risk controls come from broad permissions and early autonomy. Permission levels and staged rollout address both.