AI coding assistants have moved well past autocomplete. With the Model Context Protocol (MCP), tools like VS Code Copilot can now reach beyond the codebase itself — querying internal documentation, checking server health, and even running operational playbooks, all from a chat window a developer already has open. For DevOps teams, this isn’t just a novelty. It’s a genuine shift in how day-to-day operational work gets done.

This post walks through a real architecture for bringing AI into a DevOps workflow: retrieval-augmented generation (RAG) for internal knowledge, an MCP server that safely executes Ansible playbooks against real infrastructure, a LiteLLM gateway that routes across multiple AI providers to control cost, and OpenBao for centralized credential management. We’ll cover why this combination works, and — just as importantly — how to secure it.

The Problem This Solves

Traditional DevOps tooling assumes a human is driving: reading a runbook, SSH-ing into a box, running a command, interpreting the output. AI assistants can now sit in that loop directly — a developer can ask “check disk space on the web servers” in plain language, and the assistant retrieves relevant context, decides which tool to call, executes it, and reports back — without ever leaving the editor.

The architecture that makes this safe and maintainable looks like this:

Three architectural decisions matter here, and they’re worth understanding before adopting this pattern yourself.

1. Retrieval and reasoning are separate concerns

The MCP server that handles document search doesn’t generate answers — it returns raw, relevant chunks and lets the LLM (via LiteLLM) do the reasoning. This keeps a single source of truth for how answers are synthesized, rather than having multiple components each doing their own generation with potentially inconsistent results.

2. Trust domains get separate containers, not separate scripts

A tool that searches a read-only knowledge base and a tool that executes commands against production servers should never share a process, a credential, or a blast radius. In practice, this means: one container for RAG (read-only, low risk), and a completely separate container for infrastructure operations (executes real commands, holds SSH credentials, higher risk). Scripts and playbooks that share a trust level live together in the same container — you don’t need a new container for every individual playbook, only for genuinely different risk tiers.

3. Every operational tool is allowlisted, never freeform

This is the detail that makes AI-driven ops safe rather than reckless: the AI never gets a “run arbitrary command” tool. Instead, each operational capability — checking disk space, checking service status — is its own explicitly defined function, backed by a specific, pre-written playbook and a fixed set of valid targets (inventory groups, not arbitrary hostnames). The LLM chooses which allowed operation to run and against which allowed target — never what to run.

The Advantages

Faster incident response. A developer can ask a natural-language question and get a synthesized answer pulling from both internal documentation and live infrastructure state, instead of manually checking three different systems.

Lower barrier to operational knowledge. Junior team members can query runbooks and system status conversationally instead of needing to already know where documentation lives or which playbook does what.

Auditable by design. Because every tool call goes through an explicit, named function — not a freeform shell — every action taken by the AI is logged, attributable to a specific person, and restricted to a pre-approved set of operations.

Extensible without re-architecting. Adding a new capability (say, checking service status or reading recent log errors) means adding one more allowlisted tool to an existing server — the trust model and the plumbing around it don’t change.

Controlling Cost with LiteLLM: Multi-Provider Routing

One underrated piece of this stack is the AI gateway sitting between the coding assistant and the actual model providers. LiteLLM is a proxy that exposes a single, OpenAI-compatible API while routing requests to whichever backend model you configure — OpenAI, Anthropic, Bedrock, local models, or others — based on rules you define.

This matters for cost in a few concrete ways:

Model-tier routing. Not every request needs your most expensive model. Simple classification or short lookups can route to a cheaper, faster model, while complex reasoning tasks route to a premium one. LiteLLM lets you define this per model alias without changing anything on the client side.

Automatic fallback. If your primary provider is rate-limited or down, LiteLLM can fail over to a secondary provider automatically, so a cost-saving choice of a cheaper primary model doesn’t come at the expense of reliability.

Centralized spend tracking and caps. Because every AI call in the organization — coding assistants, RAG embeddings, ops tooling — goes through one gateway, you get a single place to see total spend, set budgets per team or per key, and catch runaway usage before it becomes a surprise invoice.

Cheaper embeddings, separately routed from generation. In a RAG setup, embedding calls (turning a search query into a vector) are far more frequent and far cheaper per-call than generation calls. Routing them to a lightweight embedding model — while your main chat model stays on a stronger provider for reasoning — is a simple config change in LiteLLM, not a code change anywhere else in the stack.

A minimal example of what that routing configuration looks like:

model_list:
– model_name: chat-model
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_API_KEY

– model_name: embed-model
litellm_params:
model: openai/text-embedding-3-small
api_key: os.environ/OPENAI_API_KEY

Every downstream service — the coding assistant, the RAG server’s embedding calls — just references chat-model or embed-model by name. Swapping the underlying provider, adjusting fallback behavior, or capping spend happens in one place, with zero changes to the tools that consume it.

Securing the AI Infrastructure

This is the part that’s easy to underestimate. An AI agent that can execute infrastructure commands is, functionally, a new kind of privileged user — and it needs to be secured like one.

Authenticate every request, always. Every MCP server should require a valid bearer token on every call, with no exceptions for “trusted” internal traffic. A misconfigured network boundary shouldn’t be your only line of defense.

One identity per person, not one shared credential. Static, shared tokens make it impossible to answer “who did this” after the fact, and impossible to revoke access for one person without breaking it for everyone. Individual credentials, issued per person, solve both problems.

Centralize credential issuance and revocation. Rather than managing a scattered list of API keys across config files, a secrets manager (HashiCorp Vault, OpenBao, or similar) can issue tokens with built-in expiry, and validate/renew them on every request. Revoking someone’s access becomes a single API call that takes effect immediately — no redeploying a service to remove a hardcoded key.

Enforce role-based permissions at the tool level, not just the server level. Being authenticated should not automatically mean being authorized to do everything. A read-only role should be able to check disk space; only an admin role should be able to run anything destructive, if you ever add such a capability. Deny by default, and explicitly allowlist what each role can do.

Never expose a generic command-execution tool. This is the single most important rule. Every capability the AI has should be a specific, named, pre-written operation — never a raw shell, never an arbitrary “run this playbook” parameter. The AI should choose which approved action to take, never what to execute.

Bound and log everything. Timeouts prevent one hung request from blocking a tool indefinitely. Truncating output keeps a runaway response from flooding the model’s context. And logging every tool invocation — who called it, with what arguments, what happened — gives you the audit trail you’ll need when something needs investigating later.

Separate risk tiers into separate services. A compromised or misused read-only search tool and a compromised or misused infrastructure-execution tool are very different incidents. Keeping them as genuinely separate services, with separate credentials, keeps the blast radius of any single failure contained.

Closing Thoughts

AI in DevOps works best when it’s treated as a capable but constrained operator — given real, useful access to internal knowledge and infrastructure, but only ever through explicit, auditable, revocable channels. Pair that with a gateway like LiteLLM to keep costs predictable across multiple providers, and a proper secrets manager to handle identity, and what you get isn’t a novelty chatbot — it’s a genuinely useful addition to how your team operates infrastructure day to day.

The pattern scales, too: today it might be disk space checks; tomorrow it might be service restarts, deployment rollbacks, or log analysis — as long as each new capability goes through the same disciplined design: named, allowlisted, role-gated, and logged.