Credentials, trust and the AI operator problem

When you give an AI agent an API key, you're making a decision that most developers treat as routine. You add the key to an environment variable, maybe log a note about what it accesses, and move on. The agent needs the credential to do its job. That's the end of the analysis.

This works when you're the one running the agent, on your own infrastructure, in a system you fully control. It breaks down as soon as you introduce a third party into the picture — and in the world of AI orchestration, the third party is almost always there.

Quick answer

In a multi-agent system, credentials belong to the deployer — not the builder or the runtime. The builder declares what credentials are needed; the deployer supplies the actual values; the runtime holds them during execution. The trust problem is ensuring each party sees only what it needs: the builder sees that a credential is required, the deployer controls the value, and the runtime uses it without exposing it to the agent's prompt.

The operator relationship

Consider the structure of a deployed AI team. A builder designs the team — defines the agents, writes the prompts, declares what secrets will be needed. A deployer installs that team into their own environment and supplies the actual credentials: the Zendesk API key, the Slack bot token, the database connection string. A runtime executes the team, which means it actually handles the credentials during execution.

Three parties, one secret. The builder declared that the secret is needed. The deployer provided it. The runtime holds it while the agents run.

This is the operator problem: in a multi-party system, who holds credentials, under what conditions, and with what visibility to each party?

The naive answer is that the deployer controls everything — they supplied the credentials, they should control them. But that's not quite right either. The deployer installed a team whose agents were defined by the builder, and those agents will use the credentials in ways the deployer may not have anticipated. If the builder's prompt for an email agent is aggressive about sending volume, the deployer's email API key takes the consequences.

The builder's choices determine how credentials are used. The deployer's credentials determine what's possible. The runtime determines what's actually enforced. These are three separate levers held by three different parties.

What the credential vault gets right

The credential vault approach — where secret values are encrypted at rest, never returned via API, and only decrypted at the moment an agent needs them — solves the storage problem. Encryption at rest, values that can be written but never read back, audit logs showing when secrets were accessed and by which agent. This is table stakes for any serious orchestration platform.

But the vault solves a narrower problem than the operator problem. Securing storage is necessary, but it doesn't answer the harder question: what does the agent actually do with the credential once it has it?

An API key that's encrypted at rest but used by an agent with unrestricted outbound access is still an API key that can be used for anything the key permits. The vault protects against exposure of the credential itself. It doesn't protect against the agent using the credential in unexpected ways.

Access policy as the second layer

This is where the access policy matters. When an agent declares that its outbound access defaults to deny, with explicit allow rules for the specific hosts it needs to function, it becomes possible to reason about what the credential can actually reach.

An email agent with access to api.sendgrid.com and nothing else is an agent whose credentials are scoped by both the API key's permissions and the agent's declared access constraints. Even if the agent's behaviour is somehow compromised — if a prompt injection causes it to try to exfiltrate data to an external service — the runtime can block that request at the policy layer.

The access policy is a trust signal in both directions. For the deployer evaluating a team before installing it: a policy that names specific hosts is a legible claim about what the team does and doesn't do. For the operator managing deployed teams: the access logs show exactly which requests were allowed and which were blocked, by policy, per agent.

Without a declared access policy, the credential is available to the agent and the agent is available to anywhere. With one, the credential's reach is bounded by what the policy permits.

Human gates as the high-stakes control point

Credentials aren't just for reading data. They're also for taking actions — sending email, processing payments, posting to social media, modifying records. These are the actions where a mistake is hard to reverse, where the cost of a wrong decision is measured in real consequences, not debug logs.

Human gates are the appropriate tool for these situations. Not because AI agents can't make these decisions — often they can — but because the reversibility calculus is different. An agent reading data and making a wrong inference can be corrected. An agent sending an email to the wrong person, initiating a payment, or posting publicly before a human has seen the output creates a situation that can't simply be undone.

The design question is where in the pipeline to place the gate. Too early and you're reviewing everything, which defeats the purpose of automation. Too late and the irreversible action has already happened. The right placement is at the moment of escalation from reversible to irreversible — the step before the agent uses a credential to take an action in the world.

A team that declares this boundary explicitly — here is where human review happens, here are the fields the reviewer will see, here is what happens if they reject a record — is a team that has thought about trust. The structure of the gate is itself a claim about where the system trusts itself and where it defers.

The trust question that doesn't go away

There's a version of the operator problem that no technical mechanism fully solves: what happens when the builder's definition of what the agents do and the deployer's understanding of what they're getting don't match?

Good documentation helps. Public access policies help. Changelogs that describe what changed between versions help. But ultimately, a deployer who installs a team written by someone else is extending trust — to the builder's design choices, to the runtime's enforcement guarantees, to the platform's credential handling.

The mechanisms exist to make this trust tractable: declared access policies that can be read before installation, credential handling that doesn't expose secrets at rest, human gates for high-stakes actions, audit logs for post-hoc review. These don't eliminate the trust question. They make it a question about structure rather than a question about faith.

Trust in AI systems isn't different from trust in any complex software — it's earned through legibility and confirmed through accountability. The difference is that AI agents can move fast and operate with considerable autonomy, which makes the legibility requirement more urgent and the accountability requirement more explicit.

The operator problem, at its core, is a legibility problem. Make the system legible — its access constraints, its decision points, its credential usage — and trust becomes a question that can be answered rather than one that has to be assumed.

Design your AI agents in Envelope

Envelope turns a plain-language description of your workflow into a complete AI agent system — agents with named roles, model assignments, tool access, handoffs, and human review gates. Free to start, no code required.

Start designing →

Frequently asked questions

What is the AI operator problem?

It's the question of who holds credentials in a multi-party AI system, under what conditions, and with what visibility to each party — the builder who declares a secret is needed, the deployer who supplies it, and the runtime that handles it during execution.

Is a credential vault enough to keep AI agent credentials secure?

No. A vault solves the storage problem — encryption at rest, no plaintext read-back — but it doesn't limit what an agent actually does with a credential once it has it. That requires a separate access policy layer.

What does an access policy do that a credential vault doesn't?

An access policy that defaults to deny, with explicit allow rules for specific hosts, bounds what a credential can actually reach — even if the agent's behaviour is somehow compromised, the runtime can block requests outside the policy.

Where should human approval gates sit when agents use credentials to take action?

At the moment of escalation from reversible to irreversible — the step right before a credential is used to send an email, process a payment, or post publicly. Too early and you're reviewing everything; too late and the action already happened.

Why can't technical mechanisms alone solve trust between AI team builders and deployers?

Because trust ultimately depends on whether the builder's definition of what the agents do matches the deployer's understanding of what they're getting. Access policies and audit logs make that trust tractable, but a deployer installing someone else's team is always extending some trust.

What makes an AI system's credential handling trustworthy?

Legibility: declared access policies readable before installation, credential handling that never exposes secrets at rest, human gates for high-stakes actions, and audit logs for post-hoc review.

---

→ Access policy reference — how to declare per-agent network access controls in an Envelope spec
→ Export your spec — what travels in the .envelope.json file and what stays in the vault