Event Delivery for AI Agents: Webhook Idempotency, Cost Limits, and Audit Trails

Why Event Delivery for AI Agents Is a Different Problem

A traditional webhook consumer is dumb on purpose. Stripe sends payment_intent.succeeded, your handler flips a row in the database, done. The handler does exactly one predetermined thing. If each event takes effect exactly once — at-least-once delivery plus a duplicate check — you've solved the problem.

An AI agent is not dumb on purpose. You deliver an event, and the agent decides what the event means and which tools to call in response. It might issue a refund, open a pull request, message a customer, spin up infrastructure, or call three more tools whose outputs feed back into its next decision. The same event delivered twice doesn't just write a row twice — it can launch two independent chains of actions that diverge.

So the question "did the event arrive?" is no longer the hard part. The hard parts are: did it arrive exactly once, did the agent's response stay inside the limits you set, and can you reconstruct what it did afterwards? Those are delivery problems, governance problems, and audit problems bundled into one. That bundle is what we mean by event delivery for AI agents.

What Webhook Idempotency Means

An operation is idempotent when running it multiple times produces the same result as running it once. Webhook idempotency applies that to incoming webhook requests: if the same event reaches your handler twice, the second delivery attempt changes nothing. The customer is charged once, the email goes out once, the row is updated once.

You need it because webhook providers deliver at-least-once, not exactly-once. Provider retries fire on timeouts, on network issues, and on any non-2xx status code — including the case where your handler finished the work but the response never made it back. Stripe, for example, keeps retrying a failed webhook for up to three days in live mode. From the provider's side, "I never got a 200" and "the work never happened" look identical, so it sends the event again. Duplicates are not an edge case; they are part of the contract.

The fix is always the same shape: give every event a stable identifier, remember which identifiers you have processed, and make the check and the side effect succeed or fail together.

The Three Ways It Fails

1. Duplicate actions from duplicate events

At-least-once is the default contract for almost every event source on the planet. Sources retry on timeout, on a 5xx, on a network blip. Your handler that took 31 seconds when the source's timeout was 30 will receive the same event again — and the source has no idea the first one actually succeeded.

For a dumb consumer, a duplicate is an annoyance you fix with an idempotency key. For an agent, a duplicate is a second autonomous run. If the event was "customer requested a refund," the naive outcome is two refunds. If it was "deploy the main branch," it's two deploys racing each other. The agent has no memory that it already handled this event two minutes ago unless something outside the agent enforces it.

2. Cost and rate spikes

This is the failure mode that has no equivalent in classic webhook delivery. An agent that loops — reads an event, calls a tool, the tool emits another event, which the agent reads again — can run up a bill in minutes. Every tool call might be an LLM token spend, a paid API hit, or a real-money transaction. A bad prompt, a confused plan, or a malformed event can turn a single trigger into hundreds of downstream actions.

Rate limiting the inbound events is necessary but not sufficient, because the danger is on the outbound side — what the agent does per event. The control you actually want is a budget: a hard ceiling on spend or action count per agent, per window, enforced before the action fires, not discovered on next month's invoice.

3. Silent failures you can't reconstruct

When a normal webhook handler fails, you get a stack trace. When an agent does the wrong thing, you often get nothing — it "succeeded" at calling a tool, the tool did something subtly wrong, and there's no exception anywhere. Three weeks later someone asks "why did the agent close that account?" and the honest answer is that nobody can say, because the decision lived in a model's context window that no longer exists.

The only defense is an immutable record written at the boundary: which event came in, which agent received it, what it decided, which tools it called, what the policy verdicts were, and what came back. If that record requires unbundling a log archive from cold storage, no one will ever read it. It has to be queryable in seconds.

How to Implement Webhook Idempotency

Whether the consumer is a plain handler or an agent, the idempotency layer looks the same. Four rules cover most real systems:

1. Key on the provider's event ID. Stripe sends an id like evt_… in the webhook payload; GitHub sends an X-GitHub-Delivery header. Use that as the idempotency key, scoped by provider. A hash of the request body is a weak fallback: two genuinely different events can have the same body, and you lose the link back to the provider's own logs.
2. Enforce it with a unique constraint, not a lookup. "Check whether we've seen this ID, then process" has a race: two concurrent deliveries both pass the check. A PRIMARY KEY or UNIQUE NOT NULL constraint in the database makes the second insert fail with a constraint violation instead.
3. Record the key in the same database transaction as the business logic. If you store the key first and the handler crashes, the retry is skipped and the work never happens. If you do the work first and then crash, the retry does it again. One transaction removes both gaps.
4. Acknowledge duplicates; don't fail them. A duplicate is not an error. Answer a provider's webhook with a 2xx, because any other status makes it retry an event you already handled. For your own API clients, a separate status such as 409 lets the caller tell a duplicate apart from a failure.

In PostgreSQL the whole pattern is one table and one INSERT:

CREATE TABLE processed_webhooks (
  provider    text        NOT NULL,
  event_id    text        NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (provider, event_id)
);

And the handler (Node.js with pg, Express-style):

app.post("/webhooks/stripe", async (req, res) => {
  const event = verifyStripeSignature(req); // raw body + signing secret
  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    const { rowCount } = await client.query(
      `INSERT INTO processed_webhooks (provider, event_id)
       VALUES ('stripe', $1) ON CONFLICT DO NOTHING`,
      [event.id]
    );
    if (rowCount === 0) {            // seen before: acknowledge, do nothing
      await client.query("ROLLBACK");
      return res.sendStatus(200);
    }
    await applyBusinessLogic(client, event); // same transaction as the key
    await client.query("COMMIT");
    res.sendStatus(200);
  } catch (e) {
    await client.query("ROLLBACK");
    throw e;                         // non-2xx, so the provider retries
  } finally {
    client.release();
  }
});

Concurrency is handled by the database: if two deliveries of the same event arrive together, the second INSERT waits on the unique index until the first transaction commits, then conflicts and does nothing.

The case this pattern exists for is the one where the work succeeded but the provider never saw the 200:

Two practical notes. First, keep keys at least as long as the provider's retry window; if you prune them sooner, a late retry looks like a new event. A Redis SET NX with a TTL works as a fast pre-check, but it isn't part of your database transaction, so keep the constraint as the source of truth. Second, idempotency stops at your database. If the business logic calls another API — issuing a refund, sending an email — pass an idempotency key downstream too. Stripe accepts an Idempotency-Key header on POST requests for exactly this.

That handles a plain webhook consumer. An agent needs one more layer, because the risky part is not the event itself but what the agent decides to do with it.

The Guardrails That Contain All Three

Notice that none of these are fixed by making the agent smarter. They're fixed by putting a deterministic layer between the event source and the agent — one that does the boring, non-negotiable parts the agent can't be trusted to remember:

• Idempotent delivery. Every inbound event carries a stable identity. The same event is admitted once; retries and replays are recognized and collapsed, so a duplicate can never become a duplicate action.
• Budget & rate gates. Each agent runs under its own credentials and its own ceiling. When the budget or rate limit is hit, the action is blocked before it executes, with a clear reason — not after the spend.
• Approvals & rollback. Sensitive actions — payouts, deletes, deploys — can require human-in-the-loop approval. Multi-step sequences run as atomic bundles: if step three fails, steps one and two roll back instead of leaving the world half-changed.
• An audit trail by default. Every event, decision, policy verdict, and action is recorded immutably and stays inspectable. Reconstruction stops being archaeology.

In AgentDelivery, the first two layers work like this. Every inbound event needs a key: an Idempotency-Key or X-Event-Key header, or an event_key field in the JSON body. A unique index on the source and key admits each event once. A repeat gets 409 duplicate_event_key with the ID of the original event, so the sender can tell it was a duplicate. It is recorded in the log and not counted against your event quota. On the action side, an agent calls execute with its decision ID as the idempotency key. If the same decision is submitted again, it gets the stored result of the first run back instead of a second execution.

You can build all of this yourself. It is, roughly, the same project as building reliable webhook delivery — queues, idempotency, state machines, retries, observability — plus a policy engine, per-agent identity, and an approval workflow on top. Our earlier post on how to build a reliable webhook delivery system covers the delivery half; the agent half roughly doubles the surface area, and it's the half where the failures cost real money.

Where This Leaves You

Reliable event delivery for AI agents is not "webhooks, but for AI." It's webhook delivery plus the guardrails that keep an autonomous actor from turning one event into a runaway chain of expensive, irreversible, unexplainable actions. Idempotency stops the duplicates. Budget gates stop the cost spikes. Approvals and bundles stop the irreversible mistakes. An audit trail makes the whole thing accountable.

If you'd rather understand the model before deciding to build or buy, the architecture page walks through how delivery, policy, execution, and audit fit together as one system instead of four bolted-on layers.

Webhook Idempotency FAQ

Is exactly-once webhook delivery possible?
No. Over HTTP, a sender can't tell a lost response from a lost request, so every provider has to retry, and retries mean duplicates. What you can guarantee is exactly-once effect: at-least-once delivery plus webhook idempotency on the receiving side.

What should I use as the idempotency key?
The event ID the provider assigns, such as Stripe's evt_… ID or GitHub's X-GitHub-Delivery header, scoped by provider. Only fall back to a hash of the payload when the provider sends no ID at all.

How long should I keep processed event IDs?
At least as long as the provider's longest retry window, plus a margin for manual replays. For Stripe that means more than three days; many teams simply keep them for 30 days or more.