Silent webhook failures

The worst failure in a form pipeline isn't an error. It's the week where everything reports success and the leads don't arrive — where "accepted" quietly stops meaning "delivered," and you find out because a customer asks why nobody called them back.

Nothing alerts, because from each component's point of view nothing went wrong. The form stored the response. The sender got a 2xx. The dashboard is green. The row is missing anyway.

This is a design problem with known answers. Here's where deliveries actually go, what a tool has to give you to make the gap visible, and how to check any vendor's story before you depend on it.

How a delivery goes missing without an error

Your receiver acknowledged before it committed. The most common one. The endpoint returns 200, then does the work — writes to the database, calls the CRM. If that work fails after the acknowledgment, the sender has been told to forget the event. It's gone, and both sides believe it succeeded.

Something in front of your app answered for it. A CDN, load balancer, WAF, or API gateway can return 200 to a request your application never processed, or absorb it during a deploy. The delivery log shows success because something did succeed — just not your handler.

Retries ran out quietly. Every sender gives up eventually. If exhaustion is only a status on a row nobody reads, the system has an error state with no reader.

The webhook was turned off or removed. Some platforms disable a destination automatically after consecutive failures — a reasonable protection that becomes a silent outage when the notification goes to an unmonitored address. A deleted destination can also take its delivery history with it, so the evidence disappears along with the deliveries.

A credential expired. OAuth-based destinations stop working when a token is revoked or a password changes. The failure is per-destination and often looks like "no new rows" rather than an error.

Deduplication ran on the wrong key. If you dedupe on the record ID rather than a per-delivery ID, a legitimate second event about the same record — an edit, a status change — looks like a duplicate and is dropped on purpose.

The receiver crashed mid-processing. Claim the event, start work, crash before finishing. The sender saw an acknowledgment. The work never completed. No retry is coming.

Notice how many of these are on the receiver's side. A vendor with excellent delivery semantics can still lose your data if your endpoint acknowledges too early. Both halves have to be right.

What good delivery semantics look like

This is the specification to hold any tool against:

• Signed payloads, verified against raw bytes. An HMAC over the exact request body, with the documented instruction to verify before parsing. Parsing and re-serializing changes the bytes and breaks the signature — a good doc says so explicitly.
• A stable delivery ID, separate from the record ID. It must stay constant across retries of the same delivery and differ between two events about the same record. This is the idempotency key, and it's the single most important field in the payload.
• At-least-once delivery, stated as such. Any honest system that retries will sometimes deliver twice. A vendor that implies exactly-once is either wrong or hiding a queue you can't see.
• A published retry schedule and timeout. How many attempts, at what intervals, and what counts as a failure — a non-2xx status, a timeout, and at what threshold.
• Alerting on exhaustion, to a human. A status field is not an alert. Something has to leave the system when retries run out.
• A delivery log with attempts and status codes. Per delivery: attempt count, last response status, next scheduled retry, and the error. With a stated retention window, so you know how far back you can look.
• Transport rules you can audit. HTTPS-only in production, no redirect following, and refusal to deliver to internal or private addresses. Following redirects turns a webhook into a request-forgery primitive.
• A safe manual retry. A way to re-drive failed deliveries after you fix the receiver — and clarity about whether it re-sends to destinations that already succeeded.

Audit any tool with these questions

Answerable from documentation. If they aren't, that's the finding:

1. Is the payload signed, and is the signature computed over the raw body?
2. Is there a delivery ID that's stable across retries and distinct from the response ID?
3. How many retry attempts, at what intervals?
4. What counts as a failed delivery — status codes, timeout duration?
5. What happens after the last attempt: alert, disable, or nothing?
6. Who receives the alert, and can that address be a shared inbox or channel?
7. Can I see per-delivery attempt history, and for how long?
8. Can I re-drive a failed delivery after fixing my endpoint?
9. Does the tool follow redirects, and does it allow non-HTTPS or private destinations?
10. If I delete a destination, does its delivery history survive?

Then run the test that documentation can't answer: point the webhook at an endpoint that returns 500, submit a response, and watch what the tool does over the next few hours. You'll learn more from that than from any feature page.

Make your receiver worth the guarantee

At-least-once delivery only helps if the receiver is built for it. The durable-inbox pattern:

1. Verify the signature against the raw bytes. Reject if it fails.
2. Insert the delivery ID and the verified payload into a table with a uniqueness constraint on the delivery ID. Duplicate IDs are a no-op.
3. Return 2xx only after that insert commits. If storage fails, throw — let the sender retry.
4. Process the inbox in a separate worker, with its own retry policy.

The critical rule is the ordering in steps 2 and 3. Never claim a delivery ID and then start uncommitted work: a crash between those steps loses the retry, which is exactly failure mode seven above. A simple handler can collapse this by writing the delivery ID and its domain change in one transaction.

Then monitor the inbox, not the endpoint. "No deliveries in the last hour on a form that normally gets ten" is the alert that catches everything else.

What Fillo does

Stated from the docs, so you can check each one.

Signing and identity. Each delivery POSTs JSON with X-Fillo-Signature, a hex HMAC-SHA256 over the raw request body keyed with the webhook's signing secret, and X-Fillo-Delivery-Id, which stays stable across retries of one delivery and is the intended idempotency key. X-Fillo-Event carries response.created, response.updated, or draft.abandoned. The docs are explicit that you should not dedupe on the response ID, because one response can emit both a created and a later updated delivery.

Retries. A non-2xx status or a 10-second timeout counts as a failure. Failed deliveries retry with backoff — roughly 1 minute, 5 minutes, 30 minutes, 2 hours, then 6 hours — up to 6 attempts before the delivery is marked failed. Delivery is at-least-once, and the webhook guide says so directly rather than implying otherwise.

Transport. Production webhook URLs must use HTTPS. Requests never follow redirects, and Fillo rejects embedded URL credentials, private or internal destinations, and DNS results that include private addresses.

Visibility. A form's Activity page shows one dominant status per destination — Needs attention, Retry scheduled, Healthy, or No activity — and a Recent deliveries list of the latest 50 handoffs with status, attempt count, last HTTP status, next scheduled retry, and the bounded error. Delivery rows are retained for up to 90 days. A later success doesn't hide an older undelivered response: the destination stays Needs attention until that row succeeds, is retried, or its target response is deleted.

Alerting. When retries are exhausted or an OAuth credential stops being usable, Fillo emails the form's notification address, or the workspace owner if none is set, at most once per form and destination per display day while a backlog remains. The alert names the destination, affected count, failure age, and latest error, and never includes response answers.

Repair. Retry now on a destination advances scheduled retries and resets terminally failed rows; Retry failed in the grid requeues only failed destinations for selected responses; the response drawer's Re-send deliberately sends one response to every current destination again, including ones that already delivered.

Three limits worth stating in the same breath. Delivery health and retry are management operations on the Activity page — they aren't exposed through the public renderer client or the current Management API. Removing a webhook removes its delivery rows, so that history can disappear before the 90-day ceiling. And a retry can duplicate downstream work: Sheets and Notion probe for an already-committed record, but no system makes every external side effect exactly once, which is why your receiver still needs the inbox above.

Related

• Webhooks: Headers, payload shapes, signature verification code, and the retry schedule.
• Delivery health and retry: Read destination status, inspect deliveries, and retry safely.
• Webhook signature failed: Fix a verification mismatch.
• Integrations: Configure Google Sheets, Notion, Slack, or Zapier destinations.
• Support intake example: A live form whose responses flow through these delivery paths.