Webhook Delivery With FlowChassis Backs Off Between Retries and Dead-Letters What Fails

If your product sends webhooks, you’ve built a small delivery system whether you meant to or not. Customers’ endpoints go down, answer slowly, return errors for an hour during their own deploy, or get deleted without anyone telling you. Each event has to get there once, eventually, or end up somewhere a person can see it.

The rules are well known. Retry on 5xx and timeouts, waiting longer each time. Stop on a 410 Gone. Sign every request so the receiver can trust it. Send the same delivery ID on every attempt so the receiver can drop a duplicate. And when you give up, put the event in a dead-letter queue instead of losing it.

This case study is a complete program in the repository that does all of that in one flow of four steps. It starts a real HTTP server with three customer endpoints and sends each one an order.paid event.

The Flow

The steps are queue, sign, post and mark_delivered. The interesting one is post. It sends the request and sorts the answer into three kinds: delivered, try again, or stop.

req.Header.Set("Webhook-Signature", string(sig))
// The same on every attempt, so the receiver can drop a
// delivery it has already processed.
req.Header.Set("Webhook-Id", c.Run)
resp, err := client.Do(req)
if err != nil {
	return err // network trouble: try again
}
resp.Body.Close()
switch {
case resp.StatusCode < 300:
	return nil
case resp.StatusCode == 408 || resp.StatusCode == 429 || resp.StatusCode >= 500:
	return fmt.Errorf("endpoint answered %s", resp.Status)
default:
	// 400, 404, 410: the endpoint won't take it, ever.
	return flowchassis.Permanent(fmt.Errorf("endpoint answered %s", resp.Status))
}

A plain error means “try again later”. An error wrapped in flowchassis.Permanent means “don’t bother”. The schedule for “later” is one line on the step:

Retry: flowchassis.RetryPolicy{MaxAttempts: 6, Backoff: 50 * time.Millisecond, Multiplier: 2, MaxBackoff: time.Second},

The example uses milliseconds so it finishes in two seconds. A real service would start at a few seconds and cap at an hour or so. Each wait is written to the journal with its due time, so a restart in the middle of a long backoff doesn’t reset the clock or lose the event.

The dead-letter queue is compensation. The first step, queue, has an undo, and that undo is what puts the event in the dead-letter queue:

{
	Name: "queue",
	Do: func(c *flowchassis.Ctx) error {
		return outbox.Do(c, "queue "+c.Run)
	},
	// Compensation for a delivery that failed for good.
	Undo: func(c *flowchassis.Ctx) error {
		return deadLetter.Do(c, "dead-letter "+c.Run)
	},
},

When delivery fails for good, whether from a 410 or from running out of attempts, the run compensates. Undoing queue moves the event to dead-letter. There’s no separate failure handler to remember to call.

What Happens

The program’s output, with only the closing lines about the journal’s path left out. The times vary a little from run to run:

evt-1001-acme:
  outbox     queue evt-1001-acme -> done
  /acme      request 1 for evt-1001-acme at +0s -> 503 Service Unavailable
  /acme      request 2 for evt-1001-acme at +50ms -> 503 Service Unavailable
  /acme      request 3 for evt-1001-acme at +160ms -> 200 OK
  outbox     delivered evt-1001-acme -> done
  => completed
evt-1001-globex:
  outbox     queue evt-1001-globex -> done
  /globex    request 1 for evt-1001-globex at +0s -> 410 Gone
  deadletter dead-letter evt-1001-globex -> done
  => compensated (step post: endpoint answered 410 Gone)
evt-1001-initech:
  outbox     queue evt-1001-initech -> done
  /initech   request 1 for evt-1001-initech at +0s -> 500 Internal Server Error
  /initech   request 2 for evt-1001-initech at +50ms -> 500 Internal Server Error
  /initech   request 3 for evt-1001-initech at +150ms -> 500 Internal Server Error
  /initech   request 4 for evt-1001-initech at +360ms -> 500 Internal Server Error
  /initech   request 5 for evt-1001-initech at +760ms -> 500 Internal Server Error
  /initech   request 6 for evt-1001-initech at +1.56s -> 500 Internal Server Error
  deadletter dead-letter evt-1001-initech -> done
  => compensated (step post: endpoint answered 500 Internal Server Error)

Acme was having a bad moment and recovered. Globex said the endpoint is gone, so there was one request and no pointless retries. Initech is down. Look at the times: the gaps go 50, 100, 200, 400 and 800 milliseconds, doubling each time, and after the sixth attempt the event goes to dead-letter where someone can replay it once Initech is back.

The receivers in the example check the signature with each customer’s shared secret, the way a real customer would. Acme’s 200 also proves the signature was right.

The secrets themselves stay out of the journal. A run’s input is written there, and flowchassis show prints it, so the input only names the customer. The sign step looks the secret up when it runs.

Why the Delivery ID Matters

A webhook sender can’t know for certain whether a request got through. The receiver can process the event and then the connection drops before the 200 arrives. Or the sending process dies right after the request went out. Either way the sender tries again, and that’s correct, because the alternative is losing events.

So a system like this delivers at least once, and so does FlowChassis. What makes it safe is the Webhook-Id header. It’s the run’s ID, the same on every attempt and after any restart, so the receiver can keep a list of IDs it has processed and drop a repeat. The engine gives you that stable ID without any extra work.

What To Take From It

The delivery policy lives in one place and fits on a screen. The engine keeps each event’s progress in the journal, so a deploy or crash during a backoff picks up where it was.

go run ./examples/webhook
flowchassis show /tmp/.../webhook.journal evt-1001-initech

The show command lists every attempt with its time and the time of the next one. The other case studies cover account signup, cloud provisioning and video transcoding.