Cloud Provisioning With FlowChassis Tears Down What a Failed Deploy Built
Putting a site on a new server takes a handful of calls to different providers. Create the VM. Point a DNS record at it. Get a TLS certificate. Then wait until the thing answers its health check, which can take a while because VMs take time to boot.
When the certificate step fails, you’re left with a VM you pay for by the hour and a DNS record pointing at it. Run that a few hundred times across a team and you get the cloud bill everyone recognises: servers nobody remembers creating.
This case study is a complete program in the repository. Three sites go through the same flow, and each ends a different way: completed, compensated, and failed with a clear note of what’s left.
The Flow
Four steps. Each one that creates something saves its ID in the run’s state, and its undo uses that ID to remove it:
{
Name: "add_dns",
Do: func(c *flowchassis.Ctx) error {
rec, err := cloud.DNS.Create(c, "A record "+site(c).Domain+" pointing at "+get(c, "vm"))
if err != nil {
return err
}
c.Set("dns", []byte(rec))
return nil
},
Undo: func(c *flowchassis.Ctx) error {
return cloud.DNS.Do(c, "remove "+get(c, "dns"))
},
// Also the policy for the undo: three tries to remove the
// record, then the run stops and waits for a person.
Retry: flowchassis.RetryPolicy{MaxAttempts: 3, Backoff: 50 * time.Millisecond},
},
The health check is where retries earn their keep. A fresh VM refuses connections until it’s up, so the step keeps trying, waiting longer each time:
{
Name: "health_check",
Do: func(c *flowchassis.Ctx) error {
return cloud.Compute.Do(c, "GET https://"+site(c).Domain+"/health")
},
// A VM takes a while to boot: keep checking, waiting longer
// each time, for up to about half a minute.
Retry: flowchassis.RetryPolicy{MaxAttempts: 12, Backoff: 100 * time.Millisecond, Multiplier: 2, MaxBackoff: 5 * time.Second},
},
The wait between checks is in the journal. If the program restarts during it, the wait goes on from the time recorded there and doesn’t start over.
What Happens
The program’s output, with only the closing lines about the journal’s path left out:
shop.example.com:
compute create small VM for shop.example.com -> compute-1
dns A record shop.example.com pointing at compute-1 -> dns-1
ca certificate for shop.example.com -> ca-1
compute GET https://shop.example.com/health -> error: connection refused, still booting
compute GET https://shop.example.com/health -> error: connection refused, still booting
compute GET https://shop.example.com/health -> done
=> completed
blog.example.net:
compute create small VM for blog.example.net -> compute-2
dns A record blog.example.net pointing at compute-2 -> dns-2
ca certificate for blog.example.net -> error: CAA record forbids this CA for the domain
dns remove dns-2 -> done
compute destroy compute-2 -> done
=> compensated (step issue_cert: CAA record forbids this CA for the domain)
legacy.example.org:
compute create large VM for legacy.example.org -> compute-3
dns A record legacy.example.org pointing at compute-3 -> dns-3
ca certificate for legacy.example.org -> error: CAA record forbids this CA for the domain
dns remove dns-3 -> error: DNS provider API: 503 service unavailable
dns remove dns-3 -> error: DNS provider API: 503 service unavailable
dns remove dns-3 -> error: DNS provider API: 503 service unavailable
=> failed (undo add_dns: DNS provider API: 503 service unavailable)
left for a person: vm=compute-3 dns=dns-3
The shop comes up after two refused health checks. Nobody wrote a polling loop.
The blog’s certificate is refused because the domain’s CAA record doesn’t allow that CA. Retrying won’t change a DNS policy, so the certificate step checks for that answer and wraps it in flowchassis.Permanent, and the run compensates at once:
cert, err := cloud.CA.Create(c, "certificate for "+site(c).Domain)
if errors.Is(err, errCAARefused) {
// A DNS policy won't change on a retry.
return flowchassis.Permanent(err)
}
``` Order matters here. The record is removed first, then the VM it points at is destroyed: newest first, the reverse of how they were built.
## When the Cleanup Fails Too
The third site is the case most tools leave out. The certificate is refused, compensation starts, and the DNS provider's API is down. The undo is tried three times, as its policy says, and fails each time.
At that point FlowChassis stops. It doesn't skip ahead and destroy the VM anyway. Destroying a server while a DNS record still points at its address is how you end up with a name pointing at an IP that someone else gets next. The safe thing is to leave both and say so.
The run ends `failed`. In FlowChassis that status means compensation couldn't finish and a person needs to look. The run's state still holds the IDs of what was built, so the person knows exactly what to clean up. The `flowchassis` command shows the whole story, with times:
```text
$ flowchassis show provision.journal legacy.example.org
run legacy.example.org
flow provision
status failed
error undo add_dns: DNS provider API: 503 service unavailable
started 2026-10-18T12:43:57+03:00
ended 2026-10-18T12:43:57+03:00
done create_vm, add_dns
input {"domain":"legacy.example.org","size":"large"}
state
dns = dns-3
vm = compute-3
history
12:43:57.104 started provision
12:43:57.105 create_vm: attempt 1 began
12:43:57.105 create_vm: done, next add_dns
12:43:57.105 add_dns: attempt 1 began
12:43:57.106 add_dns: done, next issue_cert
12:43:57.106 issue_cert: attempt 1 began
12:43:57.106 issue_cert: attempt 1 failed for good, compensating: CAA record forbids this CA for the domain
12:43:57.106 undo add_dns: attempt 1 began
12:43:57.106 undo add_dns: attempt 1 failed, retry at 12:43:57.156: DNS provider API: 503 service unavailable
12:43:57.157 undo add_dns: attempt 2 began
12:43:57.158 undo add_dns: attempt 2 failed, retry at 12:43:57.258: DNS provider API: 503 service unavailable
12:43:57.258 undo add_dns: attempt 3 began
12:43:57.259 undo add_dns: attempt 3 failed for good, run failed: DNS provider API: 503 service unavailable
What To Take From It
A deploy script that only handles success leaves a mess on failure. One that tries to handle every failure turns into the bigger part of the codebase. Here the flow says what each step builds and how to remove it. Whatever goes wrong, the run ends in a state you can read, and when it can’t clean up by itself it says exactly what’s left.
go run ./examples/provision
The account signup case study shows the same pattern on a smaller scale, and the video transcoding case study is about the other kind of failure: the process itself dying in the middle.