Docs / Operating

When the work jams

For a tenant administrator. Everything on this page — holding, stopping, replaying, discarding — is that rung’s, and a specialist clicking one is refused; see Administering a tenant.

For whoever is holding the pager. Failures that ran are the previous page, When something is wrong; this one is about work that is waiting, work that will not stop, and work the platform gave up delivering.

The Queue page answers two questions: what is waiting, and what is running right now.

What is waiting are the lanes. Work queues per tenant, and the page says so — the depths you see are the tenant’s, not this environment’s alone — so a deep lane means this tenant has a backlog, and the runs list tells you whose. A lane that only grows is the thing to investigate, not a number to watch.

What is running is a live list, and each row has a Kill. Use it for the run that will not end on its own. The child process really dies, and the run is recorded as killed — it shows up in the log like any other failed run, and it marks its script red.

The answer you get is what happened, never what was asked. A run that finished in the moment you pressed the button answers “Nothing was stopped” — and says why: it had already finished, or it is running somewhere this action cannot reach right now. The same honesty runs through the whole page: acting on a row that is no longer there tells you the screen was stale rather than pretending a second success.

Dead letters

A dead letter is a delivery the platform could not turn into a run at all: every attempt was made and not one of them got as far as starting the script (what “every attempt” means is on Limits and safety).

Read that sentence carefully, because it is narrower than it sounds. A script that ran and failed is not here — it has a run of its own, with its logs and its red mark, and you find it through When something is wrong. What lands in this list is the other kind of trouble: the platform itself could not deliver the work. So the list is a diagnosis record — something to look at when you are asking why deliveries are failing — and not a queue of work waiting to be picked up again.

That is why there is only one thing to do with an entry. Discard throws it away, and a dead letter is the last copy of its payload: the webhook that carried it was answered long ago, the sender will not send it again, and the platform keeps no other copy. The confirmation names exactly that. There is deliberately no button that puts one back in its lane — five deliveries already failed, and a sixth would fail the same way until whatever broke is fixed.

The list shows the entries belonging to this environment and states its horizon — an entry is kept for 7 days, then discarded. An empty list means nothing died inside that window, not that nothing ever has. If you want to hear about a dead letter the night it appears rather than find it here, that is the dlq.message occurrence on Reacting to failures.

Running an event again

Sometimes the work did run, wrongly — the script had a bug, the far end was down — and after fixing it you want the same event to run once more. Open the run the event caused and press Resubmit. The stored payload re-enters at the front door as a new event with its own identity, marked as a resubmission in the audit trail, and routed exactly as a fresh delivery would be — which also means a switched-off endpoint refuses it: the off switch keeps meaning no.

A run whose payload is no longer held answers that it cannot be replayed, rather than replaying something else.

If a waiting caller is involved — the endpoint answers a synchronous call — the caller’s side of the story (what it was told, and that its run kept going) is on Answering a caller.

Keeping a run

Every retention window eventually takes a finished run with it. Hold latches one out of that: a held run is kept until somebody releases it. Use it the moment a run looks like evidence — of a dispute, of a bug you will chase next week — and it will still be there when you get to it.

An automated browser test performs this page’s loop against a running platform, every time the product changes — a delivered event replayed and proven by the second run it causes, a run held, the lanes read with their owner named, a runaway run killed and really dying — so the page cannot outlive the behaviour it describes.