Docs / Operating

Reacting to failures

Hand-written. Unlike its generated neighbours in this folder, npm run docs will not touch this page. The function signatures it uses are generated, in function-reference.md.

An integration that nobody is watching is the normal case. It runs at two in the morning, it talks to a system that is occasionally down, and the first anybody hears about a bad night is a customer asking where their order went.

This page is about closing that gap. There are two ways, and most environments want both:

• a notification rule — the platform mails somebody. Nothing to write.
• a system trigger — the platform runs one of your scripts, handing it the failure. Anything you can write, you can do: open a ticket, post to a channel, call an ops API, retry something by hand.

What the platform reports

Five occurrences, and they are the same five wherever you choose them — in a notification rule, in a trigger, in the sample payloads.

“An entity is failing” is a transition, not a state. health.red fires when something goes from fine to failing — once. A script that then fails every ten minutes all night does not fire it again; it is already marked, and the mark stays until somebody acknowledges it. That is the difference between this and run.failed, which is one occurrence per failed run. Reading the mark and clearing it is its own page, When something is wrong.

Two things are true of all of them.

Only unattended work reports. Pressing Test in the editor never mails anybody and never runs a handler. The answer is already in front of you.

An occurrence carries no payload. It says what broke and where — the script, the run, the error’s own words — and never the order, the invoice or the customer record that caused it. If you need those, open the run: the occurrence tells you which one.

When runs get slower

metric.threshold is the one occurrence nothing reports on its own. Slow is not a fact about a run, it is a comparison against a number somebody chose — so the platform never guesses one. You set it on a notification rule:

• which time — the queue wait (how long work sat before it started), the working time (how long the script itself ran, with any suspension taken out), or the total (what a caller experienced, arrival to finish);
• which percentile — the ordinary case, or the slow tail;
• the limit, in milliseconds.

The measurement is taken once an hour, over the last day’s runs, and only over runs that finished — a run still going has no duration yet. Leave the rule’s script list empty and it watches the whole environment; name a script and it watches that script alone, which is what turns “something is slow” into “this is slow”.

Two things follow from measuring rather than reporting. An hour in which nothing finished crosses nothing — there is no measurement to compare, and the platform says nothing rather than reporting a zero. And a rule with no limit filled in watches nothing at all: it is saved, and it says so where it would have run.

Layer 1: mail somebody

Create a notification rule, choose the occurrences you want, and give it an address. That is the whole setup.

Two things about it are worth knowing before you rely on it.

It is throttled, always. 5 mails an hour by default, per rule. You can change the numbers; you cannot switch it off. This is not a limitation, it is the feature: a script that fails on every one of four thousand records overnight would otherwise send four thousand mails, and the fifth one would already have been ignored.

Nothing is hidden by the throttle. Everything held back is counted, and the next mail that does go out ends with the count and the time the silence started:

…and 340 more since 02:14 UTC, not sent to keep this mailbox usable.

So one mail never reads as one failure.

Where the mailbox comes from

A rule does not carry a mail server; it sends through an Email out entity — the same kind of configuration a sendMail from a script uses. Create one, give it its transport (an SMTP host and credentials, or a Microsoft 365 mailbox), and Activate.

With exactly one account in the environment, leave the rule’s account blank and it uses that one. With several, name the one this rule should send through.

With none, a rule usually cannot mail anybody — and whether it can is not yours to decide. The installation’s administrator can switch on a fallback that sends a tenant’s alerts from the platform’s own address when that tenant has no account of its own. It is off unless somebody turned it on, and it covers only this one case: a rule that NAMES an account which does not resolve, or names none where several exist, still refuses.

Either way this is the state worth checking for yourself, because a rule with no account still looks armed: it lists its occurrences, it is enabled, and it will report where it runs rather than where it was saved. If you have set up notification rules and never set up an Email out, ask your administrator whether that fallback is on before assuming anything has been mailed.

Layer 2: run a script

A system trigger is a trigger like any other. You add it to a main script, choose which occurrences it wants, and the platform runs that script when one happens — through the same pipeline a webhook goes through.

The occurrence arrives as the body:

const failure = input.event.body;

// { kind, tenantId, environmentId, scriptName?, runId?, taskId?, error?, metric?, at }

if (failure.kind !== "run.failed") return "not a failure I handle";

CreateTicket({
  summary: `${failure.scriptName} failed`,
  description: `${failure.error.code}: ${failure.error.message}\nRun ${failure.runId}`,
  priority: "high",
});

return `raised a ticket for ${failure.runId}`;

CreateTicket there is a configured function of yours — the handler is an ordinary script, so everything a script can do, it can do.

Watching one script instead of all of them

Only these scripts is the list of scripts whose failures this handler — or this notification rule — reacts to. Leave it empty and it hears about every script in the environment, which is the sane default for a first rule. Name scripts in it and it hears about those alone, which is how you give one integration its own reaction without every handler starting with an if.

It names scripts by their derived name — the same name a run records as its target and the log filters on.

Testing it before anything breaks

The Run panel offers a sample payload for each occurrence. Pick one and it fills the input; press Test. The samples are the real shape, so a handler that works against a sample works against the real thing.

The one rule you cannot turn off

A handler’s own failure never runs a handler.

If handle-failure breaks while reacting to a failure, that break is reported — it is mailed by your notification rules, it appears in the logs, it marks the script as failing — but it does not become another run of a handler.

One level of reaction, never a cascade. It is why you can write a handler that listens for run.failed with no filter and not think about it again: the thing that would loop cannot.

What to set up first

If you are starting from nothing, this order gets you the most for the least work:

1. One notification rule on run.failed and health.red, to an address a person actually reads. This is the whole point of the feature and it takes a minute.
2. A handler for the integration that matters most. Usually one script: raise a ticket, or post a message where the team already looks.
3. dlq.message when you start caring about throughput. Work that could not be delivered at all is rarer and more serious than a failed run — it means the platform never got to try properly. What a dead letter is, and why looking at one is the only thing to do with it, is on When the work jams.

You do not need a handler per script. One handler with a filter, or one handler that reads failure.scriptName and decides, is usually the whole of it.