a9script

Docs / Data & bulk sync

a9script authoring pack — datasets functions

The datasets functions, with their arguments and a line of real script each. Called by bare name; nothing is imported. A call marked pauses does real work outside the script: the run is saved, and continues on the next line with the result.

This is one part of the authoring pack — how a script is executed at all is the model part, and the other groups of functions are parts of their own. All of them are named at the end.

Generated — do not edit. Every block of script below is executed by the platform’s own test suite, and every error message is the one the platform produces today, so none of this can be what was true when someone last wrote it down.

dsPut(dataset, key, value) — pauses

Stores one keyed row in a named dataset — the platform-side staging ground for bulk data. Writing an existing key replaces that row’s value.

  • dataset (string) — Which dataset. A name starting run: is PRIVATE TO THIS RUN — two runs staging under "run:pulled" never see each other — and such datasets are cleaned up after the run ends; every other name is durable and shared by the environment’s runs.
  • key (string) — The row’s key — the record’s own id, usually.
  • value (any) — Any JSON value up to 1 MiB serialized. A dataset holds at most 500000 rows and 512 MiB of values; a write that would cross either cap is refused as DATASET_FULL and nothing of it lands.

Answers: Nothing.

dsPut("run:pulled", "user-1", { name: "Ada", active: true });
log("staged");

dsPutMany(dataset, rows) — pauses

Stores a whole page of keyed rows in one call — the write to use when records arrive by the hundred. The batch lands entirely or not at all.

  • dataset (string) — Which dataset — run: names are this run’s own.
  • rows (array) — A list of { key, value } rows. A key that already exists replaces its row; a cap refusal (DATASET_FULL) rolls back the WHOLE batch, so a dataset never holds half a page.

Answers: Nothing.

dsPutMany("run:pulled", [
  { key: "user-1", value: { name: "Ada" } },
  { key: "user-2", value: { name: "Bob" } }
]);

dsGet(dataset, key) — pauses

Reads one row of a dataset back by its key.

  • dataset (string) — Which dataset — run: names are this run’s own.
  • key (string) — The row’s key.

Answers: The row’s value, or undefined for a key the dataset does not hold. A stored null comes back as null — distinguishable from a missing row.

const user = dsGet("run:pulled", "user-1");
log(user === undefined ? "not staged" : "staged");

dsGetMany(dataset, keys) — pauses

Reads many rows by key in one call, answering only the rows that exist.

  • dataset (string) — Which dataset — run: names are this run’s own.
  • keys (array) — The keys to read, as a list of text entries.

Answers: A list of { key, value } rows — the same shape dsPage answers — holding the rows that were FOUND, in the order their keys were asked for. A key with no row is simply absent, so compare lengths to find the misses.

const rows = dsGetMany("run:pulled", ["user-1", "user-2", "user-9"]);
log(rows.length + " of 3 are staged");

dsPage(dataset, options?) — pauses

Reads a slice of a dataset, ordered by key — the loop-friendly way to work through staged rows without holding them all at once.

  • dataset (string) — Which dataset — run: names are this run’s own.
  • options (object, optional) — { offset, limit, where }. where narrows to rows whose value at one dot-path equals one plain value — { path: "profile.active", equals: true } — with text, numbers, true/false and null each matching only their own kind, and equals: null also matching rows where the path is absent. Left out, the platform still answers a bounded page rather than everything.

Answers: A list of { key, value } rows, ordered by key.

const active = dsPage("run:pulled", { where: { path: "active", equals: true }, limit: 100 });
log(active.length + " active users in the first page");

dsCount(dataset) — pauses

Answers how many rows a dataset holds — a dataset that does not exist counts 0.

  • dataset (string) — Which dataset — run: names are this run’s own.

Answers: The row count as a number.

log("staged " + dsCount("run:pulled") + " rows so far");

dsClear(dataset) — pauses

Removes a dataset entirely — its rows and its name. Clearing what does not exist is a no-op.

  • dataset (string) — Which dataset — run: names are this run’s own.

Answers: Nothing.

dsClear("run:pulled");
log("staging ground empty");

dsRename(from, to) — pauses

Renames a dataset — and REPLACES whatever already held the new name, in one step. This is how a run promotes its staging set: rename run:current to a durable name and it outlives the run, standing where the previous baseline stood.

  • from (string) — The dataset to rename. Renaming one that does not exist is refused.
  • to (string) — Its new name. A dataset already under this name is replaced — gone with the rename, which is the point of a promotion. A run: name makes it this run’s own again.

Answers: Nothing.

dsPut("run:pulled", "user-1", { name: "Ada" });
dsRename("run:pulled", "baseline");
log("promoted — next run diffs against this");

dsDiff(base, current) — pauses

Counts what separates two datasets — how many rows were added, changed or removed in current relative to base. Computed on the platform’s side from fingerprints stored with every write, so it costs the same however wide the rows are.

  • base (string) — The dataset to compare against — last sync’s baseline, usually.
  • current (string) — The dataset holding the fresh rows — run: names are this run’s own.

Answers: { added, changed, removed }, each a count: keys only in current, keys in both whose values differ, keys only in base. A value that merely reordered its object keys has NOT changed. A dataset that does not exist compares as empty.

dsPut("run:pulled", "u1", { name: "Ada" });
const delta = dsDiff("baseline", "run:pulled");
log(delta.added + " new, " + delta.changed + " changed, " + delta.removed + " gone");

dsDiffKeys(base, current, kind, page?) — pauses

Lists the keys behind one of dsDiff’s counts — the rows to actually act on, one class at a time, page by page.

  • base (string) — The dataset to compare against.
  • current (string) — The dataset holding the fresh rows.
  • kind (string) — "added", "changed" or "removed" — which class of keys to list.
  • page (object, optional) — { offset, limit }. Left out, the platform still answers a bounded page rather than everything — walk with offsets when a delta may be large.

Answers: The keys of that class, ordered. Feed them to dsGetMany for the rows to upsert, or use the removed ones to retire what the source no longer has.

dsPut("run:pulled", "u1", { name: "Ada" });
const fresh = dsDiffKeys("baseline", "run:pulled", "added", { limit: 100 });
log(fresh.length + " new records to create");

dsStats(dataset, path) — pauses

Groups a dataset’s rows by the value at one dot-path and counts each group — a one-call report over staged data, computed on the platform’s side.

  • dataset (string) — Which dataset — run: names are this run’s own.
  • path (string) — The dot-path to group by, such as "dept" or "profile.role".

Answers: A list of { group, count } entries, most frequent first. Each group is the value AS JSON TEXT — "active" with its quotes, 42, true — and rows where the path is absent group under "null" together with stored nulls.

const groups = dsStats("run:pulled", "dept");
for (const g of groups) { log(g.group + ": " + g.count); }

dsMapInto(source, mappingName, target, options?) — pauses

Maps every row of a staged dataset into another one, using a JSON mapping somebody configured — the whole set in one call, without the records ever passing through the script. The platform works through it a batch at a time, pausing the run in between, so a set of any size costs short segments rather than one long one.

  • source (string) — The dataset to read — run: names are this run’s own.
  • mappingName (string) — The configured JSON mapping to apply to each record. It is the same mapping a script could call by name, so what its preview showed is what these rows become; a function that is not a JSON mapping is refused.
  • target (string) — The dataset the mapped records are written to.
  • options (object, optional) — { keyPath, onError }. keyPath takes the new row’s key from the MAPPED record — the target system’s own identity — instead of keeping the source key. onError is "reject" (the default: the row goes to <target>.rejected with a reason, and the rest of the set still runs) or "fail" (the first bad row ends the call).

Answers: { rows, rejected } — how many records were written and how many could not be mapped. The rejected ones are in a dataset called <target>.rejected, keyed by the row that failed and carrying the reason, so dsPage shows exactly what to fix. That dataset is emptied each time the call starts.

dsPut("run:staged", "u1", { id: "u1", name: "Ada" });
const done = dsMapInto("run:staged", "MyMapping", "run:ready");
log(done.rows + " ready, " + done.rejected + " rejected");

dsEnrich(dataset, functionName, options) — pauses

Fills staged rows in from a second system: one configured call per row, its answer merged into the row where you say. The calls happen one after another — never at the same time — so an integration is not throttled or blocked for asking too fast, and if the far end says it is being called too often the run waits and continues with the row it was on.

  • dataset (string) — The dataset to enrich; its rows are updated in place.
  • functionName (string) — The configured function to call for each row — usually the call that reads one record of the other system.
  • options (object) — { argPath, param, into, onError }. argPath says where in the record the value to look up with is, and param which parameter it binds to (left out: the function’s first). into names the field the answer is written to. onError is "reject" (the default: the row moves to <dataset>.rejected), "null" (the row stays, with nothing at into) or "fail" (the first failed lookup ends the call).

Answers: { rows, rejected } — how many records were enriched and how many were not. For a call, the ANSWER is what the far end sent back; a status outside 200–299 counts as no answer, because a row has nothing to branch on.

dsPut("run:staged", "u1", { key: "ABC-1" });
const done = dsEnrich("run:staged", "MyFunction", { argPath: "key", into: "issue" });
log(done.rows + " records enriched");

dsDuplicates(dataset, path, options?) — pauses

Finds the values that more than one record shares — the check to run BEFORE importing into a field the target system keeps unique. Ask it with the target field’s width and it compares what that field would actually store, so two names that differ today but collide at 40 characters are found here instead of halfway through an import.

  • dataset (string) — The dataset to check — run: names are this run’s own.
  • path (string) — Which field to check, as a dot-path — "email", "profile.login". Records with nothing there are not a collision.
  • options (object, optional) — { length } — compare only the first N characters, the width of the field you are importing into. Left out, whole values are compared.

Answers: How many colliding groups there are, as a number. The groups themselves land in a dataset called <dataset>.duplicates — the shared value, how many records carry it, and their keys — so dsPage shows what to correct. Nothing is changed in the data: which of two colliding records is the right one is a business decision, not the platform’s.

dsPut("run:staged", "u1", { email: "ada@example.com" });
const clashes = dsDuplicates("run:staged", "email", { length: 40 });
log(clashes === 0 ? "safe to import" : clashes + " collisions to fix first");

dsToCsv(dataset, mappingName, options?) — pauses

Writes every row of a staged dataset into CSV files, using a columns table somebody configured — capped in size the way a bulk import wants them, each file complete with its own header line. The platform works through the set a file at a time, pausing the run in between, so a dataset of any size costs short pieces of work rather than one long one.

  • dataset (string) — The dataset to write — run: names are this run’s own.
  • mappingName (string) — The configured columns table to render each record with. It is the same one a script could call by name, so what its preview showed is what these files contain; a function that is not a columns table is refused.
  • options (object, optional) — { maxBytes, maxLines, maxLineBytes, where }. The first three cap one file — 10 MB, 10 000 records and 10 KiB per line by default; set them to what the system you are uploading to accepts. where narrows to rows whose value at one dot-path equals one plain value — { path: "active", equals: true }. A record whose line is longer than the cap is not split: it goes to <dataset>.rejected with a reason, and the rest still go out.

Answers: A list of files, in order: { blobId, name, lines, bytes, firstKey, lastKey } each — the handle to upload, a suggested file name, how many records it carries, how big it is, and the first and last dataset key in it. The same list is left in a dataset called <dataset>.chunks, and anything that could not be written is in <dataset>.rejected, keyed by the row that failed; both are emptied each time the call starts.

dsPut("run:ready", "u1", { id: "u1", name: "Ada" });
const files = dsToCsv("run:ready", "MyCsv", { maxLines: 5000 });
log(files.length + " files to upload, first is " + files[0].name);

dsFromCsv(file, into, mappingName?, options?) — pauses

Reads a stored CSV file into a dataset, one batch of records at a time, without the file ever passing through the script. This is how a file that arrived as a mail attachment or came back from a download becomes rows you can work with — a real export weighs far more than a script may hold, so reading its contents and parsing them yourself works on a sample and refuses on the file that matters.

  • file (string) — The handle of the stored file to read.
  • into (string) — The dataset the records are written to.
  • mappingName (string, optional) — The configured columns table to shape each record with — it decides which column becomes which field and what type it is. Left out, the file’s first line names the fields and every value stays text. You can pass the options in its place when there is no table.
  • options (object, optional) — { keyPath, onError, maxLineBytes }. keyPath says which field keys each row (left out, the record’s own id is used). onError is "reject" (the default: the record goes to <into>.rejected with the line it was on and a reason) or "fail" (the first bad record ends the call).

Answers: { rows, rejected } — how many records were staged and how many could not be. The rejected ones are in a dataset called <into>.rejected, keyed by the line they were on, so dsPage shows exactly what to fix; that dataset is emptied each time the call starts. A file whose row does not fit its own header ENDS the call naming the line, because everything after such a row would land in the wrong columns.

const file = writeBlob(btoa("email,name\nada@example.com,Ada\n"), "text/csv");
const done = dsFromCsv(file, "run:incoming", { keyPath: "email" });
log(done.rows + " contacts staged, " + done.rejected + " rejected");

csvLineRecord(file, line) — pauses

Turns a line number in a file this run wrote back into the record it came from. Bulk imports answer with line numbers — “line 3 was rejected” — and this is what lets a report name the customer or the order instead of a number nobody can act on. It counts lines the way the file really has them, so a value carrying a line break is accounted for.

  • file (string) — The handle of a file dsToCsv wrote in this run.
  • line (number) — The line number the other system reported, counting from 1.

Answers: { key, dataset, record } — the dataset key of the record that wrote that line, the dataset it is staged in, and the record itself. Answers null for the header line, for a line past the end of the file, and for a file this run did not write, so a report can walk whatever line numbers it was given.

dsPut("run:ready", "u1", { id: "u1", name: "Ada" });
const files = dsToCsv("run:ready", "MyCsv");
const failed = csvLineRecord(files[0].blobId, 2);
log(failed === null ? "no record wrote that line" : "line 2 is record " + failed.key);

upsertBySourceId(namespace, record, options) — pauses

Creates a record in the other system the first time it is seen and updates it every time after — the platform remembers which is which. It looks up the id mapping, calls the one of your two functions that fits, and writes the mapping back when a create succeeds, so the next run knows.

  • namespace (string) — Which kind of thing this is — one namespace per remote entity type, such as "people". The same namespace the mapping calls use.
  • record (object) — The source record itself. Its own id identifies it (id, Id or ID, or the field sourceIdPath names), and that id is what the mapping is kept under.
  • options (object) — { createFn, updateFn, idPath, sourceIdPath, namePath, recordParam, targetParam }. createFn and updateFn name the configured functions that do the work — the create gets the record, the update gets the record and the target id the platform remembered. idPath says where the create’s ANSWER carries the new id (left out: its own id). namePath says which field names the record in the report.

Answers: { action, outcome, target, reason } — whether it was a create or an update, whether it worked, and the id in the other system. A record that could not be synced answers outcome: "failed" with the reason rather than ending the run, and a row naming it lands in the report dataset either way.

const person = { id: "src-1", name: "Ada Lovelace" };
const done = upsertBySourceId("people", person, { createFn: "MyFunction", updateFn: "MyFunction" });
log(done.action + " " + person.name + ": " + done.outcome);

applyLinks(dataset, spec) — pauses

Writes the relationships between records that are already in the other system — who reports to whom, which department something belongs to. It resolves both ends of every link from the id mappings, a whole page at a time, so the only calls made are the link writes themselves.

  • dataset (string) — The staged rows to read the links from — run: names are this run’s own.
  • spec (object) — { linkFn, from, to, namePath }. linkFn names the configured function that writes ONE link. from and to each say { namespace, path, param } — which mappings that side lives in, where the row carries its id, and which parameter of your function takes the resolved id (left out: from and to).

Answers: { linked, failed }. A row whose other end is not in the target yet is not an error — it becomes a report row naming the record, and the link is written on the run that creates the missing side. A row with no link at all is skipped silently, because most records have none.

dsPut("run:people", "p1", { id: "p1", name: "Ada", managerId: "p2" });
const spec = { linkFn: "MyFunction", from: { namespace: "people", path: "id" }, to: { namespace: "people", path: "managerId" } };
const done = applyLinks("run:people", spec);
log(done.linked + " links written, " + done.failed + " waiting for the other side");

stageDiff(baseline, current, kinds, into) — pauses

Stages the RECORDS behind a comparison, ready for the phase that acts on them. dsDiff says how many rows differ and dsDiffKeys says which keys; this puts the rows themselves where you can walk them, taking a removed record from the baseline, which is the only side that still has it.

  • baseline (string) — The dataset to compare against — last sync’s copy.
  • current (string) — The dataset holding the fresh rows.
  • kinds (array) — Which differences to stage: any of "added", "changed" and "removed".
  • into (string) — The family of datasets to stage into: each class lands in its own — "run:delta" writes run:delta.added, run:delta.changed and run:delta.removed. They are emptied each time the call starts.

Answers: How many rows each class staged, as { added, changed, removed } — only the classes that were asked for. A class that matched nothing answers zero, which is an answer to branch on rather than a missing key.

const delta = stageDiff("baseline", "run:pulled", ["added", "changed"], "run:delta");
log(delta.added + " to create, " + delta.changed + " to update");

sendReport(channel, options) — pauses

Turns the report dataset into something a person reads — how many records went through, broken down by what was done and how it ended, then the ones that need a human, by name and with the reason. The counting happens where the rows are, so reporting on ten thousand records costs what ten costs.

  • channel (string) — "email" sends it as mail, through one of your email functions — the mailbox and its password are that function’s business. "request" sends it through an http function instead, which is how it reaches a ticketing system or a chat webhook.
  • options (object) — { fn, dataset, title, limit, bodyParam }. fn names the configured function that sends it, and must match the channel. dataset is the report to render (left out: report). title is the heading. limit caps how many failures are named before the body starts counting instead (left out: 20). bodyParam is the parameter your function takes the markdown body as (left out: body).

Answers: { total, failed, listed } — how many records the report covered, how many failed, and how many of those the body named. A send that does not go through fails the run rather than being reported: the thing that would carry the bad news is the thing that broke.

dsPut("report", "created:u1", { name: "Ada Lovelace", action: "created", outcome: "ok" });
const done = sendReport("request", { fn: "MyReport", title: "Nightly people sync" });
log(done.total + " records reported, " + done.failed + " need attention");

importAll(files, importFn, options?) — pauses

Uploads the files dsToCsv produced to a bulk import API and waits for each one to finish — the waiting costs nothing, because the run is suspended rather than held. When the far end answers with line numbers, they are turned back into the records that wrote them, so the report names people instead of positions in a file.

  • files (array) — The list dsToCsv answered — its handles and names, passed straight through.
  • importFn (string) — The configured function that uploads ONE file. It takes the file’s handle as its file parameter, and it is where the multipart form is authored.
  • options (object, optional) — { statusFn, pollMs, maxPolls, fileParam, jobParam, jobIdPath, donePath, doneValues, errorsPath, linePath, reasonPath }. statusFn names the function that asks how an import is going — left out, an upload that answers is treated as finished. The paths say where that system’s answer carries the job id, its state and the lines it rejected; the defaults are the common spellings (id, status, errors, line, reason).

Answers: { chunks, failed, rejected } — files sent, files the far end would not take, and records it rejected by line. Each file gets a report row, and each rejected line becomes a row naming the record it came from. An import that never finishes is given up on after maxPolls asks and reported as such, because a run must not wait for ever.

dsPut("run:ready", "u1", { id: "u1", name: "Ada" });
const files = dsToCsv("run:ready", "MyCsv");
const done = importAll(files, "MyFunction");
log(done.chunks + " files imported, " + done.rejected + " records rejected");

The rest of this pack

The page above is everything that is true only HERE. Ask for these parts by name for the rest:

  • model — how a script is executed, the shape of its input, the rules enforced while it runs, and the mistakes that do not work here.
  • functions-core — the core functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-encoding — the encoding functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-crypto — the crypto functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-text — the text functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-number — the numbers functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-structured — the structured data functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-document — the documents functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-markdown — the markdown functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-datetime — the date & time functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-email — the email functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-blob — the files functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-mapping — the mapping functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-state — the state functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-pagination — the pagination functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-control — the run control functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • functions-functions — the config functions — what each one does, its arguments, whether it pauses the run, and a line of real script.
  • mistakes — the habits from general JS that do not work here — each with the exact error it produces and the shape to write instead.
  • standard-library — every built-in the language itself provides — JSON, Math, Object, String, Array, Number, Map, Set, Date, RegExp and the bare globals — one line each, with an example and its answer.
Rendered from docs/guide/authoring-pack-functions-dataset.md in the product's own repository, at build time. Found a problem on this page? Write to the address in the footer.