Flows¶

A flow is a data pipeline that lives inside TriplyDB. It is drawn rather than programmed: tasks from a catalogue — import from a URL, run a CONSTRUCT query, validate with SHACL, recreate a service — are placed on a canvas, connected in the order they should run, and each configured through a form. A flow can then be run by hand, on a schedule, by another system over HTTP, or by a hook when data is uploaded to a dataset.

Flows are built in the console, and only there. The HTTP API can start, follow and stop runs, but creating or changing a flow through it is not supported.

This page explains the model behind that. The task types are listed in Flow tasks, everything else in the Flows reference.

Tasks work on TriplyDB, not on a stream¶

The most important thing to know about a flow is what does not travel along its edges: data. Each task names the TriplyDB items it reads and writes — a dataset, a graph, a saved query, a service — and acts on them directly. A Construct query task reads dataset raw and writes graph clean in dataset products; the task after it does not receive those statements, it reads products itself.

Three things follow from that.

• Every intermediate result is inspectable. Whatever a task wrote is an ordinary graph that can be browsed and queried while the flow is being built, and after a run has failed.
• Writing adds. A task that writes to a graph that already exists adds its statements to what is there; none replaces a graph's contents. A flow that should start from empty on every run deletes its output first.
• A task can pick out several items at once. Besides naming one dataset, a task can take every dataset of an account that carries a set of tags, so one Set access level task can publish a whole family of datasets, including ones created after the flow was built.

Edges set the order¶

The edges drawn between tasks say which must finish before which may start, and nothing else. Where the cards sit on the canvas does not matter. Tasks that no edge orders may run at the same time, which is how a flow that imports from five sources imports them in parallel.

That freedom has a catch, and TriplyDB checks for it. If two unordered tasks touch the same dataset, and at least one of them changes it in a way the other depends on — one deletes a graph the other reads, say — the result would depend on which happened to go first. A flow in that state can be saved but not run, and the builder names the two tasks and asks for an edge between them. Two tasks that only add statements to the same dataset need no edge, because the outcome is the same in either order.

Branches¶

A flow does not have to take the same path every time. A Decision task compares values — typically a variable against a fixed text — and continues along the first branch whose condition holds, or along its else branch. Tasks on the branches not taken are marked skipped.

Branches that should come together again meet in an Or task, which continues as soon as any one of its incoming branches is complete. A Stop run task ends a run early, as aborted or failed, with a message of its choosing.

Decisions compare variable values, not data. To branch on what TriplyDB holds, a Set variables task first turns the answer to a SELECT query — a count, whether a SHACL report lists violations — into variables, and the Decision compares those.

Variables¶

Variables separate what a flow does from what it does it to. A flow that imports a file and publishes it can take the file's URL and the dataset's name as variables, and serve every source of the same shape instead of being copied once per source.

A variable is declared on the flow and referred to in a task field as ?name. Where a field holds free text — an email body, a request header — the reference can sit anywhere inside it. In an inline SPARQL query, any query variable with the name of a flow variable is replaced by that variable's value. Every value is text, and a run checks that each one fits the field it lands in.

Three further kinds work the same way in tasks but come from elsewhere.

• Masked values hold secrets: API keys, passwords, tokens. Only people allowed to manage them see their values; everyone else with the right to run the flow can use them without seeing them. That separates the right to run a pipeline from the right to know its credentials.
• System variables are set by TriplyDB on every run: who started it, and a TriplyDB API token for that person that expires with the run. A run that a hook started also learns what was uploaded, and where. The token lets a task — a Send request to the TriplyDB API, or a connector script — act on TriplyDB as that person without anyone storing a token in the flow.
• Task output variables are set during a run, by a Set variables task, from the first row of a SELECT query. Only tasks that an edge orders after that task can use them.

Presets and schedules¶

A preset is a named set of variable values: the values for one source, one environment, one customer of the pipeline. Starting a run from a preset saves typing them, and makes it easy to see which configurations a flow is used with.

A schedule runs the flow at set times — a cron expression in a time zone — with values of its own. One flow can carry several schedules, nightly for one source and weekly for another.

Runs¶

A run is one execution of a flow. It records how it was started, by whom, with which values, and what each task did: its status, its log, the statements it wrote, the files it produced. The run keeps a copy of the flow as it was when the run started, so editing a flow never changes a run that is under way, and an old run still shows the tasks it actually ran.

A run acts as the person who started it; a scheduled run acts as the schedule's owner. It can read and change only what that person can, whoever built the flow. When a run fails, that person gets an email.

Every time a flow is saved, its previous state is kept. Version history in the builder loads an earlier version back into the canvas, where it becomes the current flow once saved.

Connectors¶

The built-in tasks reach what TriplyDB can reach: the public web, and TriplyDB itself. Much of an organisation's data lives elsewhere — in a relational database, on a file share, behind a firewall.

A connector agent bridges that gap. It is a small program run outside TriplyDB — on a server in the organisation's network, or on a developer's laptop — which connects out to TriplyDB and waits for work. When a flow reaches an Import via connector script or Execute via connector script task, the agent runs the named script, locally, with the flow's variables in its environment. Whatever RDF the script writes is imported into TriplyDB; its logs and other files are attached to the run.

So the code that knows how to talk to a private system stays on the network where that system is, is written in whatever language suits it, and is still started, logged and governed by the flow. How to write one is in Run custom code in a flow.