---
title: Flows
url: https://docs.triply.cc/concepts/flows/
description: 'How Flows works: tasks that act on TriplyDB itself, edges that set their order, variables
  that make one flow serve many runs, and the runs, schedules and connectors around them.'
availability:
- 'Private preview: This feature is switched on per instance, by arrangement. Get in touch if you would
  like access.'
---

# Flows

A flow is a data pipeline that lives inside TriplyDB. It is drawn rather than programmed: tasks
from a catalogue — import from a URL, run a CONSTRUCT query, validate with SHACL, recreate a
service — are placed on a canvas, connected in the order they should run, and each configured
through a form. A flow can then be run by hand, on a schedule, by another system over HTTP, or by a hook when
data is uploaded to a dataset.

Flows are built in the console, and only there. The HTTP API can start, follow and stop runs, but
creating or changing a flow through it is not supported.

This page explains the model behind that. The task types are listed in
[Flow tasks](https://docs.triply.cc/reference/flow-tasks/), everything else in the
[Flows reference](https://docs.triply.cc/reference/flows/).

## Tasks work on TriplyDB, not on a stream

The most important thing to know about a flow is what does *not* travel along its edges: data.
Each task names the TriplyDB items it reads and writes — a dataset, a graph, a saved query, a
service — and acts on them directly. A **Construct query** task reads dataset `raw` and writes
graph `clean` in dataset `products`; the task after it does not receive those statements, it
reads `products` itself.

Three things follow from that.

- **Every intermediate result is inspectable.** Whatever a task wrote is an ordinary graph that
  can be browsed and queried while the flow is being built, and after a run has failed.
- **Writing adds.** A task that writes to a graph that already exists adds its statements to
  what is there; none replaces a graph's contents. A flow that should start from empty on every
  run deletes its output first.
- **A task can pick out several items at once.** Besides naming one dataset, a task can take
  every dataset of an account that carries a set of [tags](https://docs.triply.cc/how-to/share-and-publish/), so one
  **Set access level** task can publish a whole family of datasets, including ones created after
  the flow was built.

## Edges set the order

The edges drawn between tasks say which must finish before which may start, and nothing else.
Where the cards sit on the canvas does not matter. Tasks that no edge orders may run at the same
time, which is how a flow that imports from five sources imports them in parallel.

That freedom has a catch, and TriplyDB checks for it. If two unordered tasks touch the same
dataset, and at least one of them changes it in a way the other depends on — one deletes a graph
the other reads, say — the result would depend on which happened to go first. A flow in that
state can be saved but not run, and the builder names the two tasks and asks for an edge between
them. Two tasks that only add statements to the same dataset need no edge, because the outcome is
the same in either order.

## Branches

A flow does not have to take the same path every time. A **Decision** task compares values —
typically a [variable](#variables) against a fixed text — and continues along the first branch
whose condition holds, or along its **else** branch. Tasks on the branches not taken are marked
skipped.

Branches that should come together again meet in an **Or** task, which continues as soon as any
one of its incoming branches is complete. A **Stop run** task ends a run early, as aborted or
failed, with a message of its choosing.

Decisions compare variable values, not data. To branch on what TriplyDB holds, a
[**Set variables**](https://docs.triply.cc/reference/flow-tasks/#set-variables) task first turns the answer to a
SELECT query — a count, whether a SHACL report lists violations — into variables, and the
Decision compares those.

## Variables

Variables separate what a flow does from what it does it to. A flow that imports a file and
publishes it can take the file's URL and the dataset's name as variables, and serve every source
of the same shape instead of being copied once per source.

A variable is declared on the flow and referred to in a task field as `?name`. Where a field
holds free text — an email body, a request header — the reference can sit anywhere inside it. In
an inline SPARQL query, any query variable with the name of a flow variable is replaced by that
variable's value. Every value is text, and a run checks that each one fits the field it lands in.

Three further kinds work the same way in tasks but come from elsewhere.

- **Masked values** hold secrets: API keys, passwords, tokens. Only people allowed to manage them
  see their values; everyone else with the right to run the flow can use them without seeing
  them. That separates the right to run a pipeline from the right to know its credentials.
- **System variables** are set by TriplyDB on every run: who started it, and a TriplyDB API token
  for that person that expires with the run. A run that a hook started also learns what was
  uploaded, and where. The token lets a task — a **Send request** to the
  TriplyDB API, or a connector script — act on TriplyDB as that person without anyone storing a
  token in the flow.
- **Task output variables** are set during a run, by a
  [**Set variables**](https://docs.triply.cc/reference/flow-tasks/#set-variables) task, from the first row of a
  SELECT query. Only tasks that an edge orders after that task can use them.

## Presets and schedules

A **preset** is a named set of variable values: the values for one source, one environment, one
customer of the pipeline. Starting a run from a preset saves typing them, and makes it easy to
see which configurations a flow is used with.

A **schedule** runs the flow at set times — a cron expression in a time zone — with values of its
own. One flow can carry several schedules, nightly for one source and weekly for another.

## Runs

A run is one execution of a flow. It records how it was started, by whom, with which values,
and what each task did: its status, its log, the statements it wrote, the files it produced. The
run keeps a copy of the flow as it was when the run started, so editing a flow never changes a
run that is under way, and an old run still shows the tasks it actually ran.

A run acts as the person who started it; a scheduled run acts as the schedule's owner. It can
read and change only what that person can, whoever built the flow. When a run fails, that person
gets an email.

Every time a flow is saved, its previous state is kept. **Version history** in the builder loads
an earlier version back into the canvas, where it becomes the current flow once saved.

## Connectors

The built-in tasks reach what TriplyDB can reach: the public web, and TriplyDB itself. Much of an
organisation's data lives elsewhere — in a relational database, on a file share, behind a
firewall.

A **connector agent** bridges that gap. It is a small program run outside TriplyDB — on a server
in the organisation's network, or on a developer's laptop — which connects out to TriplyDB and
waits for work. When a flow reaches an
**Import via connector script** or **Execute via connector script** task, the agent runs the
named script, locally, with the flow's variables in its environment. Whatever RDF the script
writes is imported into TriplyDB; its logs and other files are attached to the run.

So the code that knows how to talk to a private system stays on the network where that system
is, is written in whatever language suits it, and is still started, logged and governed by the
flow. How to write one is in
[Run custom code in a flow](https://docs.triply.cc/how-to/run-custom-code-in-a-flow/).