On this page:

Run custom code in a flow

This page covers running a script of your own as one task of a flow — to read a relational database, a file share or an API the built-in tasks cannot reach. The script runs on a connector agent that you run yourself — on a server in your own network, or on your laptop while you develop the script. For the task fields, see Import via connector script.

Set up a connector agent

Registering an agent takes an instance administrator. On an instance Triply hosts for you, ask Triply support.

  1. In Admin settings, open Connector agents and click Add connector.
  2. Give it a Name. Flows refer to the agent by this name.
  3. Under Accessible by, choose the user or group whose flows may use it. Left empty, it is yours.
  4. Choose its Access Level, then create it, and copy the Connector handshake token the dialog shows.

  5. Run the TriplyDB connector image anywhere that can reach both the instance's API and the systems your script needs — a server, a Kubernetes cluster, or your own laptop — passing it the API address and the handshake token. The agent connects out to TriplyDB, so nothing needs to reach it. The dialog's run instructions give the exact command for your instance; the image is in Triply's container registry, and Triply support can provide credentials. With Docker:

    docker run --rm -it \
  --volume ~/connector:/connector \
  --volume ~/connector-data:/connector-data \
  --volume ~/connector-scripts:/connector-scripts \
  IMAGE:VERSION \
  app:start \
  --api-url API_URL \
  --handshake-token HANDSHAKE_TOKEN

    Keep /connector on persistent storage: the agent stores its credentials there, and loses its registration without them.

The agent's Last Contact in the Connector agents table shows that it has connected.

Write the script

A script is an executable file in the agent's scripts directory, /connector-scripts. Mount it there, as above, or build an image of your own on top of the connector image. Build your own image too when the script needs tools the base image lacks, such as python3 — and keep the base image's entrypoint.

When a flow calls the script, it gets:

Environment variable Holds
CONNECTOR_OUTPUT_DIR The directory to write the RDF to import. Imported in full if the script succeeds; ignored if it fails.
CONNECTOR_ARTIFACTS_DIR The directory for everything else — logs, reports, samples. Each file here is attached to the run, whether the script succeeds or fails.
TRIPLYDB_FLOW__system_triplyToken A TriplyDB API token for whoever started the run, valid while it lasts. Use it for calls back to TriplyDB instead of a token of your own.
TRIPLYDB_FLOW__name The value of each flow variable and masked value, under its own name. Dashes become underscores: flow variable table-name arrives as TRIPLYDB_FLOW__table_name.

Anything the script prints on standard output or standard error is saved to script.log among the run's artifacts. A script that exits with a status other than 0 fails the task.

A script can also print control messages, one line of JSON each:

{"type": "progress", "processed": 0.25}
{"type": "stop-with-error-message", "message": "Source database refused the connection"}
  • progress reports how far the script has got, as a fraction between 0 and 1, shown on the run while it lasts.
  • stop-with-error-message stops the script and fails the task with that message. Without it, the run shows only the exit status and the last lines of standard error.

A minimal script that exports a table as N-Triples:

#!/bin/bash
set -euo pipefail
echo '{"type":"progress","processed":0}'
export-table "$TRIPLYDB_FLOW__table_name" > "$CONNECTOR_OUTPUT_DIR/data.nt"
echo '{"type":"progress","processed":1}'

export-table stands for whatever command reads your source. Make the file executable (chmod +x), or the agent cannot run it.

To write a large artifact that should be attached as soon as it is complete, rather than when the script ends, list its file name under the task's Live artifacts. Write it under a temporary name and move it into place, so it is never attached half-written.

Call it from a flow

  1. On the flow's Flow builder tab, add an Import via connector script task — or, if the script imports nothing, Execute via connector script.
  2. Choose the agent under Connector agent name, and the script under Connector script name. The list shows the scripts the agent reports each time it checks in, so a script you have just added appears within moments.
  3. For an import, choose the dataset or graph to import into.
  4. Save the flow and run it.

The run's page shows the task's progress while the script runs. Its files, script.log among them, are in the task's details, in the console and over the API — see Follow a run.