Getting Started with Testing Flows
This page walks through testing flows in the order you will need it: from a new repository, through the tests you write every day, to a flow tested on a live instance. Each step gives the commands and the decisions, and links to Testing Flows for the full explanation of each tier.
1. Set up a repository
Create the virtual environment with Python 3.14, the latest version a PythonBlock supports. Each
custom code block is built for the Python version of the environment you build the flow in, and
the platform flags a version it is about to stop supporting when you import the flow. Python 3.14
blocks need platform version 44 or later; for an older instance, use the newest version it
supports, or set HS_PYTHON_CODE_BLOCK_PYTHON_VERSION — see
Managing Third-Party Python Packages.
pip install "flows_sdk[testing]"
hs-flows init invoice_processing --flow-name invoice
cd invoice_processing
pip install -e ".[dev]" # the project, plus ruff and mypy
pytest -m "not e2e and not in_flow" # every offline tier
hs-flows init writes a project with one flow and a test for every tier, along with the
ruff, mypy and pytest configuration, including -p flows_sdk.testing.plugin, which enables
the framework’s pytest plugin. It works offline. ruff check ., ruff format --check ., mypy .
and pytest all pass in the directory it creates.
The flow and its tests are placeholders: the flow returns a fixed value, and each tier checks it. Replace them with your own. The comments in each file say what belongs there, and link to the part of Testing Flows that shows it for a real document-processing flow.
invoice_processing/
├── pyproject.toml # tool configuration, the tier markers and the pytest plugin
├── conftest.py # puts the project on sys.path for pytest
├── README.md
├── .flows-sdk/ # SDK-owned config, refreshed by `hs-flows update`
│ ├── ruff.toml
│ └── mypy.ini
└── invoice_processing/
└── flows/invoice/
├── flow.py # entry_point_workflow() -> Flow
├── functions.py # the custom code — plain, importable functions
└── tests/
├── unit/ # each function, called directly
├── flow/ # the whole graph offline, your code running for real
├── contract/ # wiring, interface and the contract snapshot (golden in snapshots/)
├── e2e/ # a live instance, with the synthetic data in data/
├── in_flow/ # customer-data evaluation, run as flows on the instance
└── data/ # synthetic data: the flow's expected output
Tests live inside the flow’s own package, so one directory holds a flow and everything that proves
it; a second flow gets its own tests/. Keep custom code in functions.py as named functions:
everything below is easier when the code under test can be imported.
2. Mark each test with its tier
Each test module says which tier it belongs to with an ordinary pytest marker:
import pytest
pytestmark = pytest.mark.unit
hs-flows init declares the markers — unit, flow, contract, e2e, in_flow —
in pyproject.toml and marks every test it generates. They are yours to extend or drop; nothing
applies or requires them. Two habits keep them reliable:
Select the offline run by exclusion,
pytest -m "not e2e and not in_flow", so a test someone forgot to mark still runs.Pair a live marker with its skip,
pytestmark = [pytest.mark.e2e, requires_instance()], so the live tiers skip cleanly where no instance is configured.
The folder a test sits in means nothing to pytest. See Marking tests with their tier.
3. Add a flow and its tests
Write flows/<name>/flow.py and functions.py, then add its tests in order of value — cheapest
first:
Contract. One class beside the flow gets wiring, interface, Python version and snapshot checks, with no stubs and no instance. If the flow calls subflows, declare them too, so each call site is checked:
from pathlib import Path import pytest from flows_sdk.testing.suites import FlowStaticChecksSuite from invoice_processing.flows.invoice.flow import entry_point_workflow pytestmark = pytest.mark.contract class TestInvoiceContract(FlowStaticChecksSuite): flow = entry_point_workflow snapshot_path = Path(__file__).parent / 'snapshots' / 'invoice.json'
Create the golden once, then commit it:
UPDATE_SNAPSHOTS=1 pytest -m contract
See Wiring tests: validate references without stubs and Contract tests: flow snapshots.
Unit. A test for each function in
functions.py, calling it directly — see Unit tests: functions.Flow graph. One test per path that matters, typically one per routing branch. Stub the platform blocks from a recorded run of a synthetic document rather than by hand — see Stubs come from a real run.
4. Run the right tests after a change
Start with the cheapest tier that can see the change:
You changed |
Run first |
|---|---|
a reference, a block, the order of blocks |
|
the code of a custom code block |
|
how blocks hand data to one another |
|
a declared input or output, a role |
|
Then, before you push, everything offline:
pytest -m "not e2e and not in_flow"
A moved golden means a caller of the flow is affected, so read its diff before committing it.
5. See what your tests cover
Use ordinary line coverage. A custom code block’s code is a plain importable function, so
pytest-cov over the flows package measures your flow logic directly. The tests live inside that
package, so the generated pyproject.toml leaves */tests/* out of the measurement:
pytest -m "unit or flow" --cov=invoice_processing.flows.invoice --cov-report=term-missing
Run it over the unit and flow tiers: the flow tier drives your code through the graph and reaches paths per-function tests miss. Read the two kinds of file differently:
functions.pyis the real measurement — an uncovered line is a branch nothing exercises.flow.pynearly all executes the moment the flow is built, so a high number says little; an uncovered line usually means a block or branch no test builds.
Line coverage cannot tell you whether a routing branch was taken. Assert it in a flow test, one
per branch, and assert the untaken side too — runner.assert_blocks_did_not_run(...) — or the
test also passes when both sides run. See Scenarios edit the recording.
6. Test on a live instance with synthetic data
Most of what a flow does on the platform — that it deploys and completes, which way it routes,
what happens after supervision, how it uses the knowledge store — is behaviour, and a synthetic
document exercises it as well as a real one. That is the e2e tier: ordinary pytest that deploys
a copy of the flow from your checkout, sends it data from tests/data/, and asserts on exact
values. The scaffold’s flow declares its identifier with flow_identifier so it can be copied —
see Flows that call each other.
Create a machine-to-machine credential on a development instance, in the System Admin group:
the tests import and deploy flows, which needs the import_flows permission, and on a fresh
instance no other group has it. The UI shows the credential’s Client ID and Client Secret
(JWK); paste the JWK into a file, and point the tests at it:
export HS_BASE_URL=https://dev-instance
export HS_M2M_CLIENT_ID=<the client id shown next to the key>
export HS_M2M_CREDS=~/.hyperscience/m2m_creds.json # the Client Secret (JWK)
pytest -m e2e
Run it in CI against the development instance. Without those variables the tier skips, so the rest of the suite stays runnable anywhere. See End-to-end tests: the repository owns the data.
7. Evaluate on customer data, where only that will do
Keep customer data for what a synthetic document cannot answer — typically evaluation, such as whether extraction stays accurate on what a customer actually sends. Customer documents must not leave the customer’s environment, so these tests are flows, deployed to the customer’s instance, that assert there and report back only pass or fail.
The scaffold’s tests/in_flow/ holds the pieces: evaluation.py (submit and wait, then compare,
as plain code), the suite flow that runs it, the wrapper flow that holds the test document, and
test_in_flow.py. They are placeholders too: the suite submits the document to the generated flow
and checks that the run returns its fixed output. Then:
Deploy the flow under test. The suite submits the document to the flow the wrapper names, so that flow must be deployed on the instance. On a customer’s instance it usually is already: it is the flow you are evaluating. The e2e tier’s copy does not count, since it is removed after each run. For the generated flow,
python -m invoice_processing.flows.invoice.flowwritesbuild/invoice.json; import it in Flow Studio and deploy it.Run it.
pytest -m in_flow, with the credentials above pointed at the customer’s instance. The first run uploads both test flows and fails, because nothing in your repository can supply a customer document. The error links to the wrapper’s settings.Configure the wrapper in Flow Studio: attach the document. A real suite also sets the result expected from it, approved from a run you inspected — see Where the expected result comes from.
Run it again. pytest reports the suite’s flow run as one test. When a case fails, the test fails with a link to the run, where you read which case failed and why.
The configured wrapper is the only record of what the tests run against, so a run that would
overwrite it fails instead. After changing the wrapper’s code, re-run with --merge-test-flows
to keep its data, or --force-test-flows to discard it — see The wrapper is protected.
8. Upgrade the SDK
pip install -U "flows_sdk[testing]"
hs-flows update # refresh the SDK-owned ruff and mypy config
pytest -m contract
pytest -m "not e2e and not in_flow"
hs-flows update overwrites the configuration the SDK owns with the copy in the new wheel, so
lint and type rules stay in step across every flows repository. A contract golden records only
what a caller of the flow can observe, so if one moves after an upgrade, review it: the new SDK
changed your flow’s interface.
See also
Testing Flows — every tier in full