Getting Started with Testing Flows

This page walks through testing flows in the order you will need it: from a new repository, through the tests you write every day, to a flow tested on a live instance. Each step gives the commands and the decisions, and links to Testing Flows for the full explanation of each tier.

  1. Set up a repository

  2. Mark each test with its tier

  3. Add a flow and its tests

  4. Run the right tests after a change

  5. See what your tests cover

  6. Test on a live instance with synthetic data

  7. Evaluate on customer data, where only that will do

  8. Upgrade the SDK

1. Set up a repository

Create the virtual environment with Python 3.14, the latest version a PythonBlock supports. Each custom code block is built for the Python version of the environment you build the flow in, and the platform flags a version it is about to stop supporting when you import the flow. Python 3.14 blocks need platform version 44 or later; for an older instance, use the newest version it supports, or set HS_PYTHON_CODE_BLOCK_PYTHON_VERSION — see Managing Third-Party Python Packages.

pip install "flows_sdk[testing]"
hs-flows init invoice_processing --flow-name invoice
cd invoice_processing
pip install -e ".[dev]"                 # the project, plus ruff and mypy
pytest -m "not e2e and not in_flow"     # every offline tier

hs-flows init writes a project with one flow and a test for every tier, along with the ruff, mypy and pytest configuration, including -p flows_sdk.testing.plugin, which enables the framework’s pytest plugin. It works offline. ruff check ., ruff format --check ., mypy . and pytest all pass in the directory it creates.

The flow and its tests are placeholders: the flow returns a fixed value, and each tier checks it. Replace them with your own. The comments in each file say what belongs there, and link to the part of Testing Flows that shows it for a real document-processing flow.

invoice_processing/
├── pyproject.toml          # tool configuration, the tier markers and the pytest plugin
├── conftest.py             # puts the project on sys.path for pytest
├── README.md
├── .flows-sdk/             # SDK-owned config, refreshed by `hs-flows update`
│   ├── ruff.toml
│   └── mypy.ini
└── invoice_processing/
    └── flows/invoice/
        ├── flow.py         # entry_point_workflow() -> Flow
        ├── functions.py    # the custom code — plain, importable functions
        └── tests/
            ├── unit/       # each function, called directly
            ├── flow/       # the whole graph offline, your code running for real
            ├── contract/   # wiring, interface and the contract snapshot (golden in snapshots/)
            ├── e2e/        # a live instance, with the synthetic data in data/
            ├── in_flow/    # customer-data evaluation, run as flows on the instance
            └── data/       # synthetic data: the flow's expected output

Tests live inside the flow’s own package, so one directory holds a flow and everything that proves it; a second flow gets its own tests/. Keep custom code in functions.py as named functions: everything below is easier when the code under test can be imported.

2. Mark each test with its tier

Each test module says which tier it belongs to with an ordinary pytest marker:

import pytest

pytestmark = pytest.mark.unit

hs-flows init declares the markers — unit, flow, contract, e2e, in_flow — in pyproject.toml and marks every test it generates. They are yours to extend or drop; nothing applies or requires them. Two habits keep them reliable:

  • Select the offline run by exclusion, pytest -m "not e2e and not in_flow", so a test someone forgot to mark still runs.

  • Pair a live marker with its skip, pytestmark = [pytest.mark.e2e, requires_instance()], so the live tiers skip cleanly where no instance is configured.

The folder a test sits in means nothing to pytest. See Marking tests with their tier.

3. Add a flow and its tests

Write flows/<name>/flow.py and functions.py, then add its tests in order of value — cheapest first:

  1. Contract. One class beside the flow gets wiring, interface, Python version and snapshot checks, with no stubs and no instance. If the flow calls subflows, declare them too, so each call site is checked:

    from pathlib import Path
    
    import pytest
    from flows_sdk.testing.suites import FlowStaticChecksSuite
    
    from invoice_processing.flows.invoice.flow import entry_point_workflow
    
    pytestmark = pytest.mark.contract
    
    
    class TestInvoiceContract(FlowStaticChecksSuite):
        flow = entry_point_workflow
        snapshot_path = Path(__file__).parent / 'snapshots' / 'invoice.json'
    

    Create the golden once, then commit it:

    UPDATE_SNAPSHOTS=1 pytest -m contract
    

    See Wiring tests: validate references without stubs and Contract tests: flow snapshots.

  2. Unit. A test for each function in functions.py, calling it directly — see Unit tests: functions.

  3. Flow graph. One test per path that matters, typically one per routing branch. Stub the platform blocks from a recorded run of a synthetic document rather than by hand — see Stubs come from a real run.

4. Run the right tests after a change

Start with the cheapest tier that can see the change:

You changed

Run first

a reference, a block, the order of blocks

pytest -m contract

the code of a custom code block

pytest -m unit

how blocks hand data to one another

pytest -m flow

a declared input or output, a role

pytest -m contract, then refresh the golden with UPDATE_SNAPSHOTS=1 if the change is intended

Then, before you push, everything offline:

pytest -m "not e2e and not in_flow"

A moved golden means a caller of the flow is affected, so read its diff before committing it.

5. See what your tests cover

Use ordinary line coverage. A custom code block’s code is a plain importable function, so pytest-cov over the flows package measures your flow logic directly. The tests live inside that package, so the generated pyproject.toml leaves */tests/* out of the measurement:

pytest -m "unit or flow" --cov=invoice_processing.flows.invoice --cov-report=term-missing

Run it over the unit and flow tiers: the flow tier drives your code through the graph and reaches paths per-function tests miss. Read the two kinds of file differently:

  • functions.py is the real measurement — an uncovered line is a branch nothing exercises.

  • flow.py nearly all executes the moment the flow is built, so a high number says little; an uncovered line usually means a block or branch no test builds.

Line coverage cannot tell you whether a routing branch was taken. Assert it in a flow test, one per branch, and assert the untaken side too — runner.assert_blocks_did_not_run(...) — or the test also passes when both sides run. See Scenarios edit the recording.

6. Test on a live instance with synthetic data

Most of what a flow does on the platform — that it deploys and completes, which way it routes, what happens after supervision, how it uses the knowledge store — is behaviour, and a synthetic document exercises it as well as a real one. That is the e2e tier: ordinary pytest that deploys a copy of the flow from your checkout, sends it data from tests/data/, and asserts on exact values. The scaffold’s flow declares its identifier with flow_identifier so it can be copied — see Flows that call each other.

Create a machine-to-machine credential on a development instance, in the System Admin group: the tests import and deploy flows, which needs the import_flows permission, and on a fresh instance no other group has it. The UI shows the credential’s Client ID and Client Secret (JWK); paste the JWK into a file, and point the tests at it:

export HS_BASE_URL=https://dev-instance
export HS_M2M_CLIENT_ID=<the client id shown next to the key>
export HS_M2M_CREDS=~/.hyperscience/m2m_creds.json   # the Client Secret (JWK)
pytest -m e2e

Run it in CI against the development instance. Without those variables the tier skips, so the rest of the suite stays runnable anywhere. See End-to-end tests: the repository owns the data.

7. Evaluate on customer data, where only that will do

Keep customer data for what a synthetic document cannot answer — typically evaluation, such as whether extraction stays accurate on what a customer actually sends. Customer documents must not leave the customer’s environment, so these tests are flows, deployed to the customer’s instance, that assert there and report back only pass or fail.

The scaffold’s tests/in_flow/ holds the pieces: evaluation.py (submit and wait, then compare, as plain code), the suite flow that runs it, the wrapper flow that holds the test document, and test_in_flow.py. They are placeholders too: the suite submits the document to the generated flow and checks that the run returns its fixed output. Then:

  1. Deploy the flow under test. The suite submits the document to the flow the wrapper names, so that flow must be deployed on the instance. On a customer’s instance it usually is already: it is the flow you are evaluating. The e2e tier’s copy does not count, since it is removed after each run. For the generated flow, python -m invoice_processing.flows.invoice.flow writes build/invoice.json; import it in Flow Studio and deploy it.

  2. Run it. pytest -m in_flow, with the credentials above pointed at the customer’s instance. The first run uploads both test flows and fails, because nothing in your repository can supply a customer document. The error links to the wrapper’s settings.

  3. Configure the wrapper in Flow Studio: attach the document. A real suite also sets the result expected from it, approved from a run you inspected — see Where the expected result comes from.

  4. Run it again. pytest reports the suite’s flow run as one test. When a case fails, the test fails with a link to the run, where you read which case failed and why.

The configured wrapper is the only record of what the tests run against, so a run that would overwrite it fails instead. After changing the wrapper’s code, re-run with --merge-test-flows to keep its data, or --force-test-flows to discard it — see The wrapper is protected.

8. Upgrade the SDK

pip install -U "flows_sdk[testing]"
hs-flows update                         # refresh the SDK-owned ruff and mypy config
pytest -m contract
pytest -m "not e2e and not in_flow"

hs-flows update overwrites the configuration the SDK owns with the copy in the new wheel, so lint and type rules stay in step across every flows repository. A contract golden records only what a caller of the flow can observe, so if one moves after an upgrade, review it: the new SDK changed your flow’s interface.

See also