Workflow Engine
Every agent execution is driven by the workflow engine: a BFS state-graph executor that runs a queue of nodes, checkpointing after each one, and dispatching to one of 11 node type handlers.
Execution Lifecycle
Create execution record → Validate workflow graph ↓BFS loop ├── pop node from queue ├── enforce timeout / visit limits / budget ├── dispatch to node handler ├── checkpoint to Redis (hot) └── queue activated downstream nodes ↓Terminate (Completed / Failed / paused state)Termination States
| State | Trigger |
|---|---|
Completed | BFS queue fully drained |
Failed | Timeout, visit limit, budget cap, or unrecoverable error |
AwaitingApproval | HumanGate node paused — waiting for human decision |
WaitingForExternalEvent | MediaGeneration node submitted — waiting for Fal.ai webhook |
AwaitingClientTool | LLM emitted a client-tool call — waiting for your app to fulfil |
AwaitingToolApproval | LLM called an MCP tool marked needs_approval — waiting for a decision via POST /executions/{id}/tool-approval. When the connector names approvers, only a matching workspace member decides and the route answers 403 to every API key; otherwise your app (the run’s own key) may decide |
Cancelled | Cancelled via API |
Node Types
| # | Node | Purpose |
|---|---|---|
| 1 | Trigger | Entry point — emits initial parameters and input |
| 2 | Ai (LLM) | LLM call via intelligence layer — sync or streaming, with output-schema validation retry |
| 3 | Router | Branch selection — rules engine, round-robin, random, or LLM-based |
| 4 | Parallel | Fan-out — run sub-graphs concurrently under a configurable semaphore |
| 5 | Join | Merge parallel results — concat, select_best, aggregate, first, last, llm_judge, or custom template |
| 6 | Memory | Recall (hybrid retrieval) or Learn (async episode queue) |
| 7 | HumanGate | Human-in-the-Loop pause — execution resumes on an approve/reject/skip decision |
| 8 | Action | HTTP / webhook dispatch with SSRF protection and configurable timeout |
| 9 | AgentInvocation | Nested agent execution as a child — max depth 3 |
| 10 | Logic | Conditionals, loops, variable mutation, schema validation, pass-through |
| 11 | MediaGeneration | Fal.ai image/video submission — the only node that pauses for an external event |
Execution Limits
| Limit | Default | Termination |
|---|---|---|
| Wall-clock timeout | 300 s | Failed — Timeout |
| Total node visits | 25 | Failed — RecursionLimitExceeded |
| Per-node visits | 10 | Failed — NodeVisitLimitExceeded |
| Aggregate output bytes | 50 MB | Failed — BadRequest |
| AgentInvocation depth | 3 | Validation error |
| Action node timeout | 30 s default, 600 s max | — |
| LLM stream buffer per node | 256 KB | Truncated, not failed |
Checkpointing
After every node execution, the engine writes a checkpoint so paused and crashed executions can resume:
| Tier | Store | TTL | Compression |
|---|---|---|---|
| Hot | Cache layer | 24 h | zstd-3 |
| Cold | Primary datastore | 30 d | — |
Write triggers (in addition to every-node saves):
| Trigger | When |
|---|---|
save_at_human_gate | HumanGate pause — includes gate info and timeout |
save_at_external_event | MediaGen pause |
save_at_client_tool_pause | LLM yields to client-tool fulfilment |
On resume, the hot checkpoint is tried first; cold is the fallback.
HumanGate (HITL)
When execution reaches a HumanGate node:
- Status →
AwaitingApproval human_gate_infosaved to execution record withtimeout_at- Your app shows an approval UI
To resume, POST to /executions/{id}/resume:
{ "decision": "approve", "comment": "Looks good — proceed", "variables": {}}| Decision | Effect |
|---|---|
approve | Execution continues downstream |
edit_approve | Continues like approve; send the edited values object |
reply | Continues like approve; send the answer in values and the reason in comment |
reject | Continues downstream, by the reject edge when the gate has one |
skip | Continues downstream, by an unlabeled edge |
values is only accepted with edit_approve or reply. It must be an object of
at most 64 KiB.
The gate’s output carries decision, reason (the comment you sent, also
served as comment),
approver_kind and approver_ref (who decided, for a workspace member user
and the user id), values and decided_at. When the gate sets
require_comment, a resume with a blank comment is refused with 400. A
schedule acknowledgement decides a gate with no reason, so require_comment does
not apply to it. The resume, tool-result and tool-approval responses, and the SSE
events, omit approver_kind, approver_ref and values from a gate’s output; the
execution record keeps them for workspace members.
Routing by decision
Label an outgoing edge of the gate approve, reject, edit_approve or
reply and the gate routes by decision: only the edge labeled with the decision
runs. A skip runs only the unlabeled edges. A gate with no edge labeled by a
decision keeps its earlier routing: every unconditioned edge runs on every
decision. A timeout route still runs its fallback_branch.
Routing and approvers
Set definition_key to the key of one of the workspace’s routing definitions.
The gate then opens a request when it pauses, in the same transaction that
parks the run, and the definition’s approver list names who may decide it. The
request closes when the run leaves the gate: decided with the decision logged,
expired when the server applied the timeout action, withdrawn otherwise (a
skip, a cancel, a failure). A keyed gate cannot sit inside a Parallel branch: the
branch fails. A gate with no
definition_key opens no request and keeps its earlier behaviour. Saving the
workflow refuses a definition_key that names no definition in the workspace.
A key that has since gone stops the gate from pausing: the node fails and the
gate never runs ungated.
If timeout_at has passed when you call /resume, the server applies the
node’s timeout_action and ignores the decision you send:
timeout_action | Effect |
|---|---|
approve | Continues as approve |
reject | Continues as reject |
skip | Continues as skip |
route | Continues as reject, but only along the fallback_branch edge |
fail | Fails the execution (timeout); /resume answers 409 |
escalate | POSTs the escalation webhook. A successful delivery leaves the gate open; /resume still accepts a decision. The run fails only after every delivery attempt fails |
The background timeout sweep fails the run for fail, and for escalate only after every delivery attempt fails. A delivered escalation webhook leaves the gate open. For
approve, reject, skip and route it does not yet apply the decision: it
ends the run as completed without running the nodes after the gate (issue
#1203 closes this gap). Only a /resume that arrives after the deadline and
before the next sweep applies those four actions. Once the sweep has ended the
run, /resume answers 409 execution_terminal.
Timeout configuration
timeout_mode says whether the gate has a deadline. timeout_action says what
happens when it arrives.
timeout_mode | Deadline | timeout_seconds |
|---|---|---|
none | None. The gate waits until a human decides. | Ignored |
immediate | Already due when the gate pauses. | Ignored |
after_duration | The pause time plus timeout_seconds. | Required, greater than 0 |
timeout_action is one of fail (the default), approve, reject, skip,
route (continues to the fallback_branch step, given as a downstream node id or edge label), escalate (POSTs
escalation_webhook and leaves the gate open when delivery succeeds; only an exhausted delivery failure ends the run), keep_waiting or hand_over. The
last two leave the gate open at the deadline for a person: neither applies a
decision, and /resume after the deadline takes the person’s decision. A gate
saved without timeout_mode keeps its earlier meaning: no timeout_seconds is
none, 0 is immediate, and above 0 is after_duration.
timeout_policy sets the same action and an optional backup approver:
{ "timeout_mode": "after_duration", "timeout_seconds": 3600, "timeout_policy": { "backup_approver": { "role": "admin" }, "action": "reject" }}backup_approver is {"role": "owner" | "admin" | "member"} or
{"user_id": "<uuid>"}. When timeout_policy is present it replaces
timeout_action, and auto_reject cannot be combined with it. The backup
approver is stored here; the escalation worker reassigns the gate to it. A gate
whose deadline is none has no timeout action, so “no timeout” and “0 seconds”
stay two different timeout_mode values and neither reads as fail.
recheck_seconds (above 0, or absent) sets how often a missed decision is
looked for. It is stored with the gate; the HITL processor runs the check.
Saving the workflow refuses a gate that cannot behave as written:
after_duration without a timeout_seconds above 0, and a prompt or
timeout_seconds of the wrong type. It also refuses a gate that has a deadline
and cannot carry out its action: route without a fallback_branch, route
to a fallback_branch the gate has no outgoing edge to (by target node id or
edge label), or escalate without an http(s) escalation_webhook. Only a gap
the save adds
is refused. A gate already stored with that gap (a route with no step acts as
reject, an escalate with no URL fails the run) runs as it always did, so it
still saves, copies, rolls back, promotes and migrates unchanged; changing it
to the other gap, or giving a gate that had no deadline one, is refused. A gate
with timeout_mode: none never applies its action, so it is not checked for
one. An explicit timeout_mode is authoritative at run time: a snooze_until
value in the execution’s input variables moves the deadline only of a gate saved
without timeout_mode; none and immediate ignore it. A gate saved with no prompt at all (the message shape some system agents carry) cannot be read
at run time and waits for a human with no deadline, whatever else it sets.
MediaGeneration Node
MediaGeneration is the only async-pausing node:
- Path A (first visit): Submits job to Fal.ai with a webhook URL. Returns
immediately with status
WaitingForExternalEvent. - Fal.ai delivers result via webhook — Athena injects
ai_result:{node_id}into execution variables and resumes from checkpoint. - Path B (resume): Node detects
ai_result:{node_id}in variables and processes the result, continuing the workflow.
Client-Tool Calls
If an LLM node emits a tool call for a client-registered tool, execution suspends:
- Status →
AwaitingClientTool - SSE emits
tool_call_awaiting_clientevent - Your app runs the tool locally and POSTs result to
POST /executions/{id}/tool-result
{ "tool_call_id": "tc_abc123", "outcome": "success", "result": { "balance": 42.50 }}The last fulfilled tool call resumes execution automatically.
Retry Policy
Individual nodes can be configured with retry and timeout settings:
| Retryable errors | Non-retryable |
|---|---|
ExternalService, RateLimit, ServiceUnavailable | Validation, NotFound, Unauthorized, Forbidden, PaymentRequired, Conflict |
Retry delay is linear by default; configurable to exponential backoff
(base_delay × 2^attempt, capped at 5 minutes).
Related Docs
- Architecture — 3-layer execution pipeline
- Intelligence — LLM dispatch invoked by Ai/LLM node
- Credits — credit pre-deduct/reconcile wired into every AI node
- Execute — client-tool calls, HITL resume, SSE events
- Human-in-the-loop — events, processor, adapters, Spaces
- Error Codes — workflow error codes