Skip to content

Workflow Engine

Every agent execution is driven by the workflow engine: a BFS state-graph executor that runs a queue of nodes, checkpointing after each one, and dispatching to one of 11 node type handlers.


Execution Lifecycle

Create execution record → Validate workflow graph
↓
BFS loop
├── pop node from queue
├── enforce timeout / visit limits / budget
├── dispatch to node handler
├── checkpoint to Redis (hot)
└── queue activated downstream nodes
↓
Terminate (Completed / Failed / paused state)

Termination States

StateTrigger
CompletedBFS queue fully drained
FailedTimeout, visit limit, budget cap, or unrecoverable error
AwaitingApprovalHumanGate node paused — waiting for human decision
WaitingForExternalEventMediaGeneration node submitted — waiting for Fal.ai webhook
AwaitingClientToolLLM emitted a client-tool call — waiting for your app to fulfil
AwaitingToolApprovalLLM called an MCP tool marked needs_approval — waiting for a decision via POST /executions/{id}/tool-approval. When the connector names approvers, only a matching workspace member decides and the route answers 403 to every API key; otherwise your app (the run’s own key) may decide
CancelledCancelled via API

Node Types

#NodePurpose
1TriggerEntry point — emits initial parameters and input
2Ai (LLM)LLM call via intelligence layer — sync or streaming, with output-schema validation retry
3RouterBranch selection — rules engine, round-robin, random, or LLM-based
4ParallelFan-out — run sub-graphs concurrently under a configurable semaphore
5JoinMerge parallel results — concat, select_best, aggregate, first, last, llm_judge, or custom template
6MemoryRecall (hybrid retrieval) or Learn (async episode queue)
7HumanGateHuman-in-the-Loop pause — execution resumes on an approve/reject/skip decision
8ActionHTTP / webhook dispatch with SSRF protection and configurable timeout
9AgentInvocationNested agent execution as a child — max depth 3
10LogicConditionals, loops, variable mutation, schema validation, pass-through
11MediaGenerationFal.ai image/video submission — the only node that pauses for an external event

Execution Limits

LimitDefaultTermination
Wall-clock timeout300 sFailed — Timeout
Total node visits25Failed — RecursionLimitExceeded
Per-node visits10Failed — NodeVisitLimitExceeded
Aggregate output bytes50 MBFailed — BadRequest
AgentInvocation depth3Validation error
Action node timeout30 s default, 600 s max—
LLM stream buffer per node256 KBTruncated, not failed

Checkpointing

After every node execution, the engine writes a checkpoint so paused and crashed executions can resume:

TierStoreTTLCompression
HotCache layer24 hzstd-3
ColdPrimary datastore30 d—

Write triggers (in addition to every-node saves):

TriggerWhen
save_at_human_gateHumanGate pause — includes gate info and timeout
save_at_external_eventMediaGen pause
save_at_client_tool_pauseLLM yields to client-tool fulfilment

On resume, the hot checkpoint is tried first; cold is the fallback.


HumanGate (HITL)

When execution reaches a HumanGate node:

  1. Status → AwaitingApproval
  2. human_gate_info saved to execution record with timeout_at
  3. Your app shows an approval UI

To resume, POST to /executions/{id}/resume:

{
"decision": "approve",
"comment": "Looks good — proceed",
"variables": {}
}
DecisionEffect
approveExecution continues downstream
edit_approveContinues like approve; send the edited values object
replyContinues like approve; send the answer in values and the reason in comment
rejectContinues downstream, by the reject edge when the gate has one
skipContinues downstream, by an unlabeled edge

values is only accepted with edit_approve or reply. It must be an object of at most 64 KiB.

The gate’s output carries decision, reason (the comment you sent, also served as comment), approver_kind and approver_ref (who decided, for a workspace member user and the user id), values and decided_at. When the gate sets require_comment, a resume with a blank comment is refused with 400. A schedule acknowledgement decides a gate with no reason, so require_comment does not apply to it. The resume, tool-result and tool-approval responses, and the SSE events, omit approver_kind, approver_ref and values from a gate’s output; the execution record keeps them for workspace members.

Routing by decision

Label an outgoing edge of the gate approve, reject, edit_approve or reply and the gate routes by decision: only the edge labeled with the decision runs. A skip runs only the unlabeled edges. A gate with no edge labeled by a decision keeps its earlier routing: every unconditioned edge runs on every decision. A timeout route still runs its fallback_branch.

Routing and approvers

Set definition_key to the key of one of the workspace’s routing definitions. The gate then opens a request when it pauses, in the same transaction that parks the run, and the definition’s approver list names who may decide it. The request closes when the run leaves the gate: decided with the decision logged, expired when the server applied the timeout action, withdrawn otherwise (a skip, a cancel, a failure). A keyed gate cannot sit inside a Parallel branch: the branch fails. A gate with no definition_key opens no request and keeps its earlier behaviour. Saving the workflow refuses a definition_key that names no definition in the workspace. A key that has since gone stops the gate from pausing: the node fails and the gate never runs ungated.

If timeout_at has passed when you call /resume, the server applies the node’s timeout_action and ignores the decision you send:

timeout_actionEffect
approveContinues as approve
rejectContinues as reject
skipContinues as skip
routeContinues as reject, but only along the fallback_branch edge
failFails the execution (timeout); /resume answers 409
escalatePOSTs the escalation webhook. A successful delivery leaves the gate open; /resume still accepts a decision. The run fails only after every delivery attempt fails

The background timeout sweep fails the run for fail, and for escalate only after every delivery attempt fails. A delivered escalation webhook leaves the gate open. For approve, reject, skip and route it does not yet apply the decision: it ends the run as completed without running the nodes after the gate (issue #1203 closes this gap). Only a /resume that arrives after the deadline and before the next sweep applies those four actions. Once the sweep has ended the run, /resume answers 409 execution_terminal.

Timeout configuration

timeout_mode says whether the gate has a deadline. timeout_action says what happens when it arrives.

timeout_modeDeadlinetimeout_seconds
noneNone. The gate waits until a human decides.Ignored
immediateAlready due when the gate pauses.Ignored
after_durationThe pause time plus timeout_seconds.Required, greater than 0

timeout_action is one of fail (the default), approve, reject, skip, route (continues to the fallback_branch step, given as a downstream node id or edge label), escalate (POSTs escalation_webhook and leaves the gate open when delivery succeeds; only an exhausted delivery failure ends the run), keep_waiting or hand_over. The last two leave the gate open at the deadline for a person: neither applies a decision, and /resume after the deadline takes the person’s decision. A gate saved without timeout_mode keeps its earlier meaning: no timeout_seconds is none, 0 is immediate, and above 0 is after_duration.

timeout_policy sets the same action and an optional backup approver:

{
"timeout_mode": "after_duration",
"timeout_seconds": 3600,
"timeout_policy": {
"backup_approver": { "role": "admin" },
"action": "reject"
}
}

backup_approver is {"role": "owner" | "admin" | "member"} or {"user_id": "<uuid>"}. When timeout_policy is present it replaces timeout_action, and auto_reject cannot be combined with it. The backup approver is stored here; the escalation worker reassigns the gate to it. A gate whose deadline is none has no timeout action, so “no timeout” and “0 seconds” stay two different timeout_mode values and neither reads as fail.

recheck_seconds (above 0, or absent) sets how often a missed decision is looked for. It is stored with the gate; the HITL processor runs the check.

Saving the workflow refuses a gate that cannot behave as written: after_duration without a timeout_seconds above 0, and a prompt or timeout_seconds of the wrong type. It also refuses a gate that has a deadline and cannot carry out its action: route without a fallback_branch, route to a fallback_branch the gate has no outgoing edge to (by target node id or edge label), or escalate without an http(s) escalation_webhook. Only a gap the save adds is refused. A gate already stored with that gap (a route with no step acts as reject, an escalate with no URL fails the run) runs as it always did, so it still saves, copies, rolls back, promotes and migrates unchanged; changing it to the other gap, or giving a gate that had no deadline one, is refused. A gate with timeout_mode: none never applies its action, so it is not checked for one. An explicit timeout_mode is authoritative at run time: a snooze_until value in the execution’s input variables moves the deadline only of a gate saved without timeout_mode; none and immediate ignore it. A gate saved with no prompt at all (the message shape some system agents carry) cannot be read at run time and waits for a human with no deadline, whatever else it sets.


MediaGeneration Node

MediaGeneration is the only async-pausing node:

  1. Path A (first visit): Submits job to Fal.ai with a webhook URL. Returns immediately with status WaitingForExternalEvent.
  2. Fal.ai delivers result via webhook — Athena injects ai_result:{node_id} into execution variables and resumes from checkpoint.
  3. Path B (resume): Node detects ai_result:{node_id} in variables and processes the result, continuing the workflow.

Client-Tool Calls

If an LLM node emits a tool call for a client-registered tool, execution suspends:

  1. Status → AwaitingClientTool
  2. SSE emits tool_call_awaiting_client event
  3. Your app runs the tool locally and POSTs result to POST /executions/{id}/tool-result
{
"tool_call_id": "tc_abc123",
"outcome": "success",
"result": { "balance": 42.50 }
}

The last fulfilled tool call resumes execution automatically.


Retry Policy

Individual nodes can be configured with retry and timeout settings:

Retryable errorsNon-retryable
ExternalService, RateLimit, ServiceUnavailableValidation, NotFound, Unauthorized, Forbidden, PaymentRequired, Conflict

Retry delay is linear by default; configurable to exponential backoff (base_delay × 2^attempt, capped at 5 minutes).