> ## Documentation Index
> Fetch the complete documentation index at: https://forest-docs-forest-runtime-automated-inboxes.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Forest Runtime
> Run your workflow steps on your own infrastructure — the Forest orchestrator never sees your records.
Forest's orchestrator coordinates your workflows but never executes the steps itself — that runs on **Forest Runtime**, deployed on your own infrastructure. Forest Runtime polls the orchestrator for pending steps, runs them locally, and reports only the results back.
Setting it up is a **one-time addition, not a migration**: you deploy Forest Runtime and either point your agent at it or embed it in the agent — your collections and existing workflows are untouched.
Because it talks to your data through your own Forest agent, **the Forest orchestrator never sees your records** — data read and written by data steps stays within your infrastructure.
One exception: **AI steps and MCP Tasks** send prompt content, which can include record data, to an LLM provider and remote tools — so workflows aren't fully air-gapped. See [AI provider](#ai-provider).
## Do I need Forest Runtime?
Yes — Forest Runtime is what executes your workflow steps. **[Webhook-triggered workflows](/product/process/workflows/triggers) especially**: a webhook can fire at any time, with no guarantee anyone has Forest open in a browser, so they can only run server-side.
Running on infrastructure you control also keeps the records handled by data steps out of Forest's infrastructure — decisive for compliance or data-residency requirements, or when a step needs access to systems reachable only from within your network.
## How it works
1. A workflow is triggered — by a user or a [webhook](/product/process/workflows/triggers). The Forest orchestrator queues the pending steps.
2. Forest Runtime polls the orchestrator and pulls the steps assigned to it.
3. Each step runs locally, reaching your data and actions through your Forest agent.
4. It reports the step outcome back to the orchestrator, which advances the workflow.
```mermaid theme={null}
flowchart LR
O["Forest orchestrator
(coordinates)"] -- "pending steps" --> R["Forest Runtime
(your infrastructure)"]
R -- "data & actions" --> A["Your Forest agent"] --> DB[("Your business
database")]
R -. "state (DATABASE_URL)" .-> P[("Forest Runtime's
Postgres")]
R -- "outcomes only, never records" --> O
```
## Prerequisites
* A recent **Forest Admin agent** — the minimum version depends on how you run Forest Runtime (below).
* A **PostgreSQL** database for production: Forest Runtime persists its run state there. A database-free mode exists for testing only.
## Running Forest Runtime
Run Forest Runtime one of two ways:
* **[Embedded in your Node.js agent](#embedded-in-the-nodejs-agent)** — one line in your agent, nothing separate to deploy. The simplest option.
* **[Standalone](#standalone-docker-or-cli)** — a separate process (Docker or CLI), to scale it independently of your agent or to use it with a Ruby agent.
### Embedded in the Node.js agent
Requirements:
* **`@forestadmin/agent`** — embedded mode arrived in **1.84.0**, but the options shown below (`schema`, `ai`, `encryptionKey`, the tuning knobs) landed in later releases, so use a **recent** version.
* **Node.js ≥ 22.12.0** — required by the executor package.
* **The executor package**, installed at the **exact version your agent pins**. It's an *optional peer dependency* the agent loads dynamically at runtime, so `tsc` won't flag it when it's missing — but `agent.start()` throws *"The embedded workflow executor requires the `@forestadmin/workflow-executor` package"*. The pin is exact (a `^` range conflicts), so install that specific version — find it in your lockfile or with `npm info @forestadmin/agent@ peerDependencies` (pin the version you actually run — without it, npm answers for `latest`, whose pin may differ):
```bash theme={null}
npm install @forestadmin/workflow-executor@
```
* **TypeScript 5.5+** — the package ships zod 4, which is only tested against TypeScript 5.5 and later. On TypeScript 4.x its typings can't even be parsed, and `skipLibCheck` won't help (it suppresses type errors, not the syntax errors these typings trigger).
Then add one line — the executor runs inside the agent process, so there's nothing else to deploy:
```js theme={null}
createAgent(options)
.addDataSource(/* ... */)
.addWorkflowExecutor({ database: { uri: process.env.DATABASE_URL } })
.start();
```
Safe to reuse the database your agent already reads: the executor keeps its tables in a dedicated **`forest`** Postgres schema, so they stay out of the `public` schema your datasource introspects and never show up as collections in your panel. Pass a `schema` if you'd rather name it differently.
Creating that schema needs the `CREATE` privilege on the database. If your role doesn't have it, have an administrator create the schema and grant the role access to it — the executor checks whether the schema already exists before trying to create one, so a pre-created schema boots fine with schema-level privileges only.
It inherits your agent's secrets and Forest connection, so you only configure:
| Option | Description |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database` | Postgres connection — a URI or Sequelize options. Persists run state. Its tables live in the `forest` schema unless you pass a different `schema` here. Required unless `inMemory`. |
| `inMemory` | `true` runs without a database (testing only — runs are lost on restart). |
| `agentUrl` | How the executor reaches your agent. Auto-derived when the agent runs on its own server; **required** when the agent is mounted on Express, Fastify or NestJS. |
| `port` | Loopback port the executor listens on internally (default `3400`). |
| `ai` | Bring your own LLM instead of Forest's AI server: `{ provider: 'anthropic' \| 'openai', model, apiKey }`, or `{ provider: 'bedrock', model, region }`. [Bedrock](#amazon-bedrock) takes no `apiKey` — it signs in through AWS — and its `region` falls back to `AWS_REGION` / `AWS_DEFAULT_REGION`, one of which is required. Omit to keep using Forest's server. |
| `encryptionKey` | At-rest key (AES-256-GCM) for [OAuth-protected MCP connector](#oauth-protected-mcp-connectors) credentials. Generate with `openssl rand -hex 32`. Omit if you don't use them. |
| `pollingIntervalS`, `automationPollingIntervalS`, `stepTimeoutS`, `aiInvokeTimeoutS`, `stopTimeoutS`, `maxChainDepth`, `schemaCacheTtlS` | The same tuning knobs as [standalone](#tuning), in camelCase — same defaults. Log verbosity follows your agent's own logger, so there's no separate log-level option here. |
Embedded has full configuration parity with standalone: your own AI provider (`ai`), the encryption key, and every tuning knob are all settable here. It only inherits your agent's secrets and Forest connection; set nothing and AI steps use Forest's AI server.
### Standalone (Docker or CLI)
Run Forest Runtime as its own service — the way to run it with any agent other than the v2 Node.js one (which can also [embed](#embedded-in-the-nodejs-agent) it), and to scale or deploy it separately.
**1. Point your agent at it** with the workflow executor URL, so the agent mounts the route that forwards workflow requests to Forest Runtime (relaying the JWT). It's supported across both agent generations:
| Agent | Config | Min version |
| ----------------------------------------- | -------------------------------------- | ----------- |
| `@forestadmin/agent` — Node.js (v2) | `createAgent({ workflowExecutorUrl })` | 1.79.0 |
| `forest-express-sequelize` — Node.js (v1) | `Liana.init({ workflowExecutorUrl })` | 9.6.4 |
| `forest-express-mongoose` — Node.js (v1) | `Liana.init({ workflowExecutorUrl })` | 9.6.3 |
| `forest_admin_rails` — Ruby (v2) | `config.workflow_executor_url` | 1.31.0 |
| `forest_liana` — Ruby (v1) | `ForestLiana.workflow_executor_url` | 9.18.0 |
The route itself lives in `forest-express`, which both wrappers pin to an exact version. The minimums above pull in `forest-express` 10.7.0, so that's the number to look for in a lockfile. You never install or configure it yourself.
```js Node.js (v2) theme={null}
createAgent({
// ...
workflowExecutorUrl: process.env.WORKFLOW_EXECUTOR_URL, // e.g. http://localhost:3400
})
```
```js forest-express-sequelize / -mongoose (v1) theme={null}
Liana.init({
// ...
workflowExecutorUrl: process.env.WORKFLOW_EXECUTOR_URL,
})
```
```ruby forest_admin_rails (v2) theme={null}
# config/initializers/forest_admin_rails.rb
ForestAdminRails.configure do |config|
# ...
config.workflow_executor_url = ENV['WORKFLOW_EXECUTOR_URL']
end
```
```ruby forest-rails (v1) theme={null}
# config/initializers/forest_liana.rb
ForestLiana.workflow_executor_url = ENV['WORKFLOW_EXECUTOR_URL']
```
The Python agents (`agent-python`, `django-forestadmin`) don't support the workflow executor yet.
If the workflow executor URL is left unset, the agent returns `404` on those routes and Forest Runtime never receives any work. When the agent and Forest Runtime run on separate hosts, use an internal address the agent can reach on Forest Runtime's HTTP port (default `3400`).
**2. Run the executor** as a Docker image or via the CLI:
```bash Docker theme={null}
docker run -d \
--add-host=host.docker.internal:host-gateway \
-e FOREST_ENV_SECRET="your-env-secret" \
-e FOREST_AUTH_SECRET="your-auth-secret" \
-e AGENT_URL="http://host.docker.internal:3351" \
-e DATABASE_URL="postgres://user:pass@host.docker.internal:5432/mydb" \
-p 3400:3400 \
ghcr.io/forestadmin/workflow-executor:latest
```
```bash npx theme={null}
FOREST_ENV_SECRET="your-env-secret" \
FOREST_AUTH_SECRET="your-auth-secret" \
AGENT_URL="https://your-agent-url" \
DATABASE_URL="postgres://user:pass@localhost:5432/mydb" \
npx @forestadmin/workflow-executor
```
| Variable | Required | Description |
| -------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `FOREST_ENV_SECRET` | yes | Your environment secret — same value as your agent. Also in [app.forestadmin.com](https://app.forestadmin.com) → Settings → Environments. |
| `FOREST_AUTH_SECRET` | yes | The same `authSecret` value your agent is configured with — ask whoever operates the agent. |
| `AGENT_URL` | yes | URL where your Forest Admin agent is running (e.g. `http://localhost:3351`). |
| `DATABASE_URL` | yes | Postgres connection string. Required unless you pass the `--in-memory` flag (testing only — state is lost on restart). |
When Forest Runtime runs in Docker and your agent runs on the host machine, use `host.docker.internal` instead of `localhost` in `AGENT_URL` and `DATABASE_URL`. On Linux Docker Engine that hostname doesn't exist by default, so the `--add-host=host.docker.internal:host-gateway` flag above is required to resolve it (on Docker Desktop it's already provided and the flag is harmless).
### Network requirements
A standalone Forest Runtime opens these connections (an embedded executor makes the same outbound calls from the agent process, with no extra inbound port):
| Direction | Flow | Purpose |
| --------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| Inbound | Agent → Forest Runtime `:3400` | The agent forwards workflow requests. |
| Outbound | Forest Runtime → `api.forestadmin.com` (HTTPS) | Polling the orchestrator for pending steps. |
| Outbound | Forest Runtime → `AGENT_URL` | Data steps read/write through your agent. |
| Outbound | Forest Runtime → your Postgres | State persistence. |
| Outbound | Forest Runtime → LLM provider & remote MCP tools | AI steps and MCP Tasks (Forest's AI server by default, or your own provider — see [AI provider](#ai-provider)). |
On first boot Forest Runtime auto-creates its `workflow_step_executions` table, then polls the orchestrator every 30 seconds (`POLLING_INTERVAL_S`) for work.
Forest Runtime is stateless apart from its Postgres database: you can run several instances against the same database for high availability — each pending step is claimed by exactly one instance. If you use [OAuth-protected MCP connectors](#oauth-protected-mcp-connectors), give every instance the same encryption key.
### Health check
`GET /health` is public (no auth) and returns the runtime's current state:
```bash theme={null}
curl http://localhost:3400/health
# {"state":"running"}
```
| State | HTTP | Meaning |
| ---------- | ----- | ------------------------------------------------------------------------------------------------------------------ |
| `running` | `200` | Started and polling for work. |
| `draining` | `200` | Graceful shutdown — finishing in-flight steps (up to `STOP_TIMEOUT_S`, 30s by default), no longer taking new runs. |
| `idle` | `503` | Not started yet. |
| `stopped` | `503` | Shut down. |
For **liveness** probes, treat any `200` as healthy. For **readiness** probes, route traffic only on `{"state":"running"}` so a draining instance stops receiving new work while it finishes in-flight steps.
## AI provider
Several step types rely on an LLM: guidance, decisions, MCP Tasks, and AI-assisted data steps. By default Forest Runtime uses **Forest's AI server** — no configuration required, AI steps work out of the box.
To keep AI calls off Forest's server and use your **own provider** instead, set these variables:
| Variable | Description |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| `AI_PROVIDER` | `anthropic`, `openai` or `bedrock`. |
| `AI_MODEL` | Model name for that provider (e.g. `gpt-4.1`, `claude-sonnet-5`, `eu.anthropic.claude-sonnet-5`). |
| `AI_API_KEY` | Your API key for that provider. **Leave it unset for `bedrock`**, which signs in through AWS instead. |
`AI_PROVIDER` and `AI_MODEL` always go together. Add `AI_API_KEY` for `openai` and `anthropic`; leave it out for `bedrock` — setting it there stops Forest Runtime at startup instead of being ignored. Set none of them to keep using Forest's AI server.
### Amazon Bedrock
With `AI_PROVIDER=bedrock`, Forest Runtime reaches the model through Amazon Bedrock, so the calls stay inside your AWS account. Set `AI_MODEL` to a Bedrock model id, a cross-region inference profile id, or an inference profile ARN — and `AWS_REGION` to the region the model is enabled in.
To find the id, open AWS's [Models at a glance](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html) and pick your Claude model: its page carries the model id, and an **Inference profile IDs** section with the `eu.` and `us.` ones. Either can be copied straight into `AI_MODEL`.
**Supported models: Anthropic Claude Sonnet, Haiku and Opus.** Any other Bedrock model is refused at startup, with a message naming what is accepted. AI steps ask the model to answer in a strict, structured format, and the rest of the Bedrock catalogue either can't do that reliably or hasn't been tested on our side — so a model we can't vouch for is rejected immediately, rather than breaking halfway through a workflow.
A cross-region inference profile id works and is the usual choice, since it keeps your data in one geography: `eu.` stays in the EU, `us.` in the US. Avoid the `global.` prefix if data residency is the reason you are on Bedrock, as it routes worldwide.
Bedrock takes **no** `AI_API_KEY`. Forest Runtime signs in to AWS exactly the way the AWS CLI does, so whatever already works for your other AWS tools works here too — you don't configure anything specific to Forest. Whichever method you use, it needs permission to call `bedrock:InvokeModel` on the models you picked.
That includes [assuming a role](https://docs.aws.amazon.com/sdkref/latest/guide/access-assume-role.html), so you never have to hand long-lived keys to the container:
| Setup | What to provide |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Task role (ECS/EKS/EC2) | Nothing — AWS provides the credentials automatically. |
| Role assumed from a profile | Mount your AWS config with `role_arn` + `source_profile`, and set `AWS_PROFILE`. In Docker, mount it to **`/home/node/.aws`** — see the note below. |
| IRSA / service account (EKS) | `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`, both injected by the cluster. |
| Static keys | `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. |
Always set `AWS_REGION` (or `AWS_DEFAULT_REGION`), **even if your AWS profile already has a region** — that one isn't picked up here. Forest Runtime checks this at startup, so a missing region stops it right away instead of breaking the first workflow that uses AI.
**Mounting an AWS profile into Docker:** the image runs as the `node` user, so mount your AWS directory to `/home/node/.aws`, not `/root/.aws`. Mounting it to `/root/.aws` fails quietly — the container starts, and the first AI step reports that no credentials could be loaded.
```bash theme={null}
-v ~/.aws:/home/node/.aws:ro -e AWS_PROFILE=your-profile
```
To mount somewhere else, point `AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` at the files directly.
```bash theme={null}
AI_PROVIDER=bedrock
AI_MODEL=eu.anthropic.claude-sonnet-5
AWS_REGION=eu-west-1
```
## Automated inboxes
An [inbox](/product/manage/inbox) backed by a segment can start a workflow on its own, on every
record that enters it, with no one clicking anything. Forest Runtime is what sweeps the segment.
**It decides nothing.** It reads your data and reports what it saw; Forest starts the runs, applies
the limit of concurrent runs, and refuses a record a workflow is already handling. A runtime that
cannot reach Forest, or that you turn off, stops the automation — it never starts work twice.
### How a sweep works
Every `AUTOMATION_POLL_INTERVAL_S`, for each automated inbox:
1. It asks Forest which automated inboxes to sweep, and which records are already being handled.
2. It asks your agent whether the records whose workflow has finished are still in the segment.
3. It asks your agent for the first page of the segment, to find records nobody has handled yet.
4. It reports both answers back. Forest starts a workflow per new record.
```mermaid theme={null}
flowchart LR
R["Forest Runtime
(your infrastructure)"] -- "which inboxes?" --> O["Forest orchestrator"]
R -- "who is in the segment?" --> A["Your Forest agent"] --> DB[("Your business
database")]
R -- "what I saw" --> O -- "starts the workflows" --> O
```
A record leaves the automation when it **leaves the segment**. That is the only signal that it was
treated: a record whose workflow finished while it is still in the segment is handed to a human in
the workflow's fallback inbox instead of being run again.
### Requirements
* Forest Runtime on the version that ships automated inboxes or above. An older one is not served
the configuration, and the inbox shows as configured but inactive in its settings.
* A **segment-backed inbox**, with automation enabled on it. It is configured on the inbox, not in
the workflow's Process section.
### Running several instances
Nothing to configure. Each instance identifies itself and Forest hands the sweep to one of them, so
several instances do not sweep the same segment in parallel.
Two consequences worth knowing:
* If Forest cannot run that election, every instance sweeps. That costs duplicate reads on your
database, and it is deliberate — the alternative would stop every automation silently. No record
is ever run twice.
* After you restart or redeploy your runtime, the new process waits for the previous one's turn to
lapse before it starts sweeping. Expect up to ten minutes with no sweep.
### What the sweep can read
* **The service account's permissions apply.** The runtime reads the segment as the account the
automation runs as, so a restrictive scope on that account's role hides records from the sweep.
* **Relative dates** in a segment filter (`previous 30 days`, `today`…) are evaluated in your
project's timezone, or UTC when it has none.
* **SQL segments** need a connection name on agents that support several connections. On
`forest-rails` and `forest-express-sequelize`, which run the query against their single database,
none is needed.
### In your logs
| Line | Level | Meaning |
| -------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ |
| `Automated inbox polled` | `Info` | A sweep completed. Carries how many records were started, skipped or handed to a human. |
| `Automated inbox poll failed` | `Error` | One inbox failed to sweep this cycle. The others are unaffected. |
| `Could not reach the agent, reporting nothing for this inbox` | `Error` | Your agent did not answer. Nothing is reported, so the inbox reads as not swept rather than as swept with nothing to do. |
| `Automated inbox no longer served, dropping it for this cycle` | `Info` | Forest stopped serving this inbox — disabled, or its configuration is no longer valid. |
| `The orchestrator does not serve automated inboxes` | `Warn` | Said once. Expected if your runtime is newer than Forest; check `FOREST_SERVER_URL` if it persists. |
## OAuth-protected MCP connectors
If your workflows include [MCP Tasks](/product/process/workflows/overview) backed by OAuth-protected connectors, Forest Runtime stores each user's OAuth credentials in its database, encrypted at rest. Provide the encryption key:
| Variable | Description |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FOREST_EXECUTOR_ENCRYPTION_KEY` | At-rest encryption key (AES-256-GCM) for stored OAuth credentials. Generate with `openssl rand -hex 32`. Use a **separate** secret from `FOREST_AUTH_SECRET`. |
* Required **only** for OAuth-protected MCP connectors, and read lazily — an instance that stores no such credentials runs fine without it.
* Use the **same value on every instance** that shares a database, or an instance won't decrypt credentials written by another.
* Treat it as permanent: there is no managed rotation. Changing it forces every affected user to reconnect their connectors.
## Observability
The Docker image ships with [OpenTelemetry](https://opentelemetry.io/) APM built in, compatible with any OTLP backend (Datadog, Grafana Tempo, Jaeger, Honeycomb…). It is **off by default** and turns on as soon as you set `OTEL_EXPORTER_OTLP_ENDPOINT`. OpenTelemetry is bundled only in the Docker image, not the npm package.
### When an MCP Task can't load its tools
Forest Runtime logs the reason at `Error`, so it is in your logs without changing `LOG_LEVEL`. This is the JSON the Docker image writes to stdout; running in a terminal you get the same fields in the pretty single-line format instead.
```json theme={null}
{
"level": "Error",
"message": "MCP servers failed to load tools",
"timestamp": "2026-08-10T09:14:22.031Z",
"requestedMcpServerId": "39",
"mcpServerName": "acme-crm",
"failures": [
{ "server": "acme-crm", "kind": "connection", "error": "connect ECONNREFUSED 10.0.4.12:8080" }
]
}
```
`kind` tells you where to look:
* `auth` — the server rejected the credential (HTTP 401). For an OAuth connector the runtime refreshes the token and retries once on its own, so act only if the failure repeats without a follow-up `MCP tools loaded after refreshing the credential` line. For a static credential, renew it in the connector's configuration.
* `connection` — unreachable, refused, or slower than the 15-second per-server load timeout.
* `unknown` — the server answered but the load failed anyway, including HTTP 403 permission or scope errors that no token refresh can fix; `error` carries the reason.
A server that answers but exposes no tools is not a failure: you get an empty tool list and no error.
If your logs show a `failedConfigNames` list instead of `failures`, your runtime predates this change: it names which server failed but not why, and reports a healthy server exposing no tools as a failure. Upgrade to get the cause.
## Tuning
Beyond the required variables, these optional knobs have sensible defaults and rarely need changing:
| Variable | Default | Description |
| ---------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `HTTP_PORT` | `3400` | Port Forest Runtime's HTTP server listens on. |
| `POLLING_INTERVAL_S` | `30` | How often it polls the orchestrator for pending steps. |
| `AUTOMATION_POLL_INTERVAL_S` | `300` | How often it sweeps automated inboxes. Each sweep reads your database, so it runs an order of magnitude slower than the step poll. |
| `LOG_LEVEL` | `Info` | `Debug`, `Info`, `Warn`, or `Error`. `Debug` adds one line per MCP server with its tool count and load time. |
| `STEP_TIMEOUT_S` | `300` | Max duration of a single step. |
| `AI_INVOKE_TIMEOUT_S` | `30` | Max duration of a single AI provider invocation. |
| `STOP_TIMEOUT_S` | `30` | Grace period on shutdown to finish in-flight steps before exiting. |
| `MAX_CHAIN_DEPTH` | `50` | Max steps auto-executed per run before yielding. |
| `SCHEMA_CACHE_TTL_S` | `600` | Collection schema cache TTL. |
| `DATABASE_SCHEMA` | `forest` | Postgres schema Forest Runtime creates its tables in — already isolated from `public` by default; set it only to use a different name. |
For the remaining variables (individual database parts, `DATABASE_SSL`, in-memory testing mode, full OTel configuration), see the package [README on npm](https://www.npmjs.com/package/@forestadmin/workflow-executor).
## Learn more
Build and manage workflows in the no-code editor
Configure the connectors used by MCP Tasks