Startup Health

Overview

Navigate to Admin > Monitoring > Startup Health to see whether this environment finished its Django, graph, seed, and index bootstrap work. This page is app-level readiness, not replica scaling. Use Service Scaling to change how many pipeline workers are running. Reading the page requires Monitoring read access. Verify, Ensure, and Verify all require Monitoring write access.

Verify vs Ensure

Action	What it does
Verify	Read-only. Asks “is this check good right now?” and stores an audit row.
Verify all	Runs Verify for every applicable check. Does not create or repair anything.
Ensure	Mutating repair: seed, migrate, warm a cache, or create indexes.
Force ensure	Same as Ensure, but required for dangerous checks (large-graph DDL). The build of the read performance indexes can take many minutes, so it runs in the background on the server and the page does not wait for it.

Checks that do not apply to this environment (for example PropelAuth when unused) are hidden. They are not shown as red.

Status and risk

Status	Meaning
ready	Last run reported the check as good
degraded	Partial or retryable failure
missing	Required seed, index, or cache is not present
terminal	Needs a maintenance-window Force ensure (typical for large-graph lineage DDL)
unknown	Not verified yet, or the last run raised an error

Risk	Ensure behavior
safe	Ensure runs immediately (browser; no extra confirm)
gated	Browser confirm, then Ensure. UI hint only — a Monitoring writer can POST Ensure without `force`.
dangerous	Confirm, then Force ensure. Server rejects Ensure unless `force=true`.
ops-only	Verify only — no Ensure button

Blocks and last-run audit

Each row shows:

Blocks — workers or features that stay gated while the check is missing
Blocked by — other checks that should be ready first
Last run — relative time and who triggered Verify or Ensure
History — recent audit rows (action, status, actor, duration, message)

Redis memory

Redis has a memory limit. At that limit it refuses new writes instead of dropping data, and reads keep working. The Redis cache row tells that apart from a Redis that cannot be reached:

Redis cache row	What it means
ready	Redis answers and has room
degraded (amber)	Redis uses 90% or more of its memory limit. It still accepts writes
missing (red), Redis is full	Redis is at its memory limit: it refuses writes, reads still work
missing (red), Redis connection failed	Redis cannot be reached

When Redis is full or at 90% or more, a banner at the top of the page says so, with the memory in use and the limit. With Monitoring write access the page verifies the Redis cache row when it opens and again every minute while it stays open, so the banner shows Redis as it is now; no other row is verified by itself. Each of those runs is kept in the row’s History like any other Verify. With read access the page shows the last Verify, so the banner says when the check last ran. Check again (Monitoring write access) verifies the Redis row on the spot.

Lineage indexes (Force ensure)

The Lineage indexes check is dangerous. Verify only lists whether required indexes are operational. Ensure calls the same Redis-locked path as ensure_lineage_indexes (other checks do not take a registry-wide lock) and can stall a populated FalkorDB graph for many minutes. Django DB migrations is gated: Ensure runs migrate --noinput after a browser confirm.

Run Force ensure for lineage indexes only in a maintenance window. Do not run it while ingestion, structured data, or enrichment workers are writing to the graph. Lineage CREATE INDEX is not performed automatically on Django startup.

If Verify reports terminal, the graph is over the safety ceiling and needs an explicit Force ensure after you have scheduled downtime.

Read performance indexes (Force ensure)

The Read performance indexes check covers optional indexes that speed up graph reads on FalkorDB: one on Document(file_id), and one on name for every label of the graph that has a name (lineage records have their own check above). Without them, ingestion’s document lookups and the exact-name matching of entities scan every node of a label, which gets slower as the graph grows. They are an optimization, not a prerequisite: Experio works without them. The check is hidden on Neo4j. Nothing builds these indexes for you. A deploy, a migration, a restart and a worker start all leave them alone. Someone chooses to build them, with Force ensure or with the command below.

Action	Result
Verify	Read-only, and it never reports a fault for missing indexes. The check is ready, and its message says optional performance indexes: N not built; create them in a maintenance window, N under construction, or all N built. It is unknown only when the graph cannot be inspected. Every Verify also keeps the list of the indexes it found, which View indexes shows
View indexes	A button under the row’s message, available with Monitoring read access too. Opens a read-only list of every optional index with its status. It builds and changes nothing, and it shows what the last Verify stored, so opening it puts no load on the graph. See Which indexes are built
Force ensure	Asks you to confirm in a dialog that repeats the maintenance-window rule below, then starts the build and returns at once, with Started in the background. The server builds the missing indexes one at a time, and keeps going if you close the page. The result of the run is saved in the row’s History when the run ends: ready when every index is operational, degraded when an index was still filling when its wait ran out, or when another run already holds the build, terminal when the run failed (the graph could not be read, or a build statement failed). Unlike the lineage check, terminal here means a failed run, not a graph over the safety ceiling

Build the read indexes in a maintenance window: ingestion paused and chat quiet, from the first index until the last one is operational. Creating an index, and switching it on once it has filled, each wait for every read that is running on the graph, and the whole graph server waits with them, so a long read that is already running holds the build and every other question until it finishes. The window has to cover the whole build, not only the moment the first statement is sent.

An index on a label that already has a vector index re-fills that vector index, so that label takes longer. Dropping the index again re-fills it the same way, so a rollback needs the same window.

Which indexes are built

The row’s message only counts the indexes (all 33 built). View indexes, under the message, opens a read-only list of every optional index:

Column	What it shows
Label	The node type, for example `Person`
Property	The indexed property: `file_id` for `Document`, `name` for every other label
Type	The kind of index. These are all `RANGE` indexes
Status	Built (operational), Building (it exists and is still filling) or Missing

The list opens with a summary such as 33 of 33 built (with how many are building or missing) and the time of the last result. Missing and building indexes come first, then the built ones. The filter box narrows the list by label, property, type or status, for example missing. A missing index is not a fault: Experio works without it, and the row stays ready. The list is the one the last Verify stored, or the one a Force ensure stored when it ended with every index operational, so opening it reads nothing from the graph. When the last result has no list, the dialog says why:

The row was verified before this list existed, or a build stopped before every index was operational. With Monitoring write access, Verify now reads the list from the graph (a normal, read-only Verify). Without it, ask someone who has that access to run Verify.
A build is running. The full list is not available until the build ends. The dialog shows how many indexes are under construction.
The graph has nothing to index yet. The list is empty: No label of the graph has a name to index yet. The indexes appear once the graph holds data.

While the build runs

Building the indexes on a graph with data can take many minutes, longer than a browser waits for an answer. Force ensure therefore does not wait for the build. It answers at once, and the row shows Running in the background with a message such as optional performance indexes: 1 under construction; the build continues in the background. With Monitoring write access, the page verifies the row every minute while it stays open, so the message follows the build until it is done; each of those runs is kept in the row’s History like any other Verify. While a build runs, Verify answers from the list of indexes alone and does not read the graph, so it does not slow the build down. Only one build runs at a time. Clicking Force ensure again while one runs on the same server pod starts nothing and answers Already running. The build also holds a lock in Redis while it runs, so a click that a second server pod answers cannot start another build beside it: that run is recorded in the row’s History without building anything in parallel, usually as degraded with Another process is already ensuring the read indexes of this graph. With more than one server pod, only the pod that runs the build knows that it is running, so the row shows Running in the background only when that pod answers.

If the server restarts during a build, the build stops with it, and FalkorDB finishes the index it was filling. Run Verify to see what exists. A new Force ensure may then be recorded in History as degraded, with Another process is already ensuring the read indexes of this graph, until the lock of the stopped run expires (24 hours); an Experio operator can clear it.

From the command line

The same plan and build run from the server directory:

# print the plan, create nothing
python manage.py ensure_read_indexes

# build the missing indexes (maintenance window)
python manage.py ensure_read_indexes --force

# only these labels
python manage.py ensure_read_indexes --force --labels Contract,Project

Without --force the command lists, for every label, its node count, the indexed property, whether the index is missing, building or present, and the fields the label already has indexed, then a one-line summary. It creates nothing. A label that the graph does not have is skipped and never created, because creating an index for a missing label would create the label. With --force it takes a lock, so a second run is refused with Another process is already ensuring the read indexes of this graph. It creates the missing indexes one at a time and, after each one, checks every 2 seconds for up to 120 seconds until that index is operational before it sends the next. Read the last line of the output:

Last line	Meaning
Read indexes are operational (N created now, M already there)	Done
… still not operational after 120 s … stopped there	An index is still filling, and nothing more was sent behind it. FalkorDB keeps building it, and the maintenance window stays open until it has finished. Run the command without `--force` to check, and with `--force` again only when no index is under construction. The command exits with status 0 in this case
CREATE for … failed with an uncertain server-side outcome	A statement failed and nothing more was sent. Run the command without `--force` to see what exists. The command exits with an error

On Neo4j the command only prints the equivalent statements (written to be safe to repeat) and executes nothing.

A new instance: before the first ingestion

On a graph with no nodes the plan is empty (no label to index), and Force ensure on this page creates nothing. Plan from the default ontology instead, and confirm the graph by name:

python manage.py ensure_read_indexes --empty-graph-from-ontology --confirm-graph experio --force

--confirm-graph must equal the configured graph name: FALKOR_DATABASE in the DB tab of System Settings, experio by default. Leave out --force to see the plan first. The command refuses to run when the graph already has nodes. On an empty graph the build is quick and nothing is reading, so no maintenance window is needed. Do it before the first ingestion rather than after. To cover labels the first run did not (a custom ontology type, for example), run the command again later against the graph with data, in a maintenance window.

FalkorDB engine and settings

Two read-only rows show what the graph server really runs, read from the server itself. They appear when the graph backend is FalkorDB and are hidden on Neo4j. FalkorDB engine reads INFO modules and checks that the graph module is from FalkorDB 6.x. FalkorDB settings reads GRAPH.CONFIG GET for THREAD_COUNT, QUERY_MEM_CAPACITY, MAX_QUEUED_QUERIES and CACHE_SIZE, and INFO memory for maxmemory, and compares four of them with what the deployment asked for. Neither has an Ensure button: the engine and its limits belong to the deployment, so a difference is fixed there (see FalkorDB deployment parameters), not from this page.

Row	Status	What it means
FalkorDB engine	ready	FalkorDB 6.x is running, and the message gives its version, for example FalkorDB 6.0.3 (graph module 60003)
	missing (red)	An older engine is running, for example FalkorDB 4.22.3 is running, but this release needs FalkorDB 6.x. This is a deployment error that raises no error of its own. A 4.x engine started on a graph that 6.0.3 has saved can lose every traversal without any message, so go forward: put the pinned 6.0.3 image back, and pause ingestion until it is running (see There is no rollback to FalkorDB 4.x)
	unknown	The version could not be read: FalkorDB cannot be reached (it may be scaled to zero), or its answer lists no graph module. Never red
FalkorDB settings	ready	The running `THREAD_COUNT`, `QUERY_MEM_CAPACITY`, `MAX_QUEUED_QUERIES`, `CACHE_SIZE` and `maxmemory` (with its eviction policy) are in the message, and none differs from what the deployment asked for
	degraded	A running value differs from its deployment parameter, and the message names each one, for example THREAD_COUNT asked 4, running 2. The FalkorDB node may not have restarted since the parameters changed, or was changed by hand. Not red. The limit parameters reach FalkorDB when its pod starts and are applied at the next upgrade of the instance, so upgrade the instance and Verify again; if a value still differs, the node was changed by hand, so set it back (see FalkorDB deployment parameters)
	unknown	A value could not be read, or FalkorDB cannot be reached. A read-only FalkorDB user may not run `GRAPH.CONFIG`, so on such a connection the first four values are unknown and `maxmemory` is still shown

The values asked for are the deployment parameters FalkorDB Query Threads, FalkorDB Query Memory Cap, FalkorDB Max Queued Queries and FalkorDB Max Memory, which the deployment also gives to the server. Where the server does not have them (local development, an older deployment), the row shows the running values only and says that the values asked for are not known. A value that is not a whole number is named in the message and not compared. CACHE_SIZE has no deployment parameter, so it is shown and never compared.

Orphaned agent flow runs

An Agent Flow run executes inside the API process. If that process stops mid-run — a deploy, a pod restart, a dev-server reload — nothing is left to finish the run, and the run keeps whatever status it had: the conversation goes on looking busy for ever. The API reconciles those runs once on every start. Orphaned agent flow runs is the operator lever for the same repair: Verify reports how many stale runs exist right now, and Ensure (gated — a browser confirm, no maintenance window) marks them failed without restarting the API. What it takes and what it leaves alone:

A run parked at a Human review gate is never touched. It is not orphaned — it is waiting for a reviewer, is still listed in the Agent Inbox, and resumes the moment someone answers.
A run waiting for the parser is never touched either. Its documents are still being read by the background parser. It is not in the Agent Inbox, and nobody answers it: a sweep of the flow runner resumes it when the documents are read or its wait runs out, and fails it with a clear reason if its resume keeps dying.
A running run whose executor stopped renewing its lease is reaped after a few minutes.
A pending run that outlived the process that would have started it is reaped on the same clock. Agent flows have no queue, so nothing else will ever move it.
A running run that never recorded a lease at all — written by an older release, or by a replica still being rolled — is left alone until it has been untouched for hours, not minutes, so a long-running node is never cut off mid-work.

Reaping a run you are unsure about is not destructive. If the executor of a run this check reaped turns out to have been alive after all, that run can still settle its real result when it finishes, and the files it produced still reach the conversation.

Platform health endpoints

Three endpoints report on the platform itself. They are separate from this page.

Endpoint	Who may call it	What it does
`GET /health/`	Anyone	Checks PostgreSQL, Redis and RabbitMQ. These are critical: the endpoint answers HTTP 503 when one is unhealthy. A Redis that is full (at its memory limit, so it refuses writes) is unhealthy; at 90% or more of its limit `checks.redis` is a `warning`, which never makes the endpoint fail. `checks.redis` also gives `state` (`ok`, `near_cap`, `full` or `unreachable`) and, when Redis reports them, `used_bytes`, `max_bytes` and `percent`. The active graph (under `checks.neo4j`), MCP and the Startup Health summary (`checks.init`) are reported but never make it fail: an unavailable graph makes the status `degraded` (HTTP 200). `checks.llm` is reported as `skipped`
`GET /health/simple/`	Anyone	Answers only that the server process is running. It does not touch the database, Redis or the model provider, so an outage there never takes the server out of service. The server’s readiness check on Omnistrate is configured to call it
`GET /health/llm/`	Staff users only	Runs one real completion against the reasoning model and answers HTTP 200 `healthy` or 503 `unhealthy`. Anonymous callers and non-staff users are refused

checks.init is a last-run summary of Startup Health (ready / missing / terminal counts). It does not re-run checks and it never returns HTTP 503 by itself. Load balancers should keep using the critical set (PostgreSQL, Redis, RabbitMQ). GET /health/ used to call the LLM on every request. The endpoint is anonymous and each call cost a model call and held a request thread for as long as the provider took to answer, so the probe moved to GET /health/llm/. A monitor that alerted on checks.llm must call that path as a staff user instead. This page also has an LLM reasoning model health check (Verify only) that runs the same probe on demand. On Omnistrate the server’s readiness check is configured to call GET /health/simple/. Where the probe cannot run curl, it falls back to checking that port 8000 is listening. Liveness stays the platform’s default TCP check on port 8000, so a server that is briefly saturated is not restarted by a probe.

Deploy notes

Apply the startup_health Django migration (InitCheckRun) before relying on last-run history.
After a new environment or graph rebuild, open this page and Verify all.
Repair missing seeds with Ensure on the specific check rather than expanding migrate-init into a long blocking job.
Large Falkor graphs: Force ensure lineage only in a maintenance window, then Verify again before starting ingestion_v2, structured_data, and enrichment.
Optional read indexes on FalkorDB: on a new instance, build them before the first ingestion (see A new instance). On a graph that has data, build them only in a maintenance window. The build runs in the background, so keep the window open until the row says all N built.
After an upgrade of the graph node, or a change to a FalkorDB limit parameter, Verify FalkorDB engine and FalkorDB settings (see FalkorDB engine and settings).

Jobs & Monitoring — job list, logs, and scan-order stats
Service Scaling — Omnistrate replica counts
Graph Lineage — what lineage records are used for
Graph backend — FalkorDB limits, read indexes and deployment parameters

Getting Started

Data Sources

Knowledge Graph

User Management

Integrations

Settings

Monitoring

Overview

Verify vs Ensure

Status and risk

Blocks and last-run audit

Redis memory

Lineage indexes (Force ensure)

Read performance indexes (Force ensure)

Which indexes are built

While the build runs

From the command line

A new instance: before the first ingestion

FalkorDB engine and settings

Orphaned agent flow runs

Platform health endpoints

Deploy notes

​Overview

​Verify vs Ensure

​Status and risk

​Blocks and last-run audit

​Redis memory

​Lineage indexes (Force ensure)

​Read performance indexes (Force ensure)

​Which indexes are built

​While the build runs

​From the command line

​A new instance: before the first ingestion

​FalkorDB engine and settings

​Orphaned agent flow runs

​Platform health endpoints

​Deploy notes

​Related

Overview

Verify vs Ensure

Status and risk

Blocks and last-run audit

Redis memory

Lineage indexes (Force ensure)

Read performance indexes (Force ensure)

Which indexes are built

While the build runs

From the command line

A new instance: before the first ingestion

FalkorDB engine and settings

Orphaned agent flow runs

Platform health endpoints

Deploy notes

Related