> ## Documentation Index
> Fetch the complete documentation index at: https://docs.experio.cloud/llms.txt
> Use this file to discover all available pages before exploring further.

# Startup Health

> Verify and repair app-level migrations, seeds, indexes, and cache readiness

## Overview

Navigate to **Admin > Monitoring > Startup Health** to see whether this environment finished
its Django, graph, seed, and index bootstrap work.

This page is **app-level readiness**, not replica scaling. Use [Service Scaling](/admin-guide/service-scaling)
to change how many pipeline workers are running.

Reading the page requires Monitoring **read** access. Verify, Ensure, and Verify all require
Monitoring **write** access.

## Verify vs Ensure

| Action           | What it does                                                                |
| ---------------- | --------------------------------------------------------------------------- |
| **Verify**       | Read-only. Asks “is this check good right now?” and stores an audit row.    |
| **Verify all**   | Runs Verify for every applicable check. Does not create or repair anything. |
| **Ensure**       | Mutating repair: seed, migrate, warm a cache, or create indexes.            |
| **Force ensure** | Same as Ensure, but required for **dangerous** checks (large-graph DDL).    |

Checks that do not apply to this environment (for example PropelAuth when unused) are hidden.
They are not shown as red.

## Status and risk

| Status       | Meaning                                                                       |
| ------------ | ----------------------------------------------------------------------------- |
| **ready**    | Last run reported the check as good                                           |
| **degraded** | Partial or retryable failure                                                  |
| **missing**  | Required seed, index, or cache is not present                                 |
| **terminal** | Needs a maintenance-window Force ensure (typical for large-graph lineage DDL) |
| **unknown**  | Not verified yet, or the last run raised an error                             |

| Risk          | Ensure behavior                                                                                       |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| **safe**      | Ensure runs immediately (browser; no extra confirm)                                                   |
| **gated**     | Browser confirm, then Ensure. **UI hint only** — a Monitoring writer can POST Ensure without `force`. |
| **dangerous** | Confirm, then Force ensure. Server rejects Ensure unless `force=true`.                                |
| **ops-only**  | Verify only — no Ensure button                                                                        |

## Blocks and last-run audit

Each row shows:

* **Blocks** — workers or features that stay gated while the check is missing
* **Blocked by** — other checks that should be ready first
* **Last run** — relative time and who triggered Verify or Ensure
* **History** — recent audit rows (action, status, actor, duration, message)

## Lineage indexes (Force ensure)

The **Lineage indexes** check is **dangerous**. Verify only lists whether required indexes are
operational. Ensure calls the same **Redis-locked** path as `ensure_lineage_indexes` (other checks
do not take a registry-wide lock) and can stall a populated FalkorDB graph for many minutes.

**Django DB migrations** is **gated**: Ensure runs `migrate --noinput` after a browser confirm.

<Warning>
  Run **Force ensure** for lineage indexes only in a maintenance window. Do not run it while
  ingestion, structured data, or enrichment workers are writing to the graph. Lineage `CREATE INDEX`
  is not performed automatically on Django startup.
</Warning>

If Verify reports **terminal**, the graph is over the safety ceiling and needs an explicit Force
ensure after you have scheduled downtime.

## Orphaned agent flow runs

An [Agent Flow](/admin-guide/agent-flows) run executes inside the API process. If that process stops
mid-run — a deploy, a pod restart, a dev-server reload — nothing is left to finish the run, and the
run keeps whatever status it had: the conversation goes on looking busy for ever.

The API reconciles those runs once on every start. **Orphaned agent flow runs** is the operator lever
for the same repair: Verify reports how many stale runs exist right now, and Ensure (**gated** — a
browser confirm, no maintenance window) marks them failed without restarting the API.

What it takes and what it leaves alone:

* **A run parked at a Human review gate is never touched.** It is not orphaned — it is waiting for a
  reviewer, is still listed in the Agent Inbox, and resumes the moment someone answers.
* A **running** run whose executor stopped renewing its lease is reaped after a few minutes.
* A **pending** run that outlived the process that would have started it is reaped on the same clock.
  Agent flows have no queue, so nothing else will ever move it.
* A **running** run that never recorded a lease at all — written by an older release, or by a replica
  still being rolled — is left alone until it has been untouched for **hours**, not minutes, so a
  long-running node is never cut off mid-work.

<Note>
  Reaping a run you are unsure about is not destructive. If the executor of a run **this check reaped**
  turns out to have been alive after all, that run can still settle its real result when it finishes, and
  the files it produced still reach the conversation.
</Note>

## Platform health endpoint

`GET /health/` includes an informational `checks.init` object: a last-run **summary** of Startup
Health (ready / missing / terminal counts). It does **not** re-run checks and it never returns
HTTP 503 by itself. Load balancers should keep using the existing critical set (PostgreSQL,
Redis, RabbitMQ, LLM).

## Deploy notes

* Apply the `startup_health` Django migration (`InitCheckRun`) before relying on last-run history.
* After a new environment or graph rebuild, open this page and **Verify all**.
* Repair missing seeds with **Ensure** on the specific check rather than expanding `migrate-init`
  into a long blocking job.
* Large Falkor graphs: Force ensure lineage only in a maintenance window, then Verify again
  before starting ingestion\_v2, structured\_data, and enrichment.

## Related

* [Jobs & Monitoring](/admin-guide/jobs-monitoring) — job list, logs, and scan-order stats
* [Service Scaling](/admin-guide/service-scaling) — Omnistrate replica counts
* [Graph Lineage](/admin-guide/graph-lineage) — what lineage records are used for