> ## Documentation Index
> Fetch the complete documentation index at: https://docs.experio.cloud/llms.txt
> Use this file to discover all available pages before exploring further.

# Graph Evaluation

> Manually audit graph extraction quality for completed document ingestion jobs

## Overview

**Graph evaluation** runs an LLM-as-judge over ingested documents to assess whether extracted
graph facts match the source text and content-type schema. It is **observational only** — it does
not modify the graph or re-trigger ingestion.

Evaluation uses **Graph Lineage** records plus live graph node state (including matched existing
nodes) as input. See [Graph Lineage](/admin-guide/graph-lineage) for how ingestion writes are
recorded.

Open **Admin → Process → Graph Evaluation** for the runs list, filters, and run detail pages. You
can also start a run from a completed job via **Process → Jobs** → job detail → **Run Evaluation**.

## When you can run evaluation

The **Run Evaluation** button is available when:

| Requirement                | Details                                                    |
| -------------------------- | ---------------------------------------------------------- |
| **Terminal job status**    | Job is completed, failed, stopped, or interrupted          |
| **Unstructured ingestion** | Structured/API datasources use the statistics view instead |
| **Full ingestion mode**    | Parse-only jobs have no graph output to evaluate           |
| **Ingested files**         | At least one file reached `ingested` status                |

Only one evaluation run may be **pending** or **running** per job at a time.

## Running an evaluation

### From the Graph Evaluation page

1. Go to **Process → Graph Evaluation**
2. Click **Run Evaluation** and pick a completed unstructured job
3. Set **samples per content type** (default from `GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE`, max 50)
4. Confirm — the run is queued to the graph evaluation worker
5. Open the run detail page to watch progress and review results

### From a job detail page

1. Open **Process → Jobs** and select a completed unstructured job
2. In the **Graph Evaluation** card, click **Run Evaluation**
3. Confirm samples per content type — you are redirected to the run detail page when the run starts

Use **View Evaluations** on the job card to filter the runs list to that job.

While a run is active, the UI polls every few seconds until it completes or fails.

### Automatic runs after job completion

When **`GRAPH_EVALUATION_AUTO_RUN_AFTER_JOB`** is enabled in **Admin → System Settings**, Experio
queues an evaluation automatically once **every file** in the job has reached a terminal processing
status (including `ingested`, `failed`, `skipped`, and other finalized states). The run uses the
configured default sample count (`GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE`).

Auto-run uses the same eligibility rules as manual runs (unstructured, full ingestion, at least one
ingested file, terminal job status, no other pending/running evaluation). Parse-only jobs and jobs
with no ingested files are skipped silently.

## Reading results

### Run summary

Each run shows:

* **Status** — pending, running, completed, failed, or cancelled
* **Sample count** and **issue count** aggregated across samples
* **Summary line** — evaluated / skipped / failed counts

### Runs list

The **Graph Evaluation** page supports search, status filters, data source filters, and pagination.
Filter by job using the job link or `scanOrderId` query parameter from a job detail page.

### Samples list

On a run detail page, the left column lists sampled files with:

* **Status** — evaluated, skipped (no lineage), or failed
* **Issue count** for evaluated samples
* **Overall quality** — good, acceptable, or poor

Click a sample to open its details. The panel auto-selects the sample with the most issues.

### Sample details and issues

The right column shows:

* File name, overall quality badge, and content type
* Judge summary for the document
* **Issues (N)** — each issue includes category, severity, description, and document evidence
* **Lineage links** when the issue references a graph element with lineage records

### Issue categories

| Category                                            | Meaning                                                 |
| --------------------------------------------------- | ------------------------------------------------------- |
| `missed_entity` / `missed_relationship`             | Supported by the document and schema, absent from graph |
| `hallucinated_entity` / `hallucinated_relationship` | Present in graph but not supported by document text     |
| `incorrect_attribute`                               | Entity exists but a property contradicts the document   |
| `incorrect_relationship_endpoint`                   | Relationship type plausible but wrong source/target     |
| `schema_violation`                                  | Label or relationship not defined in the content type   |

## Skipped samples (no lineage)

Files ingested **before Graph Lineage** was enabled may have no lineage records. Those samples
are marked **skipped (no lineage)** with guidance to re-ingest the file before evaluating.

## Matched existing nodes

When ingestion links a document to a **pre-existing graph node** (entity resolution / matching),
lineage may only record a new relationship — not a full node create. Evaluation enriches the
judge input with **live graph properties** for relationship endpoints so matched entities are not
falsely flagged as missing.

## API

Authenticated admin REST endpoints (prefix `/api/graph-evaluation/`):

| Endpoint                    | Purpose                                                        |
| --------------------------- | -------------------------------------------------------------- |
| `POST /preview/`            | Estimate sample counts before starting a run                   |
| `POST /runs/`               | Create and queue an evaluation run                             |
| `GET /runs/`                | Paginated runs list (search, status, data source, job filters) |
| `GET /runs/{id}/`           | Run detail and aggregates                                      |
| `POST /runs/{id}/cancel/`   | Cancel a pending or running evaluation                         |
| `GET /runs/{id}/samples/`   | Sample list for a run                                          |
| `GET /samples/{id}/`        | Sample detail with issues                                      |
| `GET /samples/{id}/issues/` | Issues only                                                    |

## Worker and infrastructure

The **graph-evaluation** job service consumes `graph.evaluation.queue` via RabbitMQ. It uses the
**Ingestion-Large** model configuration for judge calls (with JSON-mode response parsing).

Tune worker concurrency, shutdown, sampling defaults, and auto-run in **Admin → System Settings**
(JOBS category): `GRAPH_EVALUATION_THREADS` (default `2`),
`GRAPH_EVALUATION_SHUTDOWN_TIMEOUT_SECONDS` (default `600`),
`GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE` (default `3`), and
`GRAPH_EVALUATION_AUTO_RUN_AFTER_JOB` (default `False`). See
[System Settings](/admin-guide/system-settings) for details.

In production (Omnistrate), deploy the `graph-evaluation` service built from
`Dockerfile.graph_evaluation`. For local development:

```bash theme={null}
cd jobs && docker compose up -d graph-evaluation
```

Apply Django migrations if needed:

```bash theme={null}
cd server && pipenv run python manage.py migrate graph_evaluation
```

## Related documentation

* [Jobs & Monitoring](/admin-guide/jobs-monitoring) — job list and detail pages
* [Graph Lineage](/admin-guide/graph-lineage) — provenance records used as judge input
* [Content Types](/admin-guide/content-types) — schemas compared during evaluation
* Architecture plan: `docs/plans/graph-evaluation-architecture.md`