Skip to main content

​
Overview

Graph evaluation runs an LLM-as-judge over ingested documents to assess whether extracted graph facts match the source text and content-type schema. It is observational only — it does not modify the graph or re-trigger ingestion. Evaluation uses Graph Lineage records plus live graph node state (including matched existing nodes) as input. See Graph Lineage for how ingestion writes are recorded. Open Admin → Process → Graph Evaluation for the runs list, filters, and run detail pages. You can also start a run from a completed job via Process → Jobs → job detail → Run Evaluation.

​
When you can run evaluation

The Run Evaluation button is available when: Only one evaluation run may be pending or running per job at a time.

​
Running an evaluation

​
From the Graph Evaluation page

  1. Go to Process → Graph Evaluation
  2. Click Run Evaluation and pick a completed unstructured job
  3. Set samples per content type (default from GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE, max 50)
  4. Confirm — the run is queued to the graph evaluation worker
  5. Open the run detail page to watch progress and review results

​
From a job detail page

  1. Open Process → Jobs and select a completed unstructured job
  2. In the Graph Evaluation card, click Run Evaluation
  3. Confirm samples per content type — you are redirected to the run detail page when the run starts
Use View Evaluations on the job card to filter the runs list to that job. While a run is active, the UI polls every few seconds until it completes or fails.

​
Automatic runs after job completion

When GRAPH_EVALUATION_AUTO_RUN_AFTER_JOB is enabled in Admin → System Settings, Experio queues an evaluation automatically once every file in the job has reached a terminal processing status (including ingested, failed, skipped, and other finalized states). The run uses the configured default sample count (GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE). Auto-run uses the same eligibility rules as manual runs (unstructured, full ingestion, at least one ingested file, terminal job status, no other pending/running evaluation). Parse-only jobs and jobs with no ingested files are skipped silently.

​
Reading results

​
Run summary

Each run shows:
  • Status — pending, running, completed, failed, or cancelled
  • Sample count and issue count aggregated across samples
  • Summary line — evaluated / skipped / failed counts

​
Runs list

The Graph Evaluation page supports search, status filters, data source filters, and pagination. Filter by job using the job link or scanOrderId query parameter from a job detail page.

​
Samples list

On a run detail page, the left column lists sampled files with:
  • Status — evaluated, skipped (no lineage), or failed
  • Issue count for evaluated samples
  • Overall quality — good, acceptable, or poor
Click a sample to open its details. The panel auto-selects the sample with the most issues.

​
Sample details and issues

The right column shows:
  • File name, overall quality badge, and content type
  • Judge summary for the document
  • Issues (N) — each issue includes category, severity, description, and document evidence
  • Lineage links when the issue references a graph element with lineage records

​
Issue categories

​
Skipped samples (no lineage)

Files ingested before Graph Lineage was enabled may have no lineage records. Those samples are marked skipped (no lineage) with guidance to re-ingest the file before evaluating.

​
Matched existing nodes

When ingestion links a document to a pre-existing graph node (entity resolution / matching), lineage may only record a new relationship — not a full node create. Evaluation enriches the judge input with live graph properties for relationship endpoints so matched entities are not falsely flagged as missing.

​
API

Authenticated admin REST endpoints (prefix /api/graph-evaluation/):

​
Worker and infrastructure

The graph-evaluation job service consumes graph.evaluation.queue via RabbitMQ. It uses the Ingestion-Large model configuration for judge calls (with JSON-mode response parsing). Tune worker concurrency, shutdown, sampling defaults, and auto-run in Admin → System Settings (JOBS category): GRAPH_EVALUATION_THREADS (default 2), GRAPH_EVALUATION_SHUTDOWN_TIMEOUT_SECONDS (default 600), GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE (default 3), and GRAPH_EVALUATION_AUTO_RUN_AFTER_JOB (default False). See System Settings for details. In production (Omnistrate), deploy the graph-evaluation service built from Dockerfile.graph_evaluation. For local development:
Apply Django migrations if needed:
  • Jobs & Monitoring — job list and detail pages
  • Graph Lineage — provenance records used as judge input
  • Content Types — schemas compared during evaluation
  • Architecture plan: docs/plans/graph-evaluation-architecture.md