Skip to main content

​
Overview

Graph evaluation runs an LLM-as-judge over ingested documents to assess whether extracted graph facts match the source text and content-type schema. For large samples, an optional decision model triage screens every document first and sends only the flagged ones to the LLM judge. It is observational only — it does not modify the graph or re-trigger ingestion. Repairs belong on Graph Health. Evaluation uses Graph Lineage records plus live graph node state (including matched existing nodes) as input. See Graph Lineage for how ingestion writes are recorded. Open Admin → Process → Graph Evaluation for the runs list, filters, and run detail pages. You can also start a run from a completed job via Process → Jobs → job detail → Run Evaluation.

​
When you can run evaluation

The Run Evaluation button is available when: Only one evaluation run may be pending or running per job at a time.

​
Running an evaluation

​
From the Graph Evaluation page

  1. Go to Process → Graph Evaluation
  2. Click Run Evaluation and pick a completed unstructured job
  3. Choose the judge — LLM judge or Decision model triage (see below)
  4. Set samples per content type — the default and maximum depend on the judge: LLM judge uses GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE (max 50), triage uses GRAPH_EVALUATION_TRIAGE_SAMPLES_PER_CONTENT_TYPE (max 500)
  5. Confirm — the run is queued to the graph evaluation worker
  6. Open the run detail page to watch progress and review results

​
From a job detail page

  1. Open Process → Jobs and select a completed unstructured job
  2. In the Graph Evaluation card, click Run Evaluation
  3. Confirm samples per content type — you are redirected to the run detail page when the run starts
Use View Evaluations on the job card to filter the runs list to that job. While a run is active, the UI polls every few seconds until it completes or fails.

​
Automatic runs after job completion

When GRAPH_EVALUATION_AUTO_RUN_AFTER_JOB is enabled in Admin → System Settings, Experio queues an evaluation automatically once every file in the job has reached a terminal processing status (including ingested, failed, skipped, and other finalized states). The run uses the default judge (GRAPH_EVALUATION_JUDGE_MODE) and that judge’s default sample count. Auto-run uses the same eligibility rules as manual runs (unstructured, full ingestion, at least one ingested file, terminal job status, no other pending/running evaluation). Parse-only jobs and jobs with no ingested files are skipped silently.

​
Decision model triage

Triage lets a run cover far more documents for little extra cost. Every sampled document is first scored by Microsoft-Decision-1, a decision model on Microsoft Foundry that returns probabilities instead of text: an overall quality rating (good / acceptable / poor) and, for each issue category, the probability that the problem is present. A document goes on to the LLM judge, which writes the detailed issues, when any of these is true: All three settings are fractions between 0 and 1; values outside that range are clamped. If the decision model is down, the run still finishes with the LLM judge. After 3 failed calls in a row the run stops calling the decision model, and the LLM fallback is limited to the LLM judge’s own maximum of 50 documents per content type, so an outage never costs more than an ordinary LLM-judge run. Documents past that limit are marked failed with the reason, and the run summary shows how many decision-model errors there were, how many the LLM judged instead, and how many were not judged. Documents that pass every check are recorded with the decision model’s rating and no issue list — the decision model does not write descriptions or evidence. Setup: deploy Microsoft-Decision-1 in your Foundry resource, then add it in Admin → Model Configurations with type Decision and provider Microsoft Foundry Decision (see Model Configurations), test it, and set it as the default decision model (DECISION_MODEL_CONFIG). Until a default is set, the triage option is disabled in the run dialog. Set GRAPH_EVALUATION_JUDGE_MODE to triage to make it the default for new and automatic runs. Tuning: the run summary compares the two models on documents both judged, in two groups:
  • Random audit — documents the decision model found clean. Issues the LLM finds here are the ones triage would have missed, so this is the number to watch before sending fewer documents.
  • Flagged — documents sent on for a reason. If the LLM rarely finds issues here, triage is sending more than it needs to.
The two thresholds move in opposite directions. To send fewer documents to the LLM, lower GRAPH_EVALUATION_TRIAGE_QUALITY_CONFIDENCE (a document escalates when confidence is below it) or raise GRAPH_EVALUATION_TRIAGE_ISSUE_PROBABILITY (a document escalates when a category is at or above it). Do the reverse to send more. Each sample stores the thresholds it was screened with, so older runs stay readable after a change.

​
Reading results

​
Run summary

Each run shows:
  • Status — pending, running, completed, failed, or cancelled
  • Sample count and issue count aggregated across samples
  • Summary line — evaluated / skipped / failed counts
  • Judge — LLM judge or Decision model triage
  • For triage runs: how many documents the decision model screened, accepted on its own, and sent to the LLM judge (with reasons), any decision-model errors and how they were handled, how often the two models agreed (audit and flagged documents separately), and the mean decision-model probability per issue category across every screened document. That mean is the decision model’s own signal, not a measured issue rate

​
Runs list

The Graph Evaluation page supports search, status filters, data source filters, and pagination. Filter by job using the job link or scanOrderId query parameter from a job detail page.

​
Samples list

On a run detail page, the left column lists sampled files with:
  • Status — evaluated, skipped (no lineage), or failed
  • Issue count for evaluated samples
  • Overall quality — good, acceptable, or poor
  • Decision model badge when the decision model judged the sample without the LLM
Click a sample to open its details. The panel auto-selects the sample with the most issues.

​
Sample details and issues

The right column shows:
  • File name, overall quality badge, and content type
  • Judge summary for the document
  • Decision model screening (triage runs) — the decision model’s rating and confidence, the probability for each issue category, and why the sample was or wasn’t sent to the LLM judge
  • Issues (N) — each issue includes category, severity, description, and document evidence
  • Lineage links when the issue references a graph element with lineage records

​
Issue categories

​
Skipped samples (no lineage)

Files ingested before Graph Lineage was enabled may have no lineage records. Those samples are marked skipped (no lineage) with guidance to re-ingest the file before evaluating.

​
Matched existing nodes

When ingestion links a document to a pre-existing graph node (entity resolution / matching), lineage may only record a new relationship — not a full node create. Evaluation enriches the judge input with live graph properties for relationship endpoints so matched entities are not falsely flagged as missing.

​
API

Authenticated admin REST endpoints (prefix /api/graph-evaluation/): POST /preview/ and POST /runs/ accept scan_order_id, optional samples_per_content_type, and optional judge_mode (llm or triage; defaults to GRAPH_EVALUATION_JUDGE_MODE). The preview response also returns the mode’s default and maximum sample counts and whether triage is available.

​
Worker and infrastructure

The graph-evaluation job service consumes graph.evaluation.queue via RabbitMQ. It uses the Ingestion-Large model configuration for judge calls (with JSON-mode response parsing). Tune worker concurrency, shutdown, sampling defaults, and auto-run in Admin → System Settings (JOBS category): GRAPH_EVALUATION_THREADS (default 2), GRAPH_EVALUATION_SHUTDOWN_TIMEOUT_SECONDS (default 600), GRAPH_EVALUATION_SAMPLES_PER_CONTENT_TYPE (default 3), and GRAPH_EVALUATION_AUTO_RUN_AFTER_JOB (default False). See System Settings for details. In production (Omnistrate), deploy the graph-evaluation service built from Dockerfile.graph_evaluation. For local development:
Apply Django migrations if needed:
  • Jobs & Monitoring — job list and detail pages
  • Graph Lineage — provenance records used as judge input
  • Content Types — schemas compared during evaluation
  • Architecture plan: docs/plans/graph-evaluation-architecture.md