Create the golden question list as a Markdown file named
questions.md in the engagement’s shared project folder, starting from the Golden question list template below. It is the one artifact every phase reads: modelling, validation, and acceptance. A plain file is also easy to hand to the knowledge-model copilot when it is available in your release.Kickoff and discovery
Kickoff agenda
Kickoff agenda
Use for the kickoff meeting in Phase 1 (90 minutes).
# Kickoff: <Client name> x Experio
Date: <date> Location / link: <link>
| Time | Topic | Lead |
|---|---|---|
| 0:00–0:10 | Introductions, roles (sponsor, super user, SMEs, IT, FFE, PM) | PM |
| 0:10–0:25 | Why we are doing this: business goals, success measures | Sponsor |
| 0:25–0:40 | How Experio works: data to graph to answers (short demo) | FFE |
| 0:40–0:55 | Roadmap, workshop calendar, what we need from you each week | PM |
| 0:55–1:10 | Candidate use cases and their SMEs | Sponsor, super user |
| 1:10–1:20 | Access and environment: SSO, data sources, IT contacts | FFE, IT |
| 1:20–1:30 | Homework for Workshop 1 and 2; next steps | PM |
## Decisions
- Use cases in scope: <UC1, UC2, ...>
- Success measure: <e.g., 80% of golden questions pass>
## Homework
| Owner | Task | Due |
|---|---|---|
| <SME> | Bring 10–15 real questions for your use case | <date> |
| <IT admin> | Confirm connector access to <systems> | <date> |
Use-case one-pager (one per use case, Workshop 1)
Use-case one-pager (one per use case, Workshop 1)
# Use case <UC#>: <name>
SME: <name, role> Users: <who, how many>
## Problem today
<2–3 sentences: what takes too long, what is missed, what it costs>
## Who asks, and when
| Persona | Situation | How often |
|---|---|---|
| <BD Director> | <writing a qualification section> | <weekly> |
## What a good answer looks like
<one paragraph, or paste a real example>
## Where the truth lives
| Data | System | Owner |
|---|---|---|
| <projects> | <Deltek export> | <Finance/PMO> |
## Outputs
- [ ] Chat answers - [ ] Document (DOCX/PPTX/XLSX) - [ ] Multi-step agent flow
## Success measure
<e.g., "Qualification section drafted in 30 minutes instead of 1 day">
## Golden questions
See questions.md, IDs <Q1–Q3>.
## Agent candidates
See agent candidate card(s): <name>
Golden question list (questions.md)
Golden question list (questions.md)
One row per question. Type is one of Write the expected answer before the first ingestion. If nobody can write it, the question is not ready.
lookup, list, aggregate, relationship, generative. Priority is must, should, nice. Status is draft, agreed, passing, failing, retired.# Golden questions: <Client name>
Owner: <super user> Last updated: <date>
| ID | Use case | Persona | Question | Expected answer | Source of truth | Type | Priority | Status |
|---|---|---|---|---|---|---|---|---|
| Q1 | UC1 | BD Director | Which cloud migration projects did we deliver for healthcare clients since 2023, and who led each? | 4 projects: NB-2023-044 (lead: ...), ... | Deltek projects.csv + SOWs | list | must | agreed |
| Q2 | UC1 | BD Director | Draft a past-performance summary for our work with Lakeshore Health. | Covers the 3 Lakeshore projects, each cited | SOWs, case studies | generative | must | agreed |
| Q7 | UC3 | Contracts Counsel | Which SOWs expire in the next 90 days, and for which clients? | List as of test date, checked against the contract register | SOWs | list | must | draft |
Agent candidate card (Workshop 1)
Agent candidate card (Workshop 1)
# Agent candidate: <name>
Use case: <UC#> Owner (reviewer): <name>
| Box | Answer |
|---|---|
| Trigger | <chat request / Launch a flow menu / upload> |
| Inputs (required / optional) | <role*, skills*, client> |
| Graph retrieval | <which golden question the first step answers> |
| LLM steps | <rank / classify / extract / write> |
| Human review | <who, what they check, what they may edit> |
| Output | <chat answer / DOCX / PPTX / filled XLSX> |
| Document template | <name, if any> |
| Frequency and volume | <e.g., 10 per week> |
| Why not just chat? | <fixed steps, review, file output> |
| Success measure | <e.g., shortlist accepted without rework 4 times in 5> |
Data and modelling
Data inventory (Workshop 2)
Data inventory (Workshop 2)
| # | Source | System | Structured / unstructured | Format | Owner | Key field(s) | Volume | Refresh | Access method | Sensitive? | Use cases |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Employees | Workday | structured | employees.csv | HR | employee_id, work_email | 650 rows | weekly | export to SharePoint | yes (HR) | UC2 |
| 2 | Engagements | SharePoint "Engagements" | unstructured | PDF, DOCX | BD / Legal | client folder name | ~4,000 files | continuous | connector | contracts | UC1, UC3 |
Key crosswalk (Workshops 2 and 7)
Key crosswalk (Workshops 2 and 7)
Shows how the same thing is identified in each source, so keys can be made identical before loading.
| Entity | Canonical key | Source A (field, example) | Source B (field, example) | Documents (how it appears) | Normalisation needed | Owner |
|---|---|---|---|---|---|---|
| Client | Salesforce account name | accounts.csv: name "Lakeshore Health" | projects.csv: client "Lakeshore Health, Inc." | folder "Lakeshore Health"; text "LSH" | strip suffixes; synonym LSH | Sales ops |
| Project | project_code | projects.csv: NB-2024-117 | assignments.csv: NB-2024-117 | status report title "NB-2024-117 ..." | none | PMO |
| Employee | work email | employees.csv: rchen@... | assignments.csv: rchen@... | resume: "Bob Chen" | name variants: human review | HR |
Ontology worksheet (Workshop 3)
Ontology worksheet (Workshop 3)
One block per entity type, then one table for relationships.
## Entity type: <Project>
Description: <one sentence the extraction model will read>
Key (match property): <project_code>
Vector index: <yes / no> (yes if documents must match it by meaning or name variants)
| Attribute | Type (text/number/date/boolean/list/enum) | Extraction instructions | Format hint | Required? |
|---|---|---|---|---|
| name | text | Official project name as in the SOW title | | yes |
| project_code | text | Code like NB-2024-117 | NB-YYYY-NNN | yes |
| start_date | date | Contract start, not kickoff | YYYY-MM-DD | no |
| service_line | text | Choose from @ServiceLine | | no |
## Relationships
| Relationship | From | To | Attributes (name: type) | Meaning / when to create |
|---|---|---|---|---|
| FOR_CLIENT | Project | Client | | The client that pays for the project |
| WORKS_ON_PROJECT | Employee | Project | role: text, start_date: date, end_date: date, hours: number | Person staffed on the project |
Taxonomy CSV (Workshop 4)
Taxonomy CSV (Workshop 4)
Import on the taxonomy type’s page in Model & Define > Taxonomies. The import merges with existing items. Add synonyms in the UI after import. Use either format.
- Path format
- Level columns format
path,description,status
Healthcare,,active
Healthcare > Providers,Hospitals and health systems,active
Healthcare > Payers,,active
Healthcare > Life Sciences,,active
Financial Services > Banking,,active
Financial Services > Insurance,,active
Public Sector > Federal,,active
level1,level2,level3,description,status
Cloud,,,,active
Cloud,Azure,,,active
Cloud,Azure,Azure Data Platform,Synapse Fabric and Azure SQL,active
Cloud,AWS,,,active
Taxonomy: <Industry> Source system: <Salesforce Industry picklist>
Owner of updates: <Sales ops> Refresh: <quarterly re-import>
Used in: <Client.industry extraction instructions (@Industry); "Tag project industry" rule>
Artifact type spec (Workshop 5)
Artifact type spec (Workshop 5)
# Artifact type: <Statement of Work>
Classification instructions: <what makes a document this type, and what it is NOT>
Folder filters: <"Engagements" library>
Sample files for Test Ingestion: <3–5 file names, including one hard case>
## Extraction policy
Mode: <full / metadata_and_snippet / metadata_only> Model tier: <large / medium / small>
Validation pass: <on / off> Excel override: <n/a>
## Entities
| Entity type | Processing type (Create Only / Create or Match / Match Only) | Attach Document? | Attribute overrides (extraction instructions) |
|---|---|---|---|
| Contract | Create or Match | yes | contract_type: always "SOW" |
| Client | Create or Match | no | |
| Project | Match Only | no | project_code: look in title and header |
## Relationships
| Relationship | From | To | Required upstream? |
|---|---|---|---|
| WITH_CLIENT | Contract | Client | yes (Client first) |
| GOVERNS | Contract | Project | yes |
## Test Ingestion results
| File | Top score (type) | Entities OK? | Relationships OK? | Notes |
|---|---|---|---|---|
Mapping worksheet (Workshop 6)
Mapping worksheet (Workshop 6)
# Data mapping: <projects.csv>
Data source: <name> Ontology: <name>
Record ID column: <project_code> Timestamp column (incremental sync): <last_modified>
Load order position: <3 of 5, after employees and accounts>
## Node mappings
| Column | Entity type | Attribute | Operation (create / match / create-or-match) | Match property |
|---|---|---|---|---|
| project_code | Project | project_code | create-or-match | project_code |
| project_name | Project | name | | |
| client_name | Client | name | match | name |
## Relationship mappings
| Relationship | Start entity (column → property) | End entity (column → property) | Relationship attributes |
|---|---|---|---|
| FOR_CLIENT | Project (project_code → project_code) | Client (client_name → name) | |
Check: every relationship end is created by a node mapping in this or an earlier load.
Structured matching is exact-value only; confirm key values are formatted identically.
Matching decision card (Workshop 7)
Matching decision card (Workshop 7)
# Matching: <Client>
Default strategy or override: <override>
Match key: <name>
Candidate methods: [ ] Exact [ ] Vector Similarity [ ] Fuzzy [ ] Phonetic [ ] Synonym
Normalisation: [ ] strip suffixes (Inc, LLC) [ ] prefixes [ ] abbreviations [ ] punctuation
Filters: <temporal / relationship, e.g., same Practice>
Thresholds: auto-merge ≥ <0.9> AI disambiguation <0.7–0.9> human review <0.5–0.7> new < <0.5>
Synonyms to add: <LSH = Lakeshore Health>
Vector index needed on this type: <yes / no>
Human reviewer in Conflict Resolution: <name> Expected weekly volume: <n>
Test pairs
| Value A | Value B | Should match? | Result |
|---|---|---|---|
| Lakeshore Health, Inc. | Lakeshore Health | yes | |
| Bob Chen | Robert Chen | yes (review) | |
Enrichment rule card (Workshop 8)
Enrichment rule card (Workshop 8)
# Enrichment rule: <Tag project industry>
Why: <which golden question it serves, e.g., Q1>
Target entity type: <Project> Scope: <all / only where industry is empty>
Input: [ ] full node [ ] selected attributes: <description> [ ] neighbourhood: <{related.Client.industry}>
Prompt: <instruction text; use @TaxonomyName for controlled values>
Output: [ ] attribute <industry> [ ] node [ ] relationship <type> [ ] node + relationship
Taxonomy: <@Industry>
Run: [ ] on demand from Model & Define > Enrichment Rules [ ] as a step in Process > Flows <flow name>
Test on: <10 sample nodes> Accepted if: <9 of 10 correct, checked by SME>
Owner: <name>
Access-control policy worksheet (Workshop 9)
Access-control policy worksheet (Workshop 9)
# Access control: <Client name>
Data owner sign-off: <name, date> Sponsor sign-off: <name, date>
## Baselines (list EVERY label in the graph; no row = private)
| Label | private / public_read | Reason |
|---|---|---|
| Contract | private | Confidentiality terms |
| Project | public_read | Needed firm-wide |
## Grant rules (team)
| Relationship (exact name) | Person → | Who it covers |
|---|---|---|
| WORKS_ON_PROJECT | Employee → Project | Staffed consultants |
## Propagation (hold the left, may also read the right)
| If you hold | over | direction (inbound / outbound) | you may also read |
|---|---|---|---|
| Project | GOVERNS | inbound | Contract |
## Containment (roll-up axes)
| Axis name | Relationship | Direction (down / up) | Attached to baseline |
|---|---|---|---|
## Management roll-up
Use it? <yes / no> Relationship: <REPORTS_TO> Direction: up Attached to no baseline
## People who need access without being staffed
| Group | How they are connected (relationship, data source, owner) |
|---|---|
## Test personas (at least 3)
| User | Level | Question IDs | Expected result | Preview as user result | Pass? |
|---|---|---|---|---|---|
## Rollout
Shadow from: <date> Enforce from: <date> Switch approved by: <name>
Running the project
Decision log
Decision log
| ID | Date | Workshop | Decision | Why | Alternatives considered | Decided by | Affects (config) | Status |
|---|---|---|---|---|---|---|---|---|
| D-012 | 2026-05-04 | WS7 | Documents never create Projects (Match Only) | Finance's project list is the truth | Create or Match | Marcus Lee, Sam Whitfield | SOW, Status Report artifact types | done |
proposed, agreed, done, or reversed. Never delete a row; reverse it with a new one.Golden-question scoring sheet
Golden-question scoring sheet
Score in chat, one row per question per run. Record the persona if access control is on. Root cause is one of
classification, extraction, matching, mapping, retrieval, missing data.# Scoring run <n>: <date> Assistant: <name> Scored by: <name>
| ID | Persona / user | Answer (summary or link) | Citations correct? (yes / partly / no / none) | Result (pass / partial / fail) | Root cause | Evidence (lineage, Graph Evaluation, Test Ingestion) | Fix owner | Fix |
|---|---|---|---|---|---|---|---|---|
| Q1 | BD Director | Listed 3 of 4 projects | yes | partial | matching | NB-2023-044 attached to "Lakeshore Health Inc" duplicate | FFE | add suffix normalisation, merge duplicate |
| Q7 | Contracts Counsel | Correct list | yes | pass | | | | |
Summary: pass <n> / partial <n> / fail <n> Must-have pass rate: <%>
Tuning log (pilot and validation)
Tuning log (pilot and validation)
One row per configuration change made during tuning. Recording what was reprocessed means you know which later results reflect the fix.
| Date | Symptom (question, review pattern, Graph Evaluation issue) | Root cause | Change made (page > setting) | Reprocessed (files / step, or mapping rerun) | Result | By |
|---|---|---|---|---|---|---|
| 2026-05-19 | 31 classification reviews, SOW amendments at 0.6–0.7 | classification | Artifact Types > SOW > classification instructions: "An amendment that changes an existing SOW is a Statement of Work" | Requeued affected files from Classify | Reviews dropped to 4 | FFE |
| 2026-05-20 | Q9 fail; `incorrect_attribute` on `liability_cap` | extraction | Ontology > Contract > `liability_cap` instruction: "convert amounts written in words to a number in USD" | Requeued MSAs from Ingest | Graph Evaluation: MSA poor → good | FFE, Marcus Lee |
Handover pack
Handover pack
# <Client> Experio handover pack <date>
## 1. Configuration record
- Ontology: current revision number, entity-type and relationship list, and the rationale (which golden questions each type serves)
- Taxonomies: source system, owner and refresh cadence for each
- Artifact types: spec per type (classification rules, entities, processing types, extraction policy)
- Data mappings: source file or API, owner, load order, match properties
- Matching strategies and review thresholds; who works Conflict Resolution
- Enrichment rules: purpose, inputs, outputs, when they run
- Access-control policy worksheet and mode (if used)
- Assistants, AI Instructions, document templates, agent flows
## 2. Decision log
- Full log, with open and reversed items
## 3. Accuracy
- questions.md with expected answers
- Scoring sheets for every round and the final pass rate
- Tuning log
- Known limitations accepted by the sponsor
## 4. Runbook
- Scheduled flows and what to do when one fails
- Adding a new data source or library
- Changing the ontology safely (save, check Compatibility, review stale items, reprocess)
- Refreshing a taxonomy from its source system
- Onboarding a user and assigning permissions
## 5. Operating cadence and contacts
- Weekly, monthly and quarterly tasks, with owners
- Experio support contacts and escalation path
Go-live readiness checklist
Go-live readiness checklist
## Data
- [ ] Full ingestion complete; Jobs show no stuck or failed runs
- [ ] Conflict Resolution queue empty or owned
- [ ] Ontology Compatibility shows no invalid dependents
- [ ] Graph Evaluation run on every full-mode artifact type; no "poor" results unresolved
- [ ] Enrichment rules run after the final load; scheduled Flows in place for refresh
## Accuracy
- [ ] Final scoring run: must-have golden questions meet the agreed pass rate
- [ ] Every failure has a root cause and an accepted workaround or fix date
## Access and security
- [ ] SSO configured and tested; user groups and permissions assigned
- [ ] Access control (if used) in enforce mode, persona tests passed, sign-off recorded
- [ ] Disclosure text agreed
## Consumption
- [ ] Assistants enabled; AI Instructions reviewed
- [ ] Document templates generate correct branded files
- [ ] Agent flows published, Sub-agent descriptions written, reviewers named
- [ ] MCP and Integrations instructions sent to users who need them
## People
- [ ] Pilot users trained; training for wider rollout scheduled
- [ ] Super user can change ontology, artifact types, mappings, assistants without help
- [ ] Hypercare contacts, hours and escalation path published
- [ ] Handover pack delivered (decision log, questions.md, worksheets)
Sign-off: Sponsor <name, date> Super user <name, date> FFE <name, date>
Weekly status report
Weekly status report
# <Client> x Experio: week <n> status <date>
Overall: <green / amber / red> Phase: <e.g., 2 Model>
## Done this week
- <WS3 Ontology agreed; 8 entity types, 13 relationships>
## Next week
- <WS4 Taxonomies (Tue), WS5 Artifact Types (Thu)>
## Metrics
| Measure | Last week | This week |
|---|---|---|
| Golden questions agreed | | |
| Golden questions passing | | |
| Files ingested / in review | | |
| Open Conflict Resolution items | | |
## Risks and issues
| Item | Impact | Owner | Due | Status |
|---|---|---|---|---|
## Decisions needed from the client
- <who, what, by when>
## Homework status
| Owner | Task | Due | Status |
|---|---|---|---|
Related
Running workshops
How to prepare, facilitate and follow up each workshop.
Worked example
The full Northbridge Consulting model these templates are filled from.