Skip to main content
This workshop decides what “working” means. You leave with the real questions a group of people needs answered, the answer each one should get, and the jobs they want an agent to do for them. Every later decision is tested against this list: an entity type exists because a question needs it, and the implementation is accepted when the must-have questions pass. Without it, the model gets built around whatever the documents happen to contain, and accuracy cannot be measured. The list of questions is called the golden questions. It is kept in a Markdown file called questions.md that you create for each engagement in the shared project folder, starting from the golden question list template. It is a project document, not something stored in Experio.
Run this workshop once per use case, with the SMEs for that use case. Northbridge Consulting runs it three times: UC1 Proposals & past performance (Dana Ortiz), UC2 Staffing & expertise (Sam Whitfield), UC3 Contract obligations (Helen Park). The outputs are consolidated into one questions.md afterwards.

​
At a glance

​
Scoping a use case

A use case is one group of people doing one kind of job. It is narrow enough that the same SMEs can answer every question in it. Test a proposed use case against these rules:
  • One owner. One SME can say whether an answer is right. “Proposals” has Dana; “everything BD does” does not.
  • A real job. People spend measurable time on it today (hours of searching, emails to colleagues, hand-built reports).
  • Answerable from sources you can get. If the truth only lives in someone’s head, it is not a pilot use case.
  • 10 to 25 questions. Fewer means the use case is too thin; more means it should be split.
If the kickoff produced a vague use case (“knowledge management”), narrow it at the start of the session by asking: “Whose week gets better, and at which task?”

​
Before the workshop

FFE and Experio SMEs
  • Read the kickoff notes and the use-case statement.
  • Prepare 5–10 seed questions for the use case. The internal demo question bank (docs/demo-question-bank.md in the Experio repository) has persona-based examples for BD, compliance, account handover, onboarding and expert finding.
  • Prepare the golden-question table (below) in a shared document the room can see.
  • For the second and later use cases, bring the consolidated questions.md so far. Overlaps are easier to spot live.
Client homework (send three working days before) Ask each attendee to bring:
  1. Five questions they asked or were asked last week, word for word.
  2. Two emails or chat messages where they asked a colleague for information.
  3. One report, spreadsheet or document they build by hand on a regular basis.
  4. For each, where they found the answer (system, folder, person).

​
Agenda

​
Running the workshop

​
1. Personas and jobs-to-be-done

A persona here is simply a type of user. Name each one and write their jobs as “When I…, I need to…, so that…”. Ask the room:
  • “Who does this work today? What is their title?”
  • “What triggers the work? An RFP arriving, a new project, a contract renewal?”
  • “What does done look like? What do you hand to whom?”
  • “Where does the time go?“

​
2. Harvest real questions

Work from the homework, not from imagination. Imagined questions are too general (“tell me about our healthcare work”); real ones have names, dates and filters. Ask the room:
  • “Read me a question you asked last week.”
  • “Show me the email you sent. What were you actually trying to find out?”
  • “This report you build by hand: what question does each column answer?”
  • “What question do you wish you could ask but never bother to, because it takes too long?”
  • “What follow-up do you ask after you get the first answer?”
Write every question down. Do not filter yet.

​
3. Make each question golden

For every question, fill in the columns of the table below. The expected answer is the most important column: without it, nobody can score the question later. Ask the room:
  • “What exactly is the right answer today? Name the projects, people or dates.”
  • “Where did you get that answer? Which system, which folder, which document?”
  • “If the answer is a list, how many items should it have?”
  • “Is this a must-have for launch, or a nice-to-have?”
Classify each question by type. Types fail in different ways, so a balanced list tests the whole pipeline. Set priority: Must (launch is blocked if it fails), Should (expected at launch), Could (later).

​
4. Identify agent candidates

An agent candidate is a repeatable, multi-step job that produces an output: a document, a shortlist, a digest. It becomes an agent flow in WS10. A single question is not an agent candidate; “answer six questions, rank the results, and write them into our template” is. Ask the room:
  • “Which of these questions do you always ask together, in the same order?”
  • “What do you produce at the end? Can you share a recent example?”
  • “Is there a house template or format it has to follow?”
  • “Who checks it before it goes out?”
  • “How often do you do this?”
For each candidate, record: name, trigger, inputs, steps (as questions), output format, template, human review step, frequency.

​
5. Agree success criteria

Agree measurable criteria the super user will sign off at the end of validation. Keep them few and concrete, for example:
  • At least 80% of must-have questions pass by week 7; no must-have question gives a confidently wrong answer.
  • At least 60% of should-have questions pass by week 7.
  • Every generative answer cites at least one source document the SME agrees is correct.
  • The “Past-performance write-up” agent produces a draft Dana would edit rather than rewrite.
Scoring is manual today: someone runs each question in chat and records pass / partial / fail. See Accuracy Validation.

​
6. Capture the firm’s names

While people talk, note every way they refer to their own firm. Northbridge people say “Northbridge”, “NBC” and “Northbridge Consulting Group”. These become the client configuration synonyms, which stop Experio from extracting the firm as an external company in its own documents.

​
Golden question template

​
Consolidating across use cases

After the last WS1 session, the FFE and super user merge all sessions into one questions.md.
1

Merge

Put every question in one table with its use case tag. Keep the original wording.
2

De-duplicate

Two questions are duplicates if they need the same data and the same filter, even if worded differently. Keep the clearest wording and list both use cases. “Who worked with Meridian Bank?” (UC2) and “Which of our people know Meridian Bank?” (UC1) are one question.
3

Keep variants that test different things

“Who led the Lakeshore projects?” and “Who worked on the Lakeshore projects?” look alike but test different relationship attributes. Keep both.
4

Balance

Check that each use case has lookups, lists, relationship questions and at least one generative question. Add questions where a type is missing.
5

Sign off

The super user and each use-case SME confirm the list and priorities. Freeze it as version 1. Later changes go through the decision log.

​
Worked example: Northbridge Consulting

​
UC1 Proposals & past performance (Dana Ortiz)

Persona: BD Director. Job: “When an RFP arrives, I need relevant past projects and qualification text, so that I can submit a credible proposal in days, not weeks.” Agent candidate: Past-performance write-up. Trigger: new RFP. Inputs: client or industry, service line, date range. Steps: Q1-style search, pick top 3 projects, pull scope and outcomes, write in house style. Output: DOCX from the “Past Performance” template. Review: Dana approves before it goes into a proposal. Frequency: 6–10 per month.

​
From UC2 Staffing & expertise (Sam Whitfield)

Agent candidate: Staffing shortlist. Inputs: role, skills, client. Output: ranked shortlist with reasons. Review: Sam.

​
From UC3 Contract obligations (Helen Park)

Agent candidate: Contract renewal digest. Monthly list of expiring SOWs and upcoming obligations, reviewed by Helen.

​
How questions drive the ontology

The golden questions are the raw material for Workshop 3: Ontology. Read them for four kinds of word: If a question needs a noun, verb or filter that no source contains, flag it now. It either needs a new source (WS2) or an enrichment rule (WS8).

​
Accelerate with AI

In development — availability depends on your release. The knowledge-model copilot takes questions.md as an input, together with sample documents and tables, and proposes ontology and artifact-type changes as cards that a person reviews and applies. A clean, de-duplicated questions.md with expected answers gives it much better proposals. The manual path in WS3 always works without it.
You can also use a general AI assistant to tidy the harvest: paste the raw questions and ask it to group near-duplicates and suggest a type for each. A person still decides.

​
Entering it in Experio

Most of this workshop’s output lives in questions.md, stored in the shared project folder. Only one thing is entered now:
  1. Go to Administer > Client Configuration.
  2. Enter the firm’s canonical name and the synonyms heard in the session.
  3. Save. See Client Configuration.
Agent candidates are built later in AI & Agents > Agent Flows (WS10).

​
Common pitfalls

  • Invented questions. “What do we know about healthcare?” cannot be scored. Insist on real questions with names and filters.
  • No expected answer. A question without an expected answer cannot pass or fail. Assign an owner to find it before sign-off.
  • Only generative questions. They are impressive but hide errors. Balance them with lookups and lists that have exact answers.
  • Truth in someone’s head. If the SME says “I just know”, the question cannot pass until the knowledge is written down somewhere Experio can read.
  • One giant use case. Forty questions from five teams means no single SME can validate them. Split it.
  • Skipping the second and third sessions. Running WS1 once for all use cases produces a shallow list dominated by whoever talks most.

​
Exit criteria

  • Personas and jobs-to-be-done written for this use case
  • 10–25 golden questions, each with expected answer, source, type and priority
  • At least one question of each type relevant to the use case
  • Agent candidates recorded with inputs, output, template and reviewer
  • Success criteria agreed and signed off by the super user
  • Firm name and synonyms entered in Client Configuration
  • After the last use case: consolidated, de-duplicated questions.md version 1 signed off

​
Next

Continue to Workshop 2: Data Inventory.