Skip to main content

Overview

Data sources define which folders and files Experio should scan and process from your connected cloud storage providers. Each data source is linked to a connector and specifies folder paths, scanning behavior, and filtering rules. Navigate to Admin > Data Sources > Data Sources.

Creating a Data Source

Click Add New Data Source to start a multi-step configuration wizard:
1

Choose Source Type

Select the type of data source:
  • Box — Scan folders from a Box account
  • Google Drive — Scan folders from Google Drive
  • SharePoint — Scan folders from a SharePoint site
  • File Upload — Upload files directly to Experio
2

Validate Configuration

Enter the connection details and validate that Experio can access the specified location. The system verifies credentials and folder access.
3

Configure Filters

Set up folder hierarchy and filtering rules:
  • Folder paths — Specify which folders to scan
  • Recursive scanning — Include subfolders
  • Filter expressions — Include or exclude files based on patterns
  • Ingest Excel files — On by default. Uncheck for filters that should exclude spreadsheets from graph ingestion (see Extraction Policy)
  • Excel extraction mode override — Optional per-filter Excel policy override
  • Excel max sheet characters — Optional per-sheet character cap override
4

Setup Source

Configure ingestion settings for the data source:
  • Days to sync — How far back to scan for files
  • Use OCR — Force optical character recognition on every file in the source. With it off, a PDF whose pages are pictures and whose text layer is small (fewer than 60 letters and digits a page on average, as in a signed scan with a signature line or an e-signature stamp on each page) is still read again with OCR in the parser. The OCR text is kept only when it holds more letters and digits than the text layer; otherwise the file keeps its text layer
  • Classification max pages — Limit pages sent to the classifier
  • Ingestion type — Choose Full ingestion (default) for the complete pipeline, or Parse only to stop after parsing (useful when a downstream system handles classification and embedding)
For API data sources, these options appear in the source configuration step instead.
5

Test Filters

Preview which files match your filter configuration before saving. This ensures only the intended files will be processed.

Data Source Properties

Managing Data Sources

Editing

Click on any data source to open its configuration. Modify settings and save to apply changes. Changes take effect on the next scan cycle.

Monitoring

Each data source shows:
  • Last scan time — When the source was last scanned
  • Files found — Number of files discovered
  • Files processed — Number of files successfully ingested
  • Errors — Any files that failed processing
  • Compatibility — Whether linked content types and mappings still match the ontology
If you start a job while a linked config is invalid, the request fails immediately with links to the blocking items. Fix those configs and re-validate, then start the job again. Stale configs warn only and do not block the scan.

OAuth Callbacks

For Box and SharePoint data sources, OAuth callback handling is built in. If a token expires, you’ll be prompted to re-authorize through the connector.

Parse-Only Mode

When a data source has Ingestion Type set to Parse only, the ingestion pipeline stops after downloading and parsing files. Specifically:
  • Files are downloaded from the cloud provider and parsed using the standard parser
  • The parsed text is stored as parsed/{file_id}.txt. The clean-up job deletes the parsed text of a fully ingested file once it is more than MINIO_RETENTION_DAYS days old (7 by default), wherever that job runs, but it keeps this one: in this mode it is the final result.
  • No classification, graph ingestion, or embedding occurs
  • Files reach a terminal status of parsed_only instead of ingested
This mode is useful when an external system (such as a partner pipeline) needs to consume the parsed output and handle classification and embedding independently.
Ingestion Type can only be changed when the data source has no files currently processing. If you try to switch modes while a scan is in flight, the update is rejected with a validation error. Wait for the current scan to complete (or stop it) before changing the mode. The new mode takes effect on the next scan.

File Upload

The File Upload data source type allows direct file uploads:
  • Drag and drop files onto the upload area
  • Track upload progress with visual indicators
  • Files are queued for processing automatically after upload