Overview
Some files cannot be read where they are uploaded. A 23 MB spreadsheet can need 2 GB of memory, and the OCR of a scanned PDF can take minutes and gigabytes. The API server does not do that work. It hands those files to the background parser (the async parser), a program in the parser service, next to the parser workers of the ingestion pipeline. It reads one file at a time per parser replica, in a child process with a memory limit and a time limit.- No queue service. A waiting file is a row in the database, and the background parser claims rows from there. Nothing goes through RabbitMQ, and the ingestion pipeline is unchanged.
- Results are stored in the database. For each file the parser keeps the whole text and its chunks (and, for the knowledge-model copilot, the vectors its search uses).
- People see their own files. The chips where people attach files show their own files, and each ready file has an eye that opens its stored text (Previewing the stored text). Administrators see every job on the Jobs page, but only the person who asked for a job can open the text of its files.
Where it is used
When a file goes to the background parser
By default, every file attached in the copilot or the main chat. The Parse in the background switch, in the paperclip menu of both (one switch for each conversation or session), is on until the person turns it off, and while it is on every file goes to the background parser whatever its size (the one exception is a plain.txt or .md file in the main chat, which is read at once). A new conversation starts with the switch on again.
With the switch off, and in Test Ingestion, the API server reads a file first, in a child process with no OCR, and hands it over only when that is not enough:
- Too big on sight. A file over
ASYNC_PARSE_PRECHECK_MAX_FILE_MB(1 MB by default), or a spreadsheet overASYNC_PARSE_PRECHECK_MAX_SPREADSHEET_MB(0 by default, which sends every spreadsheet, whatever its size). It is not read on the server at all. A limit of0sends every file, or every spreadsheet (but never a Test Ingestion picture, below). Raise the spreadsheet limit to have small spreadsheets read on the server. - A reading that stops. The reading reaches
UPLOAD_EXTRACTION_MAX_MEMORY_MB(512 MB by default) orUPLOAD_EXTRACTION_TIMEOUT_SECONDS(60 seconds), or the server’s container or machine has less than 768 MB of memory left. - A scan. A PDF that gives fewer than 60 letters and digits a page, on average, when it is read without OCR. Its pages are pictures, with at most a signature line or a stamp as text.
.txt, .md) in the main chat is decoded in place whatever the switch says. A zip that expands too far is refused instead of handed over, because the background parser would refuse it too. In Test Ingestion, a picture is read by OCR in place, whatever its size: the background parser’s first reading uses no OCR and would find no words in it. Nothing else can read a picture, so it keeps a memory budget and a time limit of its own, never below 1,024 MB and 300 seconds, and one that needs more fails with this file is too large to read here instead of being handed over.
These limits are low on purpose, so that the background parser does the heavy reading. In tests on a fast laptop at the earlier, higher limits, ordinary files under them needed about 140 to 365 MB and finished within about 25 seconds. That is not a maximum: a file can pass the size check and still need more, for example a spreadsheet with a very large sparse table (about 660 MB in one test) or a Word file with one enormous table (about 800 MB). Such a file is stopped at the limit and read by the background parser instead, so nothing fails. A server slower than the test machine also stops some text-heavy PDFs at the 60-second limit and hands them over. The price is waiting: a file over 1 MB, and every spreadsheet, waits for the background parser, which takes several minutes to start when it has been scaled to zero (see What happens to a file that waits). On an API server with memory to spare, raise the two pre-check limits together with UPLOAD_EXTRACTION_MAX_MEMORY_MB and UPLOAD_EXTRACTION_TIMEOUT_SECONDS; a spreadsheet needs far more memory than its size suggests, so keep its limit the lowest.
OCR choices
A scanned PDF is read by OCR only inside the background parser. Each surface chooses how it asks:- The estimate on the button is the page count times
ASYNC_PARSE_OCR_SECONDS_PER_PAGE(10 seconds by default). It limits nothing. - A question nobody answers is skipped after
ASYNC_PARSE_NEEDS_OCR_EXPIRY_HOURS(24 hours by default): the file settles as a scan nobody read. - The OCR text is kept only when it holds more letters and digits than the first reading. Otherwise the file keeps its first reading, and a scan is flagged as not read.
What happens to a file that waits
- Queue. A file starts as Queued. A parser takes one file at a time, and an idle parser looks for work every few seconds.
- Turns. The next file is taken from the person who was served longest ago, and someone who was never served goes first, so one person’s ten big files do not hold everyone else’s past the queue wait. Among one person’s files, first readings go before OCR readings, older files before newer ones, and an OCR reading that has waited 10 minutes goes first.
- Waking the parser. The parser can be scaled to zero. The
async_parsewarm-keeping policy wakes it while a file waits and keeps it up during a reading. A wake-up takes several minutes (see The parser and async document parsing). Because Parse in the background is on by default, every file attached in the copilot or the main chat (plain text in the main chat apart) can wait for a wake-up, a tiny one included. With the switch off, so can a file over 1 MB or any spreadsheet, and so can a Test Ingestion sample of that kind. - Queue wait. A file that stays Queued (or queued for OCR) longer than
ASYNC_PARSE_QUEUE_WAIT_SECONDS(60 minutes by default) fails with Parsing is unavailable right now; try again later. A file from an Agent Flow waits for its step’s deadline instead: 30 minutes by default, 2 hours at most. - A reading that dies. The parser records a heartbeat every 30 seconds while it reads (
ASYNC_PARSE_HEARTBEAT_SECONDS). When a file has had none for 120 seconds (ASYNC_PARSE_STALE_HEARTBEAT_SECONDS), another parser takes it over. AfterASYNC_PARSE_MAX_ATTEMPTS(3) readings that died under it, the file fails with Reading this file failed after 3 attempts: the parser stopped each time before it finished. A parser that shuts down cleanly puts its file back without counting an attempt, and so does a reading that stops for lack of server memory, up to three times for a file. - Limit per person. A person can have
ASYNC_PARSE_MAX_QUEUED_PER_USERfiles (10 by default) waiting or being read at once, across every surface. A scan waiting for its owner’s OCR choice counts, for up to 24 hours. A small main-chat file that the API server reads itself is not held to it. With Parse in the background on, which is the default, the API server reads none (plain text apart), so every file counts, and the 11th of 11 files dropped at once is refused until one of them settles. Turn the switch off to attach small files (spreadsheets apart) past the limit. Past the limit, a new file is refused with N of your files are already waiting to be read (at most 10 at a time). - Cancel. Removing a chat file or a copilot document, closing a copilot session, cancelling a flow run or a test, and Cancel on a waiting screen all cancel the file. A file that waits is never read, and a reading in progress stops within about half a minute.
- Retention. A finished file is deleted 7 days later (
ASYNC_PARSE_RETENTION_DAYS), unless a chat file or a copilot document still uses it. The uploaded original is deleted as soon as the file has been read, and an upload nothing points at is removed within about an hour. - Delivery. When a file settles, its surface is told once: the copilot announces it, the main chat shows a notice, and a flow run resumes. A sweep that the flow runner (the job that also runs scheduled flows) runs about once a minute handles queue waits, expired OCR questions, delivery to flows and clean-up.
Picking and previewing parsed documents
A parsed result can be used again, and read in the browser, without reading the file again. Picking a document attaches a copy of its stored text; a preview only reads it.The picker
Attach from parsed documents (in the copilot and the main chat) and the document choice of the Parse documents (async) step open the same picker, with up to three lists. A list holds only the documents that can be attached:
Each tab says this in one line under the tabs, and again as its tooltip.
Narrowing a list. Every list has a search box and a File type filter. Parse only and Ingested have more:
- Data source: one or more data sources, by name.
- Job: one run of a data source: a job is one scan of its files, with the number it has on the Jobs page (Job #113 there, Job 113 in the picker). An option also gives the day the run started, and names its data source while none or several are ticked: Job 113 · Team Box · Oct 9, 2026. Once data sources are ticked, the Job list offers only their jobs. Every row of these two lists shows its job (Job 113) beside its data source.
- Artifact type (Ingested only): the artifact type the classifier gave the file, for example Contract. It sits beside File type (the extension). A parse-only file is never classified, so Parse only has none.
- Clear filters appears while any of the three is set. It clears them, not the search or the file type. Changing the tab clears them too, and a changed filter starts again at page 1.
- How it is decided. Where the stored file decides, the server looks up the file of each candidate once, newest first, before it cuts the page; no text is read, so a page of the list stays quick. The graph’s copy of an ingested document’s text can be read unless graph access control is enforcing (
GRAPH_AUTHORIZATION_MODEset toenforce): then an ingested document with no parsed text file is not listed, except for a super-admin whileGRAPH_AUTHZ_SUPERUSER_BYPASSis on. Where the graph’s copy can stand in, an ingested document is listed without a lookup. See Graph Access Control and System Settings. - A long list. Where the stored file decides (a Parse only list, and an Ingested list while graph access control is enforcing), the server checks the newest 2,000 candidates; an Ingested list also covers only the 5,000 newest documents the graph is asked about. When a list is cut at either limit, the count ends in a plus (2,000+ documents, or the number found among the newest ones) and a tooltip says so. The search and every filter (data source, job, artifact type and file type) narrow the documents before the list is cut at those limits, so any of them reaches an older document.
- The options are capped too. Each filter lists at most 100 entries: 100 data sources, the 100 newest jobs and, on Ingested, 100 artifact types. The Parse only options come from every parse-only document you may see. The Ingested options come only from the 5,000 newest documents the graph is asked about, and only those it lets you read, so a very old data source or job may not be offered there. A note under the filters then says Only the newest data sources and jobs are listed. Search by name to find an older document. Searching by name still finds the document, because a search narrows the documents before that cut.
- Who sees which data sources. Data Source read access (Data Source Managers read) is not set per data source: a person who has it can list the Parse only documents of every data source, and the filters offer all of them. There is no setting that limits a person to some data sources. Ingested is narrower: each document must also pass the knowledge graph’s access check, so a person sees only the ingested documents their own graph access lets them read, and the filters offer only the data sources, jobs and artifact types of those documents.
- While the list loads, and when it fails. Changing the list, the search, a filter, the file type or the page clears the rows at once and shows a loading state until the new rows arrive; the rows of another query are never shown. If the list cannot be read it shows one sentence and Try again, with no count and no pager. The Ingested list asks the knowledge graph which documents the person may read, so while the graph cannot be reached it says We couldn’t check which documents you may read right now. Try again in a moment. (the server answers 503 with the code
scope_unavailable); My uploads and Parse only do not use the graph. - A first check, not a promise. A listed document can still be refused when it is attached (its stored text is empty, or the person’s access ended since the list was drawn). Attach tries every ticked document and lists every refusal at once, in one box, instead of stopping at the first. The documents that went in stay attached. A ticked document that a list of one page no longer shows when it is read again leaves the selection.
- The way back. A document that is missing is back once it is re-ingested, or the file is uploaded again.
I can’t find my document in the picker
Go through these in order; the first that fits is usually the answer. In short: My uploads holds only your own files, and three uploads are not kept there (a.txt or .md file in the main chat, a small copilot file that the API server read itself, and a Test Ingestion sample read the same way), so attach such a file again instead of picking it; a Parse only data source never fills Ingested; a file still being processed, or one that failed, is not listed; Ingested empty although the file was ingested: tick Attach document to this entity on the document’s entity, Update Entity, Save, then Requeue > Ingest Phase; the data source tabs need Data Source Managers read access; text gone: re-ingest or upload again; long list: search by name or use the filters; graph down: try again.
- You uploaded it in a chat or in the copilot. It is under My uploads, not under a data source. My uploads lists only your own files, and only once they have been read. Three kinds of upload are not kept there, so attach such a file again instead of picking it:
- a
.txtor.mdfile in the main chat, which is always read in place; - a small file in the copilot (up to 1 MB, not a spreadsheet), with Parse in the background turned off, that the API server read itself;
- a Test Ingestion sample that the API server could read itself (a file up to 1 MB that is not a spreadsheet, or a picture, which is always read there; see Test Ingestion).
- a
- It comes from a data source that only parses. A file of a data source whose Ingestion type is Parse only is under Parse only, and never under Ingested: it was read and its text was saved, but it was not added to the knowledge graph. A data source on Full ingestion puts its files under Ingested instead. A file that changed after its data source was switched to Full ingestion is read again as a new entry next to the old one, so it can be in both.
- The file is still being processed, or it failed. A file that is waiting, being downloaded, parsed, classified or ingested is not listed yet. A file whose ingestion failed, or that was skipped or stopped, is in neither tab (an earlier scan of the same file that succeeded is still listed). Check the file’s status on the Jobs page.
- The Ingested tab is empty although the file was ingested. Its artifact type has no entity that attaches the document, so the knowledge graph holds no document for the file: it is not under Ingested, and not under Parse only either. Open Artifact Types, open the artifact type, go to Entities & Relationships, edit the entity that stands for the document itself, tick Attach document to this entity, press Update Entity, then Save the artifact type. Then ingest the file again: select it in the job’s file list, click Requeue and choose the Ingest Phase. The Troubleshooting bullet The Ingested list is empty after an ingestion has the details.
- You are not allowed to see it. Parse only and Ingested are for people with Data Source Managers read access; without it the picker shows only My uploads. Ingested also lists only the documents your own graph access lets you read (see Graph Access Control). Ask an administrator.
- The stored text is gone. A document whose parsed text is no longer in storage is not listed. An ingested document stays listed while the knowledge graph keeps a copy of its text, unless graph access control is enforcing: then it needs its parsed text file, which the pipeline’s clean-up deletes
MINIO_RETENTION_DAYSdays after it was written (7 by default, in System Settings). So under enforcement an ingested document leaves the Ingested list about a week (by default) after its file was parsed, except for a super-admin whileGRAPH_AUTHZ_SUPERUSER_BYPASSis on. A Parse only file keeps its parsed text. Re-ingest the file, or upload it again. A file you uploaded yourself is kept for 7 days after it was read (by default), unless a conversation or a copilot session still uses it. - The list is very long. The picker checks only the newest documents: 2,000 for Parse only, and 5,000 for Ingested (2,000 while graph access control is enforcing). The count then ends in a plus, for example 2,000+ documents. Search by name, or use the Data source, Job, Artifact type (Ingested only) and File type filters, to reach an older document. The filters list at most 100 entries each, and on Ingested a very old data source or job may be missing from them: search by name then.
- The knowledge graph cannot be reached. The Ingested tab says We couldn’t check which documents you may read right now. Try again in a moment. Try again once the graph is up (see Graph Backend). My uploads and Parse only do not need the graph.
Previewing the stored text
The eye icon (tooltip Preview parsed text) opens the text the parser stored for a file, in a window (a chip in the main chat opens it in the right panel) that shows it a slice at a time: the first 20,000 characters, then 100,000 for each Load more. What the window offers (header, Page n dividers, search, Copy) is described in Previewing the text of a file. The database cuts the slice of a parsed file’s text, so even a text of tens of millions of characters is never loaded whole; a Parse only or Ingested document in the picker comes from its stored file (an ingested one with no stored file, from the graph’s copy of its text), which is read in full for every slice, as attaching it is. A preview reads and does nothing else: it creates, copies and caches nothing, the answer is marked so the browser does not keep it, and the server logs only ids and character counts, never the text. The eye is where the person already works with the file, and each place has its own rule for who may open it:
When a preview cannot show text, it says why:
- You can no longer open this document (it was removed, or you lost access to its source). The server answers a file the person may not read exactly as it answers one that does not exist, so the message cannot tell which. It covers a file or session that was removed or closed, a finished file that nothing used and that passed its 7-day retention, an ended access from the table above and, for a copy that picking a data-source document made, a source the person can no longer read: such a copy stays readable only while its source does. In the main chat it also covers an access check that could not run just then (the knowledge graph was unreachable): the main chat treats a check it cannot run as a refusal, so the message cannot tell the two apart.
- This document’s text is no longer stored; re-ingest it or upload the file. The picker’s eye on a document whose parsed text went after the list was drawn: the picker lists no document that already lacks it.
- The parsed text of this file is no longer stored. A picked file in the main chat whose stored parse was deleted; the message keeps no other copy.
- This file is still being read. Its text can be opened once that is done. or This scanned file is waiting for a choice about reading it with OCR. The file has no text yet. The failure’s own error appears when the file failed (Reading this file was cancelled. when it was cancelled).
- This scan was not read (no OCR was run). The main chat’s answer for a scan that nobody read with OCR. The copilot, the picker and the Jobs page show the little text the first pass found instead, with Scan not read in amber in the window’s header.
- Could not check your access right now. Try again. The answer of the picker’s eye and of the Jobs page when the knowledge graph, which checks access to an ingested document (or to a copy made by picking one), could not be reached. Retry asks again.
Monitoring
Open Admin > Process > Jobs and switch Job type to Async parse. Each row is one submission, with its source (Copilot, Main chat, Agent flow or Test ingestion), who asked, the files done out of the total, its status and when it was created. A job’s page shows the job’s Duration and lists each file with an eye to preview its text, its status, the parser used, its pages, whether OCR ran, how long it took (Took) and its error. See Async Parse Jobs. Reading the list needs Data Source Managers read access, and it shows every person’s jobs, file names and errors, but not their text: the preview of a file is for the person who asked for the job alone. The parser’s replicas, and the wake-up from zero, are on Admin > Observe > Scaling (see Service Scaling).Settings
All of these are on Admin > Administer > System Settings, in the Async Parsing tab (right after Jobs Settings), where each one can be edited. They are also Configuration Settings in the Django admin (/staff/), ASYNC_PARSE category (Async Parsing). The full list, with every default and what a change does, is in System Settings.
UPLOAD_EXTRACTION_MAX_MEMORY_MB (512 MB by default) and UPLOAD_EXTRACTION_TIMEOUT_SECONDS (60 seconds) still limit the first reading on the API server; a reading that reaches either is handed to the background parser. They are edited in the Django admin only. The three COPILOT_ATTACHMENT_OCR_* settings are no longer used: no reading on the server uses OCR, except that of a Test Ingestion picture, which they do not limit either (it keeps at least 1,024 MB and 300 seconds, and more when UPLOAD_EXTRACTION_MAX_MEMORY_MB or UPLOAD_EXTRACTION_TIMEOUT_SECONDS is set higher).
Sizing the parser
Each parser replica runs one background file at a time, next to its pipeline workers. A reading child can use up toASYNC_PARSE_MAX_MEMORY_MB (the first reading) or ASYNC_PARSE_OCR_MAX_MEMORY_MB (OCR), 3,072 MB each by default. The background program also embeds copilot files in its own process, with the balanced model baked into the parser image. It raises its own out-of-memory priority, so when the container runs short it is stopped before any pipeline worker. One such program runs in each replica (the ASYNC_PARSER_WORKERS environment variable, 1 by default).
In testing, embedding one very large spreadsheet peaked at about 3.4 GB on top of the pipeline workers, so plan for around 4 GB of free memory for this lane in each parser replica.
The API server hands over every file above its limits (1 MB, and every spreadsheet), so the parser has more files to read, one at a time in each replica. If files wait too long in the queue, add a parser replica, or raise ASYNC_PARSER_WORKERS where the memory allows it (see Service Scaling).
Troubleshooting
-
A file stays on Waiting for the parser. The parser may be starting from zero replicas, which takes several minutes, or it is busy with other files. Open Jobs > Async parse to see the file’s status and when it was created, and Scaling to see whether the parser is Scaling up or Running. Check that
async_parseis in the parser’s warm-keeping policies. A file that is still waiting after the queue wait fails (below). -
A file stays on Reading or Parsing… for a long time. One reading may take up to
ASYNC_PARSE_TIMEOUT_SECONDS, and an OCR reading up toASYNC_PARSE_OCR_TIMEOUT_SECONDS, 30 minutes each by default. If the parser’s container was restarted, another parser takes the file over after 120 seconds without a heartbeat. -
A scan was not read. The chip says Scan not read, or the file failed with scanned PDF, not read (the main chat) or Scanned PDF too large to read here (N pages)… (the copilot). Its owner chose Skip, the OCR question went unanswered for 24 hours, a flow’s step had OCR off, or the OCR read no more than the first reading did. Attach the file again and choose Read with OCR, or set the step’s Read scans with OCR to
auto. - A document is missing from the picker. Go through the checklist I can’t find my document in the picker in the picker section: it takes every reason in turn. See also Picking and previewing parsed documents.
-
The Ingested list is empty after an ingestion. The Ingested tab lists a file only when its artifact type has an entity with Attach document to this entity turned on. To fix it: Artifact Types > open the type > Entities & Relationships > pencil on the entity that stands for the document > tick that box > Update Entity > Save the type. Then re-ingest: Jobs > open the job > select the files > Requeue > Ingest Phase (a new scan alone skips unchanged files).
- An empty Ingested tab does not mean the ingestion failed: a file can be ingested and still not be listed. Ingested lists a file only when the knowledge graph holds a document for it, and the graph gets a document through one entity of the file’s artifact type: the one that has the box turned on (it carries a Document badge in the entity list). When no entity of the artifact type has it on, ingestion still writes the entities but writes no document, so the file is not under Ingested, and it is not under Parse only either.
- The entity to tick is the one that stands for the document itself: for a contract type, the Contract entity. The pencil appears when you point at an entity. Only one entity of an artifact type can have the box ticked (it is greyed out while another entity has it).
- The Requeue window opens on Download Phase: choose Ingest Phase, then confirm with Requeue Files.
- If the box was already ticked, check that the entity was really written for the file: a Match Only entity that found nothing to match, and an entity the extraction did not find in the file, write no document.
- The list is also empty for a person whose own graph access does not let them see the documents (see Graph Access Control).
- See The picker for what each list holds.
- The Ingested list says We couldn’t check which documents you may read right now. The knowledge graph could not be reached to check which ingested documents the person may read. Try again asks again once the graph is up (see Graph Backend); the other two lists are not affected.
- A preview says the document can no longer be opened. The file or its session was removed, or the person’s access ended: their Data Source Managers role, their graph access, or their access to the conversation. Check the person’s roles and graph access, and that the knowledge graph is up: the main chat treats an access check it cannot run as a refusal. The list under Previewing the stored text has the other messages.
- A job’s Took is longer than the OCR. Took runs from the file’s first start to the moment it settled, and a scan waits for its owner’s Read with OCR or Skip in between, for up to 24 hours. The tooltip on the cell says what the OCR alone took.
-
A file failed. Hover its chip for the message. The Jobs page shows it in the Error column.
-
Files wait past the queue wait, scans are never skipped, or flow runs do not resume. The sweep that does these runs in the flow runner about once a minute, and delivers finished files to the API server at
ASYNC_PARSE_DISPATCH_URL(http://127.0.0.1:8000by default; it is in the Async Parsing tab of Administer > System Settings). Check that the flow runner is running, and that the API server answers at that address. A wrong address stops finished files from reaching Agent Flows: their runs stay parked.