> ## Documentation Index > Fetch the complete documentation index at: https://docs.experio.cloud/llms.txt > Use this file to discover all available pages before exploring further. # Async Document Parsing > How files that are too big or too slow to read on the API server are read in the background, how to monitor them, and what to do when one is stuck ## Overview Some files cannot be read where they are uploaded. A 23 MB spreadsheet can need 2 GB of memory, and the OCR of a scanned PDF can take minutes and gigabytes. The API server does not do that work. It hands those files to the **background parser** (the async parser), a program in the **parser** service, next to the parser workers of the ingestion pipeline. It reads one file at a time per parser replica, in a child process with a memory limit and a time limit. * **No queue service.** A waiting file is a row in the database, and the background parser claims rows from there. Nothing goes through RabbitMQ, and the ingestion pipeline is unchanged. * **Results are stored in the database.** For each file the parser keeps the whole text and its chunks (and, for the knowledge-model copilot, the vectors its search uses). * **People see their own files.** The chips where people attach files show their own files, and each ready file has an **eye** that opens its stored text ([Previewing the stored text](#previewing-the-stored-text)). Administrators see every job on the [Jobs page](#monitoring), but only the person who asked for a job can open the text of its files. ## Where it is used | Where | What goes to the background parser | OCR for a scan | Search vectors | | - | - | - | - | | **Knowledge-model copilot** ([Attaching files](/admin-guide/admin-copilot#attaching-files)) | Every file, unless the person turns **Parse in the background** off; with it off, the files the server should not read (below). Also a picked document over 200,000 characters | Asks first for more than 2 pages | Yes | | **Main chat** ([Attaching Files](/user-guide/chat-interface#attaching-files)) | The same, and a text over 2,000,000 characters | Asks first for more than 2 pages | No | | **Agent Flows** ([Parse documents (async)](/admin-guide/agent-flows-blocks#data-nodes)) | Every file of the step | The step's setting, `auto` by default | No | | **Test Ingestion** ([Content Types](/admin-guide/content-types#test-ingestion)) | Files the server should not read | Automatic | No | ## When a file goes to the background parser **By default, every file attached in the copilot or the main chat.** The **Parse in the background** switch, in the paperclip menu of both (one switch for each conversation or session), is on until the person turns it off, and while it is on every file goes to the background parser whatever its size (the one exception is a plain `.txt` or `.md` file in the main chat, which is read at once). A new conversation starts with the switch on again. With the switch off, and in Test Ingestion, the API server reads a file first, in a child process with no OCR, and hands it over only when that is not enough: * **Too big on sight.** A file over `ASYNC_PARSE_PRECHECK_MAX_FILE_MB` (1 MB by default), or a spreadsheet over `ASYNC_PARSE_PRECHECK_MAX_SPREADSHEET_MB` (0 by default, which sends every spreadsheet, whatever its size). It is not read on the server at all. A limit of `0` sends every file, or every spreadsheet (but never a Test Ingestion picture, below). Raise the spreadsheet limit to have small spreadsheets read on the server. * **A reading that stops.** The reading reaches `UPLOAD_EXTRACTION_MAX_MEMORY_MB` (512 MB by default) or `UPLOAD_EXTRACTION_TIMEOUT_SECONDS` (60 seconds), or the server's container or machine has less than 768 MB of memory left. * **A scan.** A PDF that gives fewer than 60 letters and digits a page, on average, when it is read without OCR. Its pages are pictures, with at most a signature line or a stamp as text. With the switch off, a small file (up to 1 MB, other than a spreadsheet) is read on the server, usually in seconds. Plain text (`.txt`, `.md`) in the main chat is decoded in place whatever the switch says. A zip that expands too far is refused instead of handed over, because the background parser would refuse it too. In Test Ingestion, a picture is read by OCR in place, whatever its size: the background parser's first reading uses no OCR and would find no words in it. Nothing else can read a picture, so it keeps a memory budget and a time limit of its own, never below 1,024 MB and 300 seconds, and one that needs more fails with *this file is too large to read here* instead of being handed over. These limits are low on purpose, so that the background parser does the heavy reading. In tests on a fast laptop at the earlier, higher limits, ordinary files under them needed about 140 to 365 MB and finished within about 25 seconds. That is not a maximum: a file can pass the size check and still need more, for example a spreadsheet with a very large sparse table (about 660 MB in one test) or a Word file with one enormous table (about 800 MB). Such a file is stopped at the limit and read by the background parser instead, so nothing fails. A server slower than the test machine also stops some text-heavy PDFs at the 60-second limit and hands them over. The price is waiting: a file over 1 MB, and every spreadsheet, waits for the background parser, which takes several minutes to start when it has been scaled to zero (see [What happens to a file that waits](#what-happens-to-a-file-that-waits)). On an API server with memory to spare, raise the two pre-check limits together with `UPLOAD_EXTRACTION_MAX_MEMORY_MB` and `UPLOAD_EXTRACTION_TIMEOUT_SECONDS`; a spreadsheet needs far more memory than its size suggests, so keep its limit the lowest. ## OCR choices A scanned PDF is read by OCR only inside the background parser. Each surface chooses how it asks: | Mode | What happens to a scan | Used by | | - | - | - | | `ask` | A scan of up to `ASYNC_PARSE_OCR_ASK_ABOVE_PAGES` pages (2 by default) is read by OCR at once. A longer scan waits for its owner: **Read with OCR (\~N min)** or **Skip** on its chip, with no question in the conversation | Copilot, main chat | | `auto` | Always read by OCR, without asking | Test Ingestion and Agent Flows by default | | `force` | Always runs the OCR pass on a PDF and keeps the richer text | Agent Flows, by setting | | `off` | Never uses OCR. A scan is stored as read, flagged *Scan not read*, with whatever text it has, and fails when it has none | Agent Flows, by setting | * The estimate on the button is the page count times `ASYNC_PARSE_OCR_SECONDS_PER_PAGE` (10 seconds by default). It limits nothing. * A question nobody answers is skipped after `ASYNC_PARSE_NEEDS_OCR_EXPIRY_HOURS` (24 hours by default): the file settles as a scan nobody read. * The OCR text is kept only when it holds more letters and digits than the first reading. Otherwise the file keeps its first reading, and a scan is flagged as not read. ## What happens to a file that waits * **Queue.** A file starts as *Queued*. A parser takes one file at a time, and an idle parser looks for work every few seconds. * **Turns.** The next file is taken from the person who was served longest ago, and someone who was never served goes first, so one person's ten big files do not hold everyone else's past the queue wait. Among one person's files, first readings go before OCR readings, older files before newer ones, and an OCR reading that has waited 10 minutes goes first. * **Waking the parser.** The parser can be scaled to zero. The `async_parse` warm-keeping policy wakes it while a file waits and keeps it up during a reading. A wake-up takes several minutes (see [The parser and async document parsing](/admin-guide/service-scaling#the-parser-and-async-document-parsing)). Because **Parse in the background** is on by default, every file attached in the copilot or the main chat (plain text in the main chat apart) can wait for a wake-up, a tiny one included. With the switch off, so can a file over 1 MB or any spreadsheet, and so can a Test Ingestion sample of that kind. * **Queue wait.** A file that stays *Queued* (or queued for OCR) longer than `ASYNC_PARSE_QUEUE_WAIT_SECONDS` (60 minutes by default) fails with *Parsing is unavailable right now; try again later*. A file from an Agent Flow waits for its step's deadline instead: 30 minutes by default, 2 hours at most. * **A reading that dies.** The parser records a heartbeat every 30 seconds while it reads (`ASYNC_PARSE_HEARTBEAT_SECONDS`). When a file has had none for 120 seconds (`ASYNC_PARSE_STALE_HEARTBEAT_SECONDS`), another parser takes it over. After `ASYNC_PARSE_MAX_ATTEMPTS` (3) readings that died under it, the file fails with *Reading this file failed after 3 attempts: the parser stopped each time before it finished*. A parser that shuts down cleanly puts its file back without counting an attempt, and so does a reading that stops for lack of server memory, up to three times for a file. * **Limit per person.** A person can have `ASYNC_PARSE_MAX_QUEUED_PER_USER` files (10 by default) waiting or being read at once, across every surface. A scan waiting for its owner's OCR choice counts, for up to 24 hours. A small main-chat file that the API server reads itself is not held to it. With **Parse in the background** on, which is the default, the API server reads none (plain text apart), so every file counts, and the 11th of 11 files dropped at once is refused until one of them settles. Turn the switch off to attach small files (spreadsheets apart) past the limit. Past the limit, a new file is refused with *N of your files are already waiting to be read (at most 10 at a time)*. * **Cancel.** Removing a chat file or a copilot document, closing a copilot session, cancelling a flow run or a test, and **Cancel** on a waiting screen all cancel the file. A file that waits is never read, and a reading in progress stops within about half a minute. * **Retention.** A finished file is deleted 7 days later (`ASYNC_PARSE_RETENTION_DAYS`), unless a chat file or a copilot document still uses it. The uploaded original is deleted as soon as the file has been read, and an upload nothing points at is removed within about an hour. * **Delivery.** When a file settles, its surface is told once: the copilot announces it, the main chat shows a notice, and a flow run resumes. A sweep that the **flow runner** (the job that also runs scheduled flows) runs about once a minute handles queue waits, expired OCR questions, delivery to flows and clean-up. ## Picking and previewing parsed documents A parsed result can be used again, and read in the browser, without reading the file again. Picking a document attaches a copy of its stored text; a preview only reads it. ### The picker **Attach from parsed documents** (in the copilot and the main chat) and the document choice of the **Parse documents (async)** step open the same picker, with up to three lists. A list holds only the documents that can be attached: | List | What it holds | Listed when | | - | - | - | | **My uploads** | The person's own finished files, from the chat, the copilot and Test Ingestion | Always: the result holds its own text | | **Parse only** | Files of data sources whose **Ingestion type** is **Parse only**: they were read and their text was saved, but they were not added to the knowledge graph | The file's parsed text is still in storage | | **Ingested** | Files of data sources that went through the whole pipeline and whose document is in the knowledge graph, because an entity of their artifact type has **Attach document to this entity** turned on | The person may read the document in the graph, and the parsed text file is in storage or the text kept in the graph can be read | Each tab says this in one line under the tabs, and again as its tooltip. **Narrowing a list.** Every list has a search box and a **File type** filter. **Parse only** and **Ingested** have more: * **Data source:** one or more data sources, by name. * **Job:** one run of a data source: a **job** is one scan of its files, with the number it has on the [Jobs page](/admin-guide/jobs-monitoring) (*Job #113* there, *Job 113* in the picker). An option also gives the day the run started, and names its data source while none or several are ticked: *Job 113 · Team Box · Oct 9, 2026*. Once data sources are ticked, the Job list offers only their jobs. Every row of these two lists shows its job (*Job 113*) beside its data source. * **Artifact type** (**Ingested** only): the artifact type the classifier gave the file, for example *Contract*. It sits beside **File type** (the extension). A parse-only file is never classified, so **Parse only** has none. * **Clear filters** appears while any of the three is set. It clears them, not the search or the file type. Changing the tab clears them too, and a changed filter starts again at page 1. Each filter takes one or more values. A document is listed when it has any one of a filter's values and matches every other filter and the search. The options are only what you can already list: a data source, job or artifact type with no document you may see is not offered (the stored text is not checked, so an option may list nothing). If they cannot be loaded, a note says *Filters could not be loaded. The list still works.* and these three filters stay disabled. **Which list a file is in.** **Parse only** and **Ingested** both hold files of data sources (not files people attached in a chat), and the list follows how the file was processed. A data source whose **Ingestion type** is **Parse only** stops every file after it is read, so its files are listed under **Parse only**. A data source on **Full ingestion** runs the whole pipeline, and its files can be listed under **Ingested**, but only when the knowledge graph holds a document for the file. The graph gets that document through the entity that has **Attach document to this entity** turned on in the file's artifact type (one entity per artifact type at most; see [Content Types](/admin-guide/content-types)). A file ingested into an artifact type with no such entity is ingested, but no document is written for it, so it is in neither list. A scan skips a file that has not changed since it was read. A file that changed after its data source was switched from **Parse only** to **Full ingestion** is read again as a new entry next to the old one, and each list shows its newest entry, so the file can appear in both lists (the **Parse only** one holds the older text). A document whose stored text is gone is not listed. It is not greyed out and nothing counts it; it is missing from the list, and a list with none is empty (*No documents available to attach.*). An empty **Ingested** list says instead *Nothing here yet. A document is listed once it has been ingested, its artifact type attaches the document to an entity, and you may read it in the graph.* (see [I can't find my document in the picker](#i-cant-find-my-document-in-the-picker)). A search or a file type that leaves nothing says *No documents match your search.* in every list, and a data source, job or artifact type that leaves nothing says *No documents match these filters.* The count, the number of pages and every page's rows are those of the documents that can be attached. * **How it is decided.** Where the stored file decides, the server looks up the file of each candidate once, newest first, before it cuts the page; no text is read, so a page of the list stays quick. The graph's copy of an ingested document's text can be read unless graph access control is enforcing (`GRAPH_AUTHORIZATION_MODE` set to `enforce`): then an ingested document with no parsed text file is not listed, except for a super-admin while `GRAPH_AUTHZ_SUPERUSER_BYPASS` is on. Where the graph's copy can stand in, an ingested document is listed without a lookup. See [Graph Access Control](/admin-guide/graph-access-control) and [System Settings](/admin-guide/system-settings). * **A long list.** Where the stored file decides (a **Parse only** list, and an **Ingested** list while graph access control is enforcing), the server checks the newest 2,000 candidates; an **Ingested** list also covers only the 5,000 newest documents the graph is asked about. When a list is cut at either limit, the count ends in a plus (*2,000+ documents*, or the number found among the newest ones) and a tooltip says so. The search and every filter (data source, job, artifact type and file type) narrow the documents before the list is cut at those limits, so any of them reaches an older document. * **The options are capped too.** Each filter lists at most 100 entries: 100 data sources, the 100 newest jobs and, on **Ingested**, 100 artifact types. The **Parse only** options come from every parse-only document you may see. The **Ingested** options come only from the 5,000 newest documents the graph is asked about, and only those it lets you read, so a very old data source or job may not be offered there. A note under the filters then says *Only the newest data sources and jobs are listed. Search by name to find an older document.* Searching by name still finds the document, because a search narrows the documents before that cut. * **Who sees which data sources.** Data Source read access (**Data Source Managers** read) is not set per data source: a person who has it can list the **Parse only** documents of every data source, and the filters offer all of them. There is no setting that limits a person to some data sources. **Ingested** is narrower: each document must also pass the knowledge graph's access check, so a person sees only the ingested documents their own graph access lets them read, and the filters offer only the data sources, jobs and artifact types of those documents. * **While the list loads, and when it fails.** Changing the list, the search, a filter, the file type or the page clears the rows at once and shows a loading state until the new rows arrive; the rows of another query are never shown. If the list cannot be read it shows one sentence and **Try again**, with no count and no pager. The **Ingested** list asks the knowledge graph which documents the person may read, so while the graph cannot be reached it says *We couldn't check which documents you may read right now. Try again in a moment.* (the server answers 503 with the code `scope_unavailable`); **My uploads** and **Parse only** do not use the graph. * **A first check, not a promise.** A listed document can still be refused when it is attached (its stored text is empty, or the person's access ended since the list was drawn). **Attach** tries every ticked document and lists every refusal at once, in one box, instead of stopping at the first. The documents that went in stay attached. A ticked document that a list of one page no longer shows when it is read again leaves the selection. * **The way back.** A document that is missing is back once it is re-ingested, or the file is uploaded again. In the flow editor the picker lists the same documents, and a document that has lost its text is not listed there either. One chosen earlier stays in the step's list of picked documents until it is removed with its **×**, and the run names it with the reason (*This document's text is no longer stored; re-ingest it or upload the file.*). ### I can't find my document in the picker Go through these in order; the first that fits is usually the answer. In short: **My uploads** holds only your own files, and three uploads are not kept there (a `.txt` or `.md` file in the main chat, a small copilot file that the API server read itself, and a Test Ingestion sample read the same way), so attach such a file again instead of picking it; a **Parse only** data source never fills **Ingested**; a file still being processed, or one that failed, is not listed; **Ingested** empty although the file was ingested: tick **Attach document to this entity** on the document's entity, **Update Entity**, **Save**, then **Requeue** > **Ingest Phase**; the data source tabs need **Data Source Managers** read access; text gone: re-ingest or upload again; long list: search by name or use the filters; graph down: try again. 1. **You uploaded it in a chat or in the copilot.** It is under **My uploads**, not under a data source. **My uploads** lists only your own files, and only once they have been read. Three kinds of upload are not kept there, so attach such a file again instead of picking it: * a `.txt` or `.md` file in the main chat, which is always read in place; * a small file in the copilot (up to 1 MB, not a spreadsheet), with **Parse in the background** turned off, that the API server read itself; * a Test Ingestion sample that the API server could read itself (a file up to 1 MB that is not a spreadsheet, or a picture, which is always read there; see [Test Ingestion](/admin-guide/test-ingestion)). 2. **It comes from a data source that only parses.** A file of a data source whose **Ingestion type** is **Parse only** is under **Parse only**, and never under **Ingested**: it was read and its text was saved, but it was not added to the knowledge graph. A data source on **Full ingestion** puts its files under **Ingested** instead. A file that changed after its data source was switched to **Full ingestion** is read again as a new entry next to the old one, so it can be in both. 3. **The file is still being processed, or it failed.** A file that is waiting, being downloaded, parsed, classified or ingested is not listed yet. A file whose ingestion failed, or that was skipped or stopped, is in neither tab (an earlier scan of the same file that succeeded is still listed). Check the file's status on the **Jobs** page. 4. **The Ingested tab is empty although the file was ingested.** Its artifact type has no entity that attaches the document, so the knowledge graph holds no document for the file: it is not under **Ingested**, and not under **Parse only** either. Open **Artifact Types**, open the artifact type, go to **Entities & Relationships**, edit the entity that stands for the document itself, tick **Attach document to this entity**, press **Update Entity**, then **Save** the artifact type. Then ingest the file again: select it in the job's file list, click **Requeue** and choose the **Ingest Phase**. The Troubleshooting bullet *The Ingested list is empty after an ingestion* has the details. 5. **You are not allowed to see it.** **Parse only** and **Ingested** are for people with **Data Source Managers** read access; without it the picker shows only **My uploads**. **Ingested** also lists only the documents your own graph access lets you read (see [Graph Access Control](/admin-guide/graph-access-control)). Ask an administrator. 6. **The stored text is gone.** A document whose parsed text is no longer in storage is not listed. An ingested document stays listed while the knowledge graph keeps a copy of its text, unless graph access control is enforcing: then it needs its parsed text file, which the pipeline's clean-up deletes `MINIO_RETENTION_DAYS` days after it was written (7 by default, in [System Settings](/admin-guide/system-settings)). So under enforcement an ingested document leaves the **Ingested** list about a week (by default) after its file was parsed, except for a super-admin while `GRAPH_AUTHZ_SUPERUSER_BYPASS` is on. A **Parse only** file keeps its parsed text. Re-ingest the file, or upload it again. A file you uploaded yourself is kept for 7 days after it was read (by default), unless a conversation or a copilot session still uses it. 7. **The list is very long.** The picker checks only the newest documents: 2,000 for **Parse only**, and 5,000 for **Ingested** (2,000 while graph access control is enforcing). The count then ends in a plus, for example *2,000+ documents*. Search by name, or use the **Data source**, **Job**, **Artifact type** (**Ingested** only) and **File type** filters, to reach an older document. The filters list at most 100 entries each, and on **Ingested** a very old data source or job may be missing from them: search by name then. 8. **The knowledge graph cannot be reached.** The **Ingested** tab says *We couldn't check which documents you may read right now. Try again in a moment.* Try again once the graph is up (see [Graph Backend](/admin-guide/graph-backend)). **My uploads** and **Parse only** do not need the graph. ### Previewing the stored text The **eye** icon (tooltip *Preview parsed text*) opens the text the parser stored for a file, in a window (a chip in the main chat opens it in the right panel) that shows it a slice at a time: the first 20,000 characters, then 100,000 for each **Load more**. What the window offers (header, *Page n* dividers, search, **Copy**) is described in [Previewing the text of a file](/user-guide/chat-interface#previewing-the-text-of-a-file). The database cuts the slice of a parsed file's text, so even a text of tens of millions of characters is never loaded whole; a **Parse only** or **Ingested** document in the picker comes from its stored file (an ingested one with no stored file, from the graph's copy of its text), which is read in full for every slice, as attaching it is. A preview reads and does nothing else: it creates, copies and caches nothing, the answer is marked so the browser does not keep it, and the server logs only ids and character counts, never the text. The eye is where the person already works with the file, and each place has its own rule for who may open it: | Where the eye is | The text it opens | Who can open it | | - | - | - | | A row of the picker | The document as **Attach** would take it | The person who could pick it: their own results under **My uploads**; for **Parse only** and **Ingested**, **Data Source Managers** read access, and for **Ingested** their own graph access too | | A chip in the main chat, in the message box or under a sent message | The file's whole stored text. For a plain-text file read in place, the 10,000 characters kept on the message | The uploader and staff accounts, before and after sending. After sending, also the people with access to the conversation. A document picked from a data source also needs the reader's own access to that source at that moment | | A chip in the copilot | The document's text | The owner of the open session that holds it, with **Graph Managers** read access. A document picked from a data source is a snapshot in the session: its preview does not check the source again | | A file row on the Jobs page | The file's whole stored text | Only the person who requested the job | When a preview cannot show text, it says why: * *You can no longer open this document (it was removed, or you lost access to its source).* The server answers a file the person may not read exactly as it answers one that does not exist, so the message cannot tell which. It covers a file or session that was removed or closed, a finished file that nothing used and that passed its 7-day retention, an ended access from the table above and, for a copy that picking a data-source document made, a source the person can no longer read: such a copy stays readable only while its source does. In the main chat it also covers an access check that could not run just then (the knowledge graph was unreachable): the main chat treats a check it cannot run as a refusal, so the message cannot tell the two apart. * *This document's text is no longer stored; re-ingest it or upload the file.* The picker's eye on a document whose parsed text went after the list was drawn: the picker lists no document that already lacks it. * *The parsed text of this file is no longer stored.* A picked file in the main chat whose stored parse was deleted; the message keeps no other copy. * *This file is still being read. Its text can be opened once that is done.* or *This scanned file is waiting for a choice about reading it with OCR.* The file has no text yet. The failure's own error appears when the file failed (*Reading this file was cancelled.* when it was cancelled). * *This scan was not read (no OCR was run).* The main chat's answer for a scan that nobody read with OCR. The copilot, the picker and the Jobs page show the little text the first pass found instead, with *Scan not read* in amber in the window's header. * *Could not check your access right now. Try again.* The answer of the picker's eye and of the Jobs page when the knowledge graph, which checks access to an ingested document (or to a copy made by picking one), could not be reached. **Retry** asks again. The eye on the Jobs page is enabled for a finished file of the person's own job. A copy made by picking a data-source document is checked again when the eye is clicked, so the click can still end with the *no longer open* message or with the access check message. ## Monitoring Open **Admin > Process > Jobs** and switch **Job type** to **Async parse**. Each row is one submission, with its source (*Copilot*, *Main chat*, *Agent flow* or *Test ingestion*), who asked, the files done out of the total, its status and when it was created. A job's page shows the job's **Duration** and lists each file with an **eye** to preview its text, its status, the parser used, its pages, whether OCR ran, how long it took (**Took**) and its error. See [Async Parse Jobs](/admin-guide/jobs-monitoring#async-parse-jobs). Reading the list needs **Data Source Managers** read access, and it shows every person's jobs, file names and errors, but not their text: the preview of a file is for the person who asked for the job alone. The parser's replicas, and the wake-up from zero, are on **Admin > Observe > Scaling** (see [Service Scaling](/admin-guide/service-scaling)). ## Settings All of these are on **Admin > Administer > System Settings**, in the **Async Parsing** tab (right after **Jobs Settings**), where each one can be edited. They are also **Configuration Settings** in the Django admin (`/staff/`), **ASYNC\_PARSE** category (*Async Parsing*). The full list, with every default and what a change does, is in [System Settings](/admin-guide/system-settings#async-parsing-settings). | Setting | Default | What it controls | | - | - | - | | `ASYNC_PARSE_QUEUE_WAIT_SECONDS` | `3600` | How long a file may wait for the parser | | `ASYNC_PARSE_MAX_QUEUED_PER_USER` | `10` | Files one person may have waiting or being read | | `ASYNC_PARSE_RETENTION_DAYS` | `7` | How long a finished file is kept | | `ASYNC_PARSE_NEEDS_OCR_EXPIRY_HOURS` | `24` | How long a scan waits for its OCR choice | | `ASYNC_PARSE_OCR_ASK_ABOVE_PAGES` | `2` | Scans of this many pages or fewer are read without asking | | `ASYNC_PARSE_PRECHECK_MAX_FILE_MB`, `ASYNC_PARSE_PRECHECK_MAX_SPREADSHEET_MB` | `1`, `0` | Sizes above which a file is not read on the API server (`0` sends every file, every spreadsheet) | | `ASYNC_PARSE_MAX_MEMORY_MB`, `ASYNC_PARSE_TIMEOUT_SECONDS` | `3072`, `1800` | Memory and time for one reading | | `ASYNC_PARSE_OCR_MAX_MEMORY_MB`, `ASYNC_PARSE_OCR_TIMEOUT_SECONDS` | `3072`, `1800` | Memory and time for one OCR reading | | `ASYNC_PARSE_MAX_EMBED_CHUNKS`, `ASYNC_PARSE_EMBED_BUDGET_SECONDS` | `20000`, `600` | How much of a copilot file is embedded for search | | `ASYNC_PARSE_FLOW_DEFAULT_TIMEOUT_SECONDS`, `ASYNC_PARSE_FLOW_MAX_TIMEOUT_SECONDS` | `1800`, `7200` | How long an Agent Flow step waits, by default and at most | `UPLOAD_EXTRACTION_MAX_MEMORY_MB` (512 MB by default) and `UPLOAD_EXTRACTION_TIMEOUT_SECONDS` (60 seconds) still limit the first reading on the API server; a reading that reaches either is handed to the background parser. They are edited in the Django admin only. The three `COPILOT_ATTACHMENT_OCR_*` settings are no longer used: no reading on the server uses OCR, except that of a Test Ingestion picture, which they do not limit either (it keeps at least 1,024 MB and 300 seconds, and more when `UPLOAD_EXTRACTION_MAX_MEMORY_MB` or `UPLOAD_EXTRACTION_TIMEOUT_SECONDS` is set higher). ## Sizing the parser Each parser replica runs one background file at a time, next to its pipeline workers. A reading child can use up to `ASYNC_PARSE_MAX_MEMORY_MB` (the first reading) or `ASYNC_PARSE_OCR_MAX_MEMORY_MB` (OCR), 3,072 MB each by default. The background program also embeds copilot files in its own process, with the `balanced` model baked into the parser image. It raises its own out-of-memory priority, so when the container runs short it is stopped before any pipeline worker. One such program runs in each replica (the `ASYNC_PARSER_WORKERS` environment variable, 1 by default). In testing, embedding one very large spreadsheet peaked at about 3.4 GB on top of the pipeline workers, so plan for around 4 GB of free memory for this lane in each parser replica. The API server hands over every file above its limits (1 MB, and every spreadsheet), so the parser has more files to read, one at a time in each replica. If files wait too long in the queue, add a parser replica, or raise `ASYNC_PARSER_WORKERS` where the memory allows it (see [Service Scaling](/admin-guide/service-scaling#the-parser-and-async-document-parsing)). ## Troubleshooting * **A file stays on *Waiting for the parser*.** The parser may be starting from zero replicas, which takes several minutes, or it is busy with other files. Open **Jobs > Async parse** to see the file's status and when it was created, and **Scaling** to see whether the parser is *Scaling up* or *Running*. Check that `async_parse` is in the parser's warm-keeping policies. A file that is still waiting after the queue wait fails (below). * **A file stays on *Reading* or *Parsing…* for a long time.** One reading may take up to `ASYNC_PARSE_TIMEOUT_SECONDS`, and an OCR reading up to `ASYNC_PARSE_OCR_TIMEOUT_SECONDS`, 30 minutes each by default. If the parser's container was restarted, another parser takes the file over after 120 seconds without a heartbeat. * **A scan was not read.** The chip says *Scan not read*, or the file failed with *scanned PDF, not read* (the main chat) or *Scanned PDF too large to read here (N pages)…* (the copilot). Its owner chose **Skip**, the OCR question went unanswered for 24 hours, a flow's step had OCR off, or the OCR read no more than the first reading did. Attach the file again and choose **Read with OCR**, or set the step's **Read scans with OCR** to `auto`. * **A document is missing from the picker.** Go through the checklist [I can't find my document in the picker](#i-cant-find-my-document-in-the-picker) in the picker section: it takes every reason in turn. See also [Picking and previewing parsed documents](#picking-and-previewing-parsed-documents). * **The Ingested list is empty after an ingestion.** The **Ingested** tab lists a file only when its artifact type has an entity with **Attach document to this entity** turned on. To fix it: **Artifact Types** > open the type > **Entities & Relationships** > pencil on the entity that stands for the document > tick that box > **Update Entity** > **Save** the type. Then re-ingest: **Jobs** > open the job > select the files > **Requeue** > **Ingest Phase** (a new scan alone skips unchanged files). * An empty **Ingested** tab does not mean the ingestion failed: a file can be ingested and still not be listed. **Ingested** lists a file only when the knowledge graph holds a document for it, and the graph gets a document through one entity of the file's artifact type: the one that has the box turned on (it carries a *Document* badge in the entity list). When no entity of the artifact type has it on, ingestion still writes the entities but writes no document, so the file is not under **Ingested**, and it is not under **Parse only** either. * The entity to tick is the one that stands for the document itself: for a contract type, the *Contract* entity. The pencil appears when you point at an entity. Only one entity of an artifact type can have the box ticked (it is greyed out while another entity has it). * The **Requeue** window opens on **Download Phase**: choose **Ingest Phase**, then confirm with **Requeue Files**. * If the box was already ticked, check that the entity was really written for the file: a **Match Only** entity that found nothing to match, and an entity the extraction did not find in the file, write no document. * The list is also empty for a person whose own graph access does not let them see the documents (see [Graph Access Control](/admin-guide/graph-access-control)). * See [The picker](#the-picker) for what each list holds. * **The Ingested list says *We couldn't check which documents you may read right now*.** The knowledge graph could not be reached to check which ingested documents the person may read. **Try again** asks again once the graph is up (see [Graph Backend](/admin-guide/graph-backend)); the other two lists are not affected. * **A preview says the document can no longer be opened.** The file or its session was removed, or the person's access ended: their **Data Source Managers** role, their graph access, or their access to the conversation. Check the person's roles and [graph access](/admin-guide/graph-access-control), and that the knowledge graph is up: the main chat treats an access check it cannot run as a refusal. The list under [Previewing the stored text](#previewing-the-stored-text) has the other messages. * **A job's *Took* is longer than the OCR.** *Took* runs from the file's first start to the moment it settled, and a scan waits for its owner's **Read with OCR** or **Skip** in between, for up to 24 hours. The tooltip on the cell says what the OCR alone took. * **A file failed.** Hover its chip for the message. The Jobs page shows it in the **Error** column. | Message | Cause and what to do | | - | - | | *Parsing is unavailable right now; try again later* | The file waited longer than the queue wait. Check that the parser can start, then attach the file again | | *Reading this file failed after N attempts: the parser stopped each time before it finished* | The parser died under the file each time, usually out of memory. See [Sizing the parser](#sizing-the-parser), or attach a smaller file | | *this file is too large to read here (reading it needed more than N MB of memory); attach a smaller or text version* | The parser's own memory limit. Split or export the file, or raise `ASYNC_PARSE_MAX_MEMORY_MB` on a parser with the memory | | *reading this file took too long (more than N seconds)* | The parser's time limit. Split the file, or raise `ASYNC_PARSE_TIMEOUT_SECONDS` | | *Scanned PDF: its pages are images, and reading them with OCR needed more than N MB of memory …* (or *took more than N seconds …*) | The OCR limits. Attach a text version, or raise `ASYNC_PARSE_OCR_MAX_MEMORY_MB` or `ASYNC_PARSE_OCR_TIMEOUT_SECONDS` | | *The server ran short of memory while reading this file; attach it again later, or a smaller file* | The parser's container ran short of memory while reading the file, and again on each of the three tries after it. Attach it later, or give the parser more memory | | *No text could be read from this file.* | The file has no readable text. Attach a text version | | *The uploaded file is no longer stored; upload it again.* | The upload was deleted before the parser read it. Upload it again | | *N of your files are already waiting to be read (at most N at a time)* | The person is at the limit on background files. They can retry when one finishes, or you can raise `ASYNC_PARSE_MAX_QUEUED_PER_USER` | * **Files wait past the queue wait, scans are never skipped, or flow runs do not resume.** The sweep that does these runs in the **flow runner** about once a minute, and delivers finished files to the API server at `ASYNC_PARSE_DISPATCH_URL` (`http://127.0.0.1:8000` by default; it is in the **Async Parsing** tab of **Administer > System Settings**). Check that the flow runner is running, and that the API server answers at that address. A wrong address stops finished files from reaching Agent Flows: their runs stay parked. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.