Service Scaling

​

Overview

Navigate to Admin > Monitoring > Scaling to monitor and manually scale the platform’s document processing services. The dashboard shows each service’s health, current and maximum replicas, and how they fit into the ingestion pipeline. Reading the dashboard requires Monitoring read access. Changing settings or force-scaling a service requires Monitoring write access.

​

Dashboard

The Dashboard tab groups services into three sections:

​

Ingestion Pipeline

The core document processing chain — Reader → Downloader → Parser → Classifier → Ingestion — rendered as a left-to-right flow. An arrow between two stages turns green when both stages are healthy; a queued-message count appears under the arrow when the downstream stage has backlog waiting to be picked up. Pipeline satellites (services that support the pipeline but aren’t sequential stages — coordinator, MinIO, the Kreuzberg text extractor, enrichment, structured data, cleanup) appear as a row of cards underneath the flow.

​

Core Infrastructure (Always On)

Services that cannot be scaled manually: always-on infrastructure with no scaling configuration (server, PostgreSQL, Redis, RabbitMQ) and platform-safety-locked services (Neo4j, FalkorDB). These render with a lock icon and no selection checkbox. How many server pods to run, and the Redis memory limit, are covered in Sizing the API server and Redis.

​

Other Services

Any remaining scalable service that isn’t part of the pipeline or its satellites.

​

Health indicators

Each service card shows a colored dot summarizing its status:

​

Service Detail Page

Click any scalable service’s card to open its detail page, which shows:

• Current State — status, current replicas, scaling mode, and configured min/max bounds.
• Settings tab — edit the service’s autoscaling configuration (requires write access).
• Events tab — the scaling event history for just this service.

​

Settings

Basic settings: Advanced settings (collapsed by default, with a warning that misconfiguration can break the pipeline dependency chain):

• Capacity step size and messages per replica — control how aggressively the queue-based strategy scales.
• Queue names — which RabbitMQ queues this service’s strategy watches.
• Schedule times / window duration — for schedule-based strategies.
• Warm-keeping enabled / warm-keeping policies — keep a service warm during specific in-progress operations (scan orders, structured data scan orders, enrichment jobs, async parse files waiting for the parser or being read, dependency chains) instead of scaling it to zero.
• Dependency order — this service’s position in the pipeline dependency chain.

The Strategy shown at the top of the panel (queue, reader, schedule, manual, phoenix, propelauth) is display-only and cannot be changed from the UI.

​

The parser and async document parsing

Besides the pipeline’s files, the parser service reads the files that async document parsing takes from the knowledge-model copilot, the main chat, Test Ingestion and Agent Flows. Those files wait in a database table, not in a RabbitMQ queue, so the queue strategy never sees them and would let the parser sit at zero replicas (or scale it down in the middle of a reading). The async_parse warm-keeping policy covers them:

• While it counts: the parser is kept warm, and woken from zero, while an async parse file is queued, queued for OCR, or being read by the parser. A file counts for as long as its last sign of life is within the queue wait (ASYNC_PARSE_QUEUE_WAIT_SECONDS, 60 minutes by default). A scan waiting for someone’s Read with OCR or Skip choice never counts, because it waits for a person, not for the parser. Neither does a small main-chat file that the API server is reading itself.
• Where it is set: the parser’s Warm-keeping policies list (scan_order, dependency and async_parse on a new install). An upgrade adds async_parse once to a parser row that lacks it, and leaves every other value in the list, and Warm-keeping enabled, as you set them. If you take async_parse out, or turn warm-keeping off, a file that arrives while the parser is at zero waits until something else starts the parser, and fails after the queue wait if nothing does.
• How long a wake-up takes: a parser at zero replicas needs several minutes to start, and the platform accepts one scaling operation at a time (see Deferred Scaling Conflicts), so a chip can show Waiting for the parser (can take a few minutes) for a while. A file that is still waiting after the queue wait fails with Parsing is unavailable right now; try again later.
• Who wakes it: the autoscaler, so the policy only matters where scale-to-zero is running (see SCALE_TO_ZERO_ENABLED in System Settings).

​

Force Scaling

​

Single Service

From a service’s detail page, click Force Scale (hidden for core services) to open a dialog where you set a target replica count within the service’s configured bounds. Scaling a service to zero opens a confirmation dialog instead: you must type the service name to confirm, and scaling a stateful service (such as MinIO) to zero additionally requires acknowledging that its persistent volume data will be permanently deleted.

​

Multiple Services (Batch)

Services can be scaled together in a single operation:

1. Select services using the checkboxes on their cards (core services are not selectable).
2. Click Force scale selected in the actions bar that appears.
3. Set a target replica count per service — the dialog previews whether each service will scale up, down, or stay unchanged. A reason is required for the audit trail.
4. Confirm to apply all changes as one batch.

Results are reported per service; services that fail to scale stay selected so you can retry them. Each batch is assigned a correlation ID shown with the results, which links the individual scaling events in the All Events tab for auditing. Batch scaling requests are rate-limited to a few per minute — scaling actions take about a minute to materialize (pods must start), so rapid resubmission has no effect.

​

Deferred Scaling Conflicts

The platform serializes scaling operations per instance: only one resource can converge at a time. If you force-scale a service while another resource’s scaling operation is still in progress, the request is deferred rather than treated as a failure — you’ll see “A previous scaling operation is still in progress. Please try again in a few seconds.” in muted text rather than a red error, and no failed event is recorded. Wait a few seconds and retry.

​

Sizing the API server and Redis

​

How many server processes and pods

The server runs the web application and the chat. You cannot scale it from this page. A deployment runs one server pod with one server process in it by default, and the platform can run up to three pods. A server process uses at most one CPU core, and one process carries about 30 people chatting at the same time. The number of processes is the number of pods times the processes in each pod. Add processes as more people chat at the same time: Two Omnistrate deployment parameters set the numbers. Both are set for each deployment, so set them on yours before the busy period starts, and lower them again afterwards, because extra pods and processes use more cloud resources. Ask your platform operator to change them. A third parameter in the table, Database Pool Size per Process, adds no capacity: it sets how many database connections each server process keeps open, and the default rarely needs a change. Rules for choosing the number of processes in a pod. Start from the size of the server node, then check the database node.

• Each process needs about 1 GiB of memory on top of the others, and connections to PostgreSQL of its own. When idle it keeps about 3 open: 1 for web requests, 1 for the chat agent’s memory store and 1 for quick reads of a chat’s saved state. Under load one process can use:
  • up to Database Pool Size per Process connections for web requests and WebSocket calls, 6 by default;
  • up to 16 for the chat agent’s memory store;
  • up to 4 for quick reads of a chat’s saved state, for example when a chat is checked for questions that wait for an answer;
  • 1 for each chat answer in progress, to save the answer’s checkpoints. In the load test below, one process had 55 to 78 answers in progress at once;
  • 1 short extra connection each time some chat actions run, for example when a chat is forked.
• Database node: keep pods times processes at 2 or less on a node with 4 GB of memory, and at 4 or less with 8 GB. 4 GB is the default Azure database node.

What this is based on. In a load test with 80 simulated users chatting at the same time, one server process was fully busy: it used 101 to 108 percent of one CPU core while 55 to 78 answers were in progress. Each call to the language model costs the server about 0.09 CPU-seconds, which puts 80 users at about 2.4 to 2.7 CPU cores of server work. A server process uses at most one core, so one process carries about 30 users, two carry about 60 and three carry about 90. In a later test on a 4-vCPU server node, one pod with 2 processes answered all 80 users, with half of the answers complete within about a minute, and the two processes shared the work evenly. With 3 or 4 busy processes in one 4-vCPU pod, CPU use reaches 75 to 100 percent, so the automatic scale-up can start to work and add pods, up to three. Each added pod runs the same number of processes, and each process opens its own connections to PostgreSQL, so check the database rule above before you raise the processes on a deployment with a small database node. What a server process does on a stop, and when it stops by itself.

• When a pod is stopped or replaced, for example during an upgrade, every process in it keeps serving while the chat answers, admin copilot turns and web requests it is handling finish, for up to 40 seconds, and then stops. The platform allows 60 seconds. An idle pod stops within a few seconds. Agent flow runs and the copilot’s file readings are not waited for: a flow run that was in progress is taken over again after its lease runs out, as after any restart.
• If one server process stops unexpectedly, for example because the system ran out of memory, the whole pod restarts. The other processes finish their work first, as above. A pod that keeps failing to start shows up as repeated restarts of the server pod, not as a process that is silently restarted inside it.
• Each process writes its own log file in the server’s log directory, named after the pod and the process (for example server.server-0.p2.log), so the processes in a pod do not overwrite each other when a log file is rotated. The log collector reads all of them.

Some features keep their state in the process that runs them, and a second process can be reached by a reconnecting browser instead of the first:

• The admin copilot streams a turn to the process that runs it. A browser that reconnects can land on another process, and then shows no stream until the turn ends.
• A graph migration shows its progress, and can be aborted, only through the process that runs it. Set Server Processes per Pod and Server Minimum Pods to 1 for the window.

Scheduled flows and automatic scaling run once, not once per pod. Every server pod starts its own flow runner (it starts scheduled flows) and its own autoscaler, but only one copy of each is active at a time. The active copy holds a lease in Redis. With one pod, the default, there is one copy of each and it is the active one. With two or more pods, the other copies stay idle: they try to take the lease at the start of each cycle, and take over by themselves. A deploy can also run two pods side by side for a short time, if the platform starts the new pod before it stops the old one. In the logs of a pod’s flow runner and autoscaler, the active copy says ACTIVE and an idle copy says STANDBY. With one pod you only see ACTIVE, except that a replacement pod can say STANDBY for up to 90 seconds after a crash, until the lease of the pod that died runs out. The copy that takes over reads its state from the database. The autoscaler’s short cooldown timers start again, as after any restart.

​

Redis memory limit

Redis holds the cached settings and the live updates that agent flows and notifications send to your browser. It refuses new writes at a memory cap that sits below what its node can give it, so a full Redis answers with an out-of-memory error and keeps serving reads, instead of being stopped by the platform for using too much memory. While it refuses writes, cached settings fall back to their last known values and no stored key is dropped, but live updates can fail until memory is freed. The cap is the Omnistrate deployment parameter Redis Max Memory (redisMaxMemory, in bytes). Its default is 1717986918 (1.6 GiB), about 70 percent of the 2.37 GiB that Redis can use on the default Azure node. That node has 4 GiB of memory and offers 2.72 GiB of it to pods, and the system pods on it use about 0.35 GiB of that, which leaves Redis about 2.37 GiB. The cap counts only the data Redis holds, but the platform stops Redis on its real memory use, which runs about 1.2 times the data after many writes and deletes. A cap at 70 percent keeps that real use about 0.4 GiB under the limit. If you move Redis to another size with the Small Instance parameters, take the memory that node offers to pods (its allocatable memory), subtract about 0.35 GiB for system pods, and set Redis Max Memory to about 70 percent of the rest. On the smaller default AWS and GCP nodes (2 GiB of memory) the default is above what Redis can use and has no effect, so set it there.

​

Scaling Events

The All Events tab (global) and each service’s Events tab (per-service) show the scaling audit trail: timestamp, action (scale up/down, manual scale up/down/to-zero, or failed), replica change, who triggered it (or “Autoscaler” for automatic actions), and the reason given.