Separating worker nodes

For high-traffic instances with many followers, you can improve performance by separating the web server from background workers. This allows you to scale each component independently based on your workload.

When to consider worker separation

Worker separation is beneficial when:

• You have thousands of followers and experience slow activity delivery
• Your instance handles heavy federation traffic (many incoming/outgoing posts)
• The web server becomes less responsive during peak activity times
• You want to scale horizontally across multiple servers

Note

For most single-user instances with moderate activity, running everything in a single process (the default) is perfectly adequate. Worker separation adds deployment complexity and is only recommended for high-traffic scenarios.

How it works

Hollo has six main components:

1. Web server: Handles HTTP requests (API, web UI)
2. Fedify message queue: Processes ActivityPub inbox/outbox messages
3. Import worker: Handles background data import jobs
4. Cleanup worker: Handles background cleanup jobs
5. Remote replies scrape worker: Fetches remote replies collections slowly
6. Poll notification tasks: Notify local authors and voters after expiry

By default (NODE_TYPE=all), all six run in a single process. You can separate them using the NODE_TYPE environment variable:

NODE_TYPE	Web server	Fedify queue	Import worker	Cleanup worker	Remote replies scrape worker	Poll notifications
all (default)	✓	✓	✓	✓	✓	✓
web	✓	✗	✗	✗	✗	✗
worker	✗	✓	✓	✓	✓	✓

Import and cleanup tasks

Import and cleanup run as Fedify tasks on a separate PostgreSQL queue, with at most four item attempts per worker process. Web nodes register and enqueue these tasks but never consume queues. Worker and combined nodes also run a recovery pass once a minute, examining at most 100 active jobs per kind. This pass restores missed dispatches and interrupted work; it does not execute items itself. Progress counts terminal results, rather than claimed items.

Before upgrading from the polling workers, stop all older web and worker nodes, apply the database migrations, and deploy the same version to every node. Do not mix task definitions across deployments. Existing pending and processing items are recovered; incomplete legacy upload batches are marked failed and must be submitted again. The existing federation queue and its messages retain their identity.

A worker can use up to 24 database connections: the main pool of 20 and four isolated item-lock connections. Queue separation does not isolate CPU or storage load. Shutdown stops new attempts and waits up to 30 seconds for running work. Forced termination leaves unfinished items recoverable. Transient failures retry at five-second intervals, failing after five recorded failures; process termination does not consume that budget. External activity delivery may repeat after interruption, using the saved activity IDs. Job cancellation prevents new work and does not undo actions already performed. Already committed import relationships still deliver their saved activities, including after cancellation during a retry.

All nodes share the same PostgreSQL database, which acts as the message queue backend using LISTEN/NOTIFY for real-time message delivery.

Remote replies scrape tasks

Remote replies scraping shares the dedicated task queue with import and cleanup. Each worker runs at most four application tasks, including at most two scrapes. Backpressure counts ready messages, so delayed wakeups do not block other work. PostgreSQL job and origin records still enforce collection cooldown, host request spacing, and HTTP 429 backoff across workers. Normal execution and retry wakeups use tasks; a startup and once-a-minute recovery pass examines at most 100 stale jobs and 100 stale orphan origin leases, and dispatches at most 50 runnable scrapes. It repairs missed enqueues and interrupted attempts without fetching remote collections itself. Jobs created within a transaction are dispatched by recovery after commit.

Stop all older nodes before upgrading and deploy compatible task definitions on every node. Existing pending jobs are recovered, and processing jobs are reclaimed after their 15-minute heartbeat lease expires. Shutdown interrupts throttling waits and returns settled attempts to pending. A forced termination leaves the lease to expire. HTTP 429 retries use the recorded due time; other scrape errors remain terminal failures. The existing scrape environment variables and authenticated local-actor fetching retain their meaning.

Poll notification tasks

Poll expiry notifications share the dedicated Fedify task queue. Web nodes schedule tasks; worker and combined nodes execute them, reloading the current expiry and local recipients. Changed expiries and duplicate messages cannot notify early or duplicate owner/poll notifications. Very distant expiries use wakeups of at most seven days. Delivery may occur after the recorded expiry.

Startup and once-a-minute recovery enqueue at most 100 expired polls with missing notifications per pass. The scan advances through the backlog even while earlier tasks are pending or failed. It covers existing polls, downtime, missed post-commit dispatches, and exhausted task retries; recovery itself does not create notifications. Delayed messages do not count toward ready-queue backpressure. Transient task errors use exponential backoff for up to three attempts, then recovery can try again. Shutdown lets running transactions finish; interrupted work remains recoverable. Stop older nodes before upgrading and run compatible task definitions on every node.

The shared database stores at most 100 pending poll messages, including delayed wakeups and retries, with one wakeup per poll for new schedules. Repeated votes and expiry updates coalesce: an earlier expiry advances the existing wakeup; a later expiry is reloaded when that wakeup runs. At the limit, an earlier wakeup replaces the latest one. Deferred or displaced polls are recovered after expiry. Bursts can also defer admission when its database lock cannot be acquired within one second. These limits apply across nodes, rather than per worker. They bound stored work, not notifications or running handlers.

Delayed poll wakeups use the queue’s five-second polling interval rather than allocating a timer in each worker. Recovery still pauses at 200 shared ready messages. The limit of 100 matches one recovery page; it is not a throughput guarantee. Stop all older producers before upgrading. Legacy queued messages count toward the limit, but an existing backlog above 100 drains gradually; new admissions do not increase that backlog.

The hollo.poll-notifications.queue logger reports capacity deferrals, displaced wakeups and lock timeouts at warning level, at most once per reason per minute per process. Inspect storage pressure with the query below. Enqueue counters include coalesced and deferred attempts, so they do not measure stored messages.

SELECT count(*) AS queued,
       count(*) FILTER (WHERE created + delay <= CURRENT_TIMESTAMP) AS ready,
       count(*) FILTER (WHERE created + delay > CURRENT_TIMESTAMP) AS delayed,
       count(*) FILTER (
         WHERE message->>'type' = 'task'
           AND message->>'taskName' = 'hollo.poll-notification.v1'
       ) AS pending_polls
FROM hollo_task_message_v1;

Docker Compose setup

Here’s an example compose.yaml for running separate web and worker nodes:

services:
  db:
    image: postgres:17-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: hollo
      POSTGRES_PASSWORD: password
      POSTGRES_DB: hollo
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

  web:
    image: ghcr.io/dahlia/hollo:latest
    restart: unless-stopped
    depends_on:
      - db
    ports:
      - "3000:3000"
    environment:
      - NODE_TYPE=web
      - DATABASE_URL=postgresql://hollo:password@db/hollo
      - SECRET_KEY=${SECRET_KEY}
      - DRIVE_DISK=fs
      - FS_STORAGE_PATH=/data/storage
      - STORAGE_URL_BASE=https://hollo.example.com/assets
      - BEHIND_PROXY=true
    volumes:
      - ./data/storage:/data/storage

  worker:
    image: ghcr.io/dahlia/hollo:latest
    restart: unless-stopped
    depends_on:
      - db
    environment:
      - NODE_TYPE=worker
      - DATABASE_URL=postgresql://hollo:password@db/hollo
      - SECRET_KEY=${SECRET_KEY}
      - DRIVE_DISK=fs
      - FS_STORAGE_PATH=/data/storage
      - STORAGE_URL_BASE=https://hollo.example.com/assets
    volumes:
      - ./data/storage:/data/storage

Note

• Both web and worker nodes need access to the same storage volume
• The web node is exposed on port 3000; worker nodes don’t need port exposure
• You can run multiple worker nodes by adding more services

Scaling workers

To run multiple worker nodes, add more worker services:

services:
  # ... db and web services ...

  worker-1:
    image: ghcr.io/dahlia/hollo:latest
    restart: unless-stopped
    depends_on:
      - db
    environment:
      - NODE_TYPE=worker
      # ... other environment variables ...

  worker-2:
    image: ghcr.io/dahlia/hollo:latest
    restart: unless-stopped
    depends_on:
      - db
    environment:
      - NODE_TYPE=worker
      # ... other environment variables ...

PostgreSQL’s LISTEN/NOTIFY ensures that each message is processed by only one worker.

Manual installation

For manual installations, you can run separate processes using different NODE_TYPE values.

Web node

NODE_TYPE=web pnpm prod

This starts only the web server on the configured port.

Worker node

NODE_TYPE=worker pnpm prod
# or
pnpm worker

This starts the Fedify message queue, import worker and cleanup worker without the web server.

Caution

When running NODE_TYPE=web, ensure at least one NODE_TYPE=worker node is running. Otherwise, incoming activities won’t be processed and outgoing posts won’t be delivered.

Systemd example

If you’re using systemd, create separate service files:

/etc/systemd/system/hollo-web.service

[Unit]
Description=Hollo Web Server
After=network.target postgresql.service

[Service]
Type=simple
User=hollo
WorkingDirectory=/opt/hollo
Environment="NODE_TYPE=web"
EnvironmentFile=/opt/hollo/.env
ExecStart=/usr/bin/pnpm prod
Restart=on-failure

[Install]
WantedBy=multi-user.target

/etc/systemd/system/hollo-worker.service

[Unit]
Description=Hollo Worker
After=network.target postgresql.service

[Service]
Type=simple
User=hollo
WorkingDirectory=/opt/hollo
Environment="NODE_TYPE=worker"
EnvironmentFile=/opt/hollo/.env
ExecStart=/usr/bin/pnpm worker
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then enable and start both services:

sudo systemctl enable hollo-web hollo-worker
sudo systemctl start hollo-web hollo-worker

Monitoring

Check worker status

For Docker Compose:

# View web node logs
docker compose logs -f web

# View worker node logs
docker compose logs -f worker

For systemd:

# View web node logs
sudo journalctl -u hollo-web -f

# View worker node logs
sudo journalctl -u hollo-worker -f

Worker log messages

When a worker node starts, you should see:

Worker started (Fedify activity/task queues + Job recovery)

Watch for messages about processing activities, imports, cleanups, and remote replies scraping to confirm the worker is functioning correctly.

Troubleshooting

Web node works but activities aren’t processed

Problem: You can access the web UI, but incoming activities (follows, likes, posts) aren’t being processed.

Solution: Ensure at least one worker node is running with NODE_TYPE=worker.

Worker node won’t start

Problem: Worker node exits with an error.

Solution: Check that:

• DATABASE_URL is correct and the database is accessible
• The database has the latest migrations applied
• Storage configuration (DRIVE_DISK, FS_STORAGE_PATH, etc.) is correct

Activities are processed slowly

Problem: There’s a delay in processing federation activities.

Solution: Add more worker nodes to process messages in parallel. PostgreSQL’s message queue will distribute work among all available workers.

Storage access errors

Problem: Worker nodes can’t access uploaded files.

Solution: Ensure:

• All nodes (web and worker) have access to the same storage
• For filesystem storage: the storage volume is mounted on all nodes
• For S3 storage: all nodes have the same S3 credentials

Performance considerations

• Web nodes: Lightweight, can run with minimal resources (512MB-1GB RAM)
• Worker nodes: More resource-intensive, especially during high federation activity (1GB-2GB RAM recommended)
• Database: Shared by all nodes; ensure it has adequate resources (2GB+ RAM recommended for busy instances)
• Concurrency: Each worker processes up to 10 ActivityPub messages and four application tasks concurrently on separate queues. Remote replies scraping uses at most two of those four task slots. These limits apply per worker process; adding workers increases total concurrency.

Best practices

• Start with NODE_TYPE=all (default) until you experience performance issues
• Monitor resource usage to determine when separation is needed
• Run at least one dedicated worker node when using NODE_TYPE=web
• Use a reverse proxy (nginx, Caddy) in front of web nodes for load balancing
• Keep storage accessible to all nodes (shared volume or S3)
• Monitor logs from all nodes for errors and performance issues