Production Topology & Scaling

For the DevOps team. Describes production as it runs from October 2026 (scale-out milestone M1), the deploy pipeline, and the planned next step — several API copies behind nginx — which is documented here but not yet applied.

Current production setup

One Docker Compose project on hiperteam-srv1 (/home/deploy/hiperteam, Compose v5). The compose file docker-compose.prod.yml and nginx/nginx.conf live only on the server — they are not in the repository.

Container	Image	Role	Does
api	hiperteam-api	ROLE=api	All user traffic: HTTP API and WebSockets (/socket.io)
worker	hiperteam-api	ROLE=worker	Queue workers (lead/account/opportunity/product imports, account enrichment) and the Facebook lead consumer
scheduler	hiperteam-api	ROLE=scheduler	The scheduled jobs (@Cron) and one-off boot tasks. Run exactly one.
redis	redis:7-alpine	—	Job queues, locks, rate limits, socket.io fan-out
rabbitmq	rabbitmq:3.12-management-alpine	—	Facebook lead messages
nginx	nginx:alpine	—	TLS, static web app, proxies /api/ and /socket.io/ to api

PostgreSQL runs on a separate host, hiperteam-db1 (database hiperteam_master).

Process roles

All three app containers run the same image; the ROLE environment variable decides what each process does.

ROLE	HTTP + WebSockets	Queue workers + message consumers	Scheduled jobs + boot tasks
all (default, also when unset)	✔	✔	✔
api	✔	—	—
worker	(health only)	✔	—
scheduler	(health only)	—	✔

Combinations such as ROLE=api,worker are allowed. Every process listens on port 3000 inside its container so /api/health/* works for its health check; only api containers should receive user traffic. The startup log line shows the role, e.g. API running on http://localhost:3000 — role: worker.

Guarantees that make the split safe:

• Scheduled jobs never run twice at once. Every @Cron job takes a Redis lock (cron-lock:<job>) for the duration of the run, so a second scheduler or a slow run overlapping the next tick is skipped, not duplicated.
• Real-time events reach every user. socket.io uses the Redis adapter, so an event raised in any process (for example import progress from the worker) reaches the user on whichever api copy holds their connection.
• Queues work from any process. Every role can add jobs; only worker processes run them.
• Scheduled work runs per workspace. On each tick the scheduler only queues one tenant-jobs job per active workspace; workers run them in parallel. A job type never runs twice at once for the same workspace.
• Workflows run on workers. Saving a record only records the workflow run and queues it; a worker executes it. A run whose worker stops mid-way is marked failed — Interrupted (never restarted from the top, its actions may already have created records) and can be retried from the failed step on the workflow's Runs tab.
• Outgoing messages are paced and retried. Notification emails, SMS, WhatsApp and chat messages go through the notification-delivery queue: temporary failures (mail relay throttling, timeouts, HTTP 429/5xx) are retried with backoff for about 15 minutes, permanent ones fail at once. Every email path — including direct sends such as invites — is limited per sending account across all processes.

Background job settings

All optional; set in .env, applied on the next restart of worker (and api for the send rate).

Variable	Default	What it does
TENANT_JOBS_CONCURRENCY	8	Per-workspace scheduled jobs one worker runs at once
WORKFLOW_RUN_CONCURRENCY	10	Workflow runs one worker executes at once
NOTIFICATION_DELIVERY_CONCURRENCY	10	Queued messages one worker sends at once
NOTIFICATION_DELIVERY_ATTEMPTS	6	Tries per message (30 s, 1, 2, 4, 8 min apart)
OUTBOUND_SMTP_PER_SECOND	5	Emails per second per SMTP / SES account, across all processes. The current relay rejects more than ~10/s. Raise only for a provider with a higher limit.

Shared storage

api and worker must mount the same volume at /app/uploads:

    volumes:
      - uploads:/app/uploads

The API stores uploaded import files there and the worker reads them; the worker writes the "failed rows" file there and the API serves it. Without the shared volume imports fail. (Documents, attachments and avatars are in object storage, not on this volume.)

Redis requirements

• maxmemory-policy must be noeviction. With an LRU policy Redis silently deletes keys when full — queued jobs, locks and rate-limit counters included. BullMQ logs IMPORTANT! Eviction policy is allkeys-lru. It should be "noeviction" when this is wrong.
• Current setting (compose command:): --maxmemory 256mb --maxmemory-policy noeviction.
• The Redis password may contain shell-special characters. To run a Redis command, go through the API container instead of redis-cli -a inside sh -c:

docker compose -f docker-compose.prod.yml exec -T api node -e "
const R = require('ioredis');
const r = new R({ host: process.env.REDIS_HOST, port: +process.env.REDIS_PORT, password: process.env.REDIS_PASSWORD });
r.config('GET', 'maxmemory-policy').then((x) => { console.log(x); r.quit(); });
"

Reverse proxy

nginx/nginx.conf proxies /api/ and /socket.io/ to http://api:3000. nginx resolves the api name only when it loads, so after the api container is recreated it must be reloaded (nginx -s reload, no dropped connections). The deploy does this automatically.

TRUST_PROXY=1 must be set in .env so per-IP rate limits see the visitor's address (from X-Forwarded-For) rather than nginx's.

Deploy pipeline

.github/workflows/deploy.yml, on every push to main:

Detect Changes → Test API / Test Web / E2E smoke tests → Build images → Deploy API / Deploy Web

1. Detect Changes — which of API, web, docs, CI changed. The root package-lock.json counts as both API and web.
2. E2E smoke tests — fresh Postgres 18 + Redis on the runner, the real API and web app, then 71 browser tests (sign-in + every page) and 22 API tests (tenant isolation, permissions, tenant manager, health). Any failure stops the deploy. On failure the run has an e2e-report artifact (Playwright report, traces, server logs). Edits to workflow files run the tests but never deploy.
3. Build images (main only, after the tests pass) — builds the API image from apps/api/Dockerfile and a small alpine image holding the built web app, tags both with the commit SHA and pushes them to ghcr.io/contegris-pvt-ltd/hiperteam-api / hiperteam-web (private; access inherited from the repository).
4. Deploy API — over SSH, with the job's short-lived token (logged out right after the pull):
   1. pull the image and tag it hiperteam-api (the compose api service has build: but no image:, so Compose names its image hiperteam-api);
   2. run migrations first from the new image in a one-off container (docker compose run --rm --no-deps api npm run migrate:run:prod and migrate:tenants:prod), so new code never starts against an old schema;
   3. restart api (and api2 when it exists) one at a time with up --wait, reloading nginx after each;
   4. restart worker and scheduler if defined;
   5. keep the last 5 API images, prune the rest.
5. Deploy Web — pull the web image, copy its /dist into frontend-build/ (served by nginx), restart nginx. Keeps the last 3.

Nothing is built on the production server.

Rollback

cd ~/hiperteam
docker images ghcr.io/contegris-pvt-ltd/hiperteam-api          # last 5 deployed commits
docker tag ghcr.io/contegris-pvt-ltd/hiperteam-api:<sha> hiperteam-api
docker compose -f docker-compose.prod.yml up -d --no-build api worker scheduler
docker compose -f docker-compose.prod.yml exec -T nginx nginx -s reload

Migrations are additive, so rolling the code back without rolling the database back is safe.

Master-schema migrations

Tenant migrations run automatically in the deploy. Master-schema migrations (apps/api/src/scripts/master-migrations.ts — plans, usage, tenant manager tables) cannot: the API's database user intelli_hiper_app has no CREATE on schema master, so the deploy logs a warning and skips them. Apply them as postgres on hiperteam-db1:

sudo -u postgres psql -d hiperteam_master -v ON_ERROR_STOP=1 -f /tmp/<migration>.sql

Each such migration ends by recording itself in master.master_migrations and granting intelli_hiper_app what it needs.

Operational commands

All from ~/hiperteam on hiperteam-srv1.

Task	Command
Containers and health	docker compose -f docker-compose.prod.yml ps
Role of each process	docker compose -f docker-compose.prod.yml logs api worker scheduler | grep "role:"
Imports being processed	docker compose -f docker-compose.prod.yml logs worker | grep "import job" | tail
Scheduled jobs in the last day	docker compose -f docker-compose.prod.yml logs scheduler --since 24h
Create a tenant-manager admin	docker compose -f docker-compose.prod.yml exec -T api npm run platform:admin:prod -- create --email you@company.com --name "Name"
Reset that admin's password / authenticator	… npm run platform:admin:prod -- reset-password --email … / reset-mfa
Make older uploaded files private (one-off; dry run without --apply)	… npm run files:privatize:prod -- --apply

Logs are JSON in production; each line carries requestId, tenant and userId (queue jobs: job:<queue>:<id>). Health: GET /api/health/live (process up) and GET /api/health/ready (database and Redis reachable, otherwise 503).

Load testing

The manual Load test workflow (Actions → Load test → Run workflow) starts a fresh database and Redis on the runner, the API in production mode, creates N synthetic workspaces (npm run seed:synthetic) and runs the k6 script apps/api/test/load/crm-load.js (browse 60 %, write 25 %, search 10 %, reports 5 %). The run page shows p50 / p95 / p99 per kind and per endpoint against the targets (errors < 0.5 %, p95 read < 300 ms, write < 600 ms, search < 500 ms, report < 1500 ms). It never touches production.

GitHub runners vary between runs; compare several runs before and after a change rather than single runs.

Planned: several API copies (M1 stage 2)

Status: not applied. Scheduled for after the scaling project. The application already supports it — no code change is needed; the deploy already restarts api2 one at a time when it exists. What remains is server configuration.

Benefits: zero-downtime deploys (one copy always serves), and capacity for more concurrent users.

Before starting: free -h — each API copy needs roughly 250–300 MB.

Order matters: nginx refuses a config that names api2 before the api2 container exists.

Step 1 — compose: add api2, sharper health checks

Change the api health check and add api2 (same image, no host port):

  api:
    # …existing build / ports / env_file…
    environment:
      - ROLE=api
    volumes:
      - uploads:/app/uploads
    stop_grace_period: 30s
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/health/ready"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s

  api2:
    image: hiperteam-api
    env_file:
      - .env
    environment:
      - ROLE=api
    volumes:
      - uploads:/app/uploads
    depends_on:
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    restart: always
    stop_grace_period: 30s
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/health/ready"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s

docker compose -f docker-compose.prod.yml up -d --no-build --wait api api2
docker compose -f docker-compose.prod.yml exec -T nginx nginx -s reload

Users are still served by api only at this point.

Step 2 — nginx: send traffic to both

Back up first: cp nginx/nginx.conf nginx/nginx.conf.bak-stage1.

Inside http { … }, above the first server {:

    upstream crm_api {
        server api:3000  max_fails=2 fail_timeout=10s;
        server api2:3000 max_fails=2 fail_timeout=10s;
        keepalive 32;
    }

    # A browser always reaches the same copy (socket.io's polling fallback needs it)
    upstream crm_socket {
        ip_hash;
        server api:3000  max_fails=2 fail_timeout=10s;
        server api2:3000 max_fails=2 fail_timeout=10s;
    }

In the hiperteam.intellicon.io 443 server, replace the two locations:

        location /api/ {
            proxy_pass http://crm_api;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout 120s;
            # A copy refusing connections (e.g. restarting) → the other one.
            # Requests already sent are never repeated.
            proxy_next_upstream error timeout http_502 http_503;
            proxy_next_upstream_tries 2;
        }

        location /socket.io/ {
            proxy_pass http://crm_socket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout 120s;
        }

docker compose -f docker-compose.prod.yml exec -T nginx nginx -t        # must report "test is successful"
docker compose -f docker-compose.prod.yml exec -T nginx nginx -s reload

Never reload after a failed nginx -t.

Step 3 — verify

docker compose -f docker-compose.prod.yml logs --since 2m api  | grep -c '"context":"HTTP"'
docker compose -f docker-compose.prod.yml logs --since 2m api2 | grep -c '"context":"HTTP"'

Both counts grow during normal use. Sign-in, pages, live notifications and import progress all work. Optional failover test: stop api2, keep using the CRM, start api2, reload nginx.

Undo

1. cp nginx/nginx.conf.bak-stage1 nginx/nginx.conf and reload nginx.
2. Remove api2 from the compose file, then docker compose -f docker-compose.prod.yml up -d --no-build --remove-orphans api.

Notes

• User sessions are cached per copy for up to 10 seconds, so a deactivation or role change reaches every copy within 10 seconds.
• Each copy holds its own database connection pool (default 10). With 2 API copies + worker + scheduler that is about 40 connections — within PostgreSQL's default of 100.