Skip to main content

Operations Guide

This guide covers day-2 operations of a running Open ACE deployment: health probes and monitoring endpoints, the scheduler worker, data collection, TLS certificates, data retention and compliance, and logs and alerts. For initial installation see DEPLOYMENT.md; for cluster deployment see KUBERNETES.md; for reverse proxy see NGINX.md.

Health and Monitoring Endpoints​

All endpoints below are served by the web application on port 19888 (defined in app/__init__.py).

EndpointPurposeReturnsFailure behavior
/livezLiveness probe{"status": "alive", "timestamp": ...}Always 200 while the process can respond. Does NOT check dependencies, to avoid restart storms.
/readyzReadiness probe{"status": "ready" | "not_ready", "checks": {...}, "action": ...}503 if any critical check fails.
/metricsPrometheus metricsPrometheus text format (via prometheus_flask_exporter, group_by_endpoint, latency buckets 0.01–5 s)503 JSON fallback if prometheus_flask_exporter is not installed.
/security-statusSecurity baselineBaseline check results plus security_mode (mode / source / explicit), pilot_metadata, and migration_status503 when the baseline reports unhealthy or security mode resolution errors.
/healthDeprecatedDelegates to /readyz, then adds deprecated: true, a migration message, and version (git commit)Same as /readyz. Use /livez + /readyz instead.

/readyz runs nine checks and reports each under checks:

  • security_mode — must be explicitly set (OPENACE_SECURITY_MODE); pilot metadata in production mode produces a warning
  • database — probe connection with 2 s timeout (dedicated connection, not from the pool)
  • schema_version — Alembic head check via SchemaCompatibilityService (PostgreSQL production only; SQLite reports skipped; an active emergency bypass forces 503)
  • config_dir, workspace_dir — exist and are writable/readable
  • encryption_keys — encryption key registry initialized
  • init_status — application initialization completed
  • ssh_sync — no SSH sync failure warning file present (see Logs and Alerts)
  • frontend_build — management UI build artifacts present (fails readiness only in production mode)

When not ready, the response includes an action hint such as running alembic upgrade head.

Which endpoint a Kubernetes probe should use​

k8s/deployment.yaml wires the web pod as follows (port http = 19888):

          livenessProbe:
            httpGet:
              path: /livez
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 10
            failureThreshold: 3
          # Startup probe for slow initialization (Issue #2186)
          startupProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 10
            failureThreshold: 60  # 300s max startup time
  • Liveness → /livez (process alive only; dependency failures must not trigger pod restarts)
  • Readiness → /readyz (dependency failures remove the pod from Service endpoints)
  • Startup → /readyz with failureThreshold: 60 (up to 300 s for first boot and migrations)

The web pod also carries Prometheus scrape annotations (prometheus.io/scrape: "true", port 19888, path /metrics); with multiple gunicorn workers the deployment sets PROMETHEUS_MULTIPROC_DIR for multi-process metric aggregation. The Docker Compose healthcheck for the open-ace service polls http://localhost:19888/readyz (30 s interval, 3 retries, 30 s start period).

Scheduler Service​

Background work (data fetch, quota enforcement, autonomous workflows, alert compensation, scheduler health monitor, SSO cleanup) is decoupled from web workers (Issue #2187). The split is controlled by SCHEDULER_MODE:

  • SCHEDULER_MODE=scheduler — the process starts all background schedulers with database leader election (the scheduler_leaders table guarantees a single active leader)
  • SCHEDULER_MODE=web or unset — no schedulers are started (web worker)
  • Development python3 server.py runs web and schedulers in one process

Docker Compose​

The scheduler is an opt-in Compose profile (docker-compose.yml, service scheduler):

  scheduler:
    image: ${IMAGE_NAME:-openace/open-ace:latest}
    container_name: open-ace-scheduler
    restart: unless-stopped
    profiles:
      - scheduler
    environment:
      - FLASK_ENV=production
      - PYTHONUNBUFFERED=1
      - SCHEDULER_MODE=scheduler
      - SECRET_KEY=${SECRET_KEY:-}
      - OPENACE_ENCRYPTION_KEY=${OPENACE_ENCRYPTION_KEY:-}
      - DATABASE_URL=postgresql://${DB_USER:-ace}:${DB_PASSWORD:-dev-password-change-in-production}@postgres:5432/${DB_NAME:-ace}
      # Issue #3095: Pass DB_PASSWORD as standalone env var for security baseline check.
      - DB_PASSWORD=${DB_PASSWORD:-}
      - SCHEDULER_HEARTBEAT_INTERVAL=${SCHEDULER_HEARTBEAT_INTERVAL:-10}
      - SCHEDULER_HEARTBEAT_TIMEOUT=${SCHEDULER_HEARTBEAT_TIMEOUT:-60}
      - SCHEDULER_LOCK_TIMEOUT=${SCHEDULER_LOCK_TIMEOUT:-1800}
      - SCHEDULER_METRICS_PORT=9090
      # #3237: must match the app service exactly.
      - OPENACE_AGENT_STATE_ROOT=/var/lib/openace/agent-state

Enable it with docker compose --profile scheduler up -d. It shares the config-data, workspace-data, and agent-state volumes with the app service — the shared agent-state volume is deliberate (#3237), because purge routes run in the web container while the scheduler writes the transcripts. Its health check polls http://localhost:9090/metrics.

Heartbeat and election parameters (read in app/services/leader_election.py):

VariableDefaultMeaning
SCHEDULER_HEARTBEAT_INTERVAL10 (seconds)How often the leader refreshes its heartbeat row
SCHEDULER_HEARTBEAT_TIMEOUT60 (seconds)Heartbeat age after which leadership is considered lost
SCHEDULER_LOCK_TIMEOUT1800 (seconds)Distributed lock expiration
SCHEDULER_METRICS_PORT9090Port of the scheduler's own metrics HTTP server

The scheduler worker (python -m app.scheduler_worker) starts its own metrics server on port 9090 with three endpoints: /livez (alive), /health (alive plus is_leader from leader election), and /metrics (Prometheus). At startup it validates explicit security mode, schema version (REQUIRE_HEAD), scheduler tables, and data-fetch configuration drift (FETCH_USE_SUDO wrapper/sudoers, Issues #2543/#3145 — with critical drift it marks fetch degraded and skips collection jobs rather than running doomed tasks).

Kubernetes​

k8s/scheduler-deployment.yaml runs open-ace-scheduler as the counterpart: 2 replicas (main + backup, only one active leader), command: ["python", "-m", "app.scheduler_worker"], the same heartbeat env values (10/60/1800/9090), probes against /livez and /health on port metrics (9090), a PodDisruptionBudget with minAvailable: 1, a PriorityClass (open-ace-scheduler, value 1000000) to prevent eviction, pod anti-affinity, and the shared open-ace-data PVC for /workspace, agent state, and config — OPENACE_AGENT_STATE_ROOT must match the web deployment exactly.

On bare metal, reference units are scripts/openace.service (web, no schedulers) and scripts/openace-scheduler.service (SCHEDULER_MODE=scheduler, runs all background services via leader election).

Data Collection (cron)​

Usage data is collected from CLI tool data directories by per-tool scripts in scripts/:

ScriptCollects
scripts/fetch_claude.pyClaude Code sessions
scripts/fetch_codex.pyCodex CLI sessions
scripts/fetch_qwen.pyQwen Code sessions
scripts/fetch_openclaw.pyOpenClaw sessions
scripts/fetch_zcode.pyZCode CLI databases

Two collection paths exist:

  1. In-app scheduler (default in container deployments): the data fetch scheduler runs the fetch scripts as subprocesses every 300 s by default (minimum 60 s). Configure with DATA_FETCH_INTERVAL / DATA_FETCH_ENABLED environment variables or the data_fetch section of config.json (environment variables win). In multi-user containers the scripts run through /usr/local/bin/openace-fetch-wrapper with sudoers rules (FETCH_USE_SUDO=true, generate rules with scripts/generate-sudoers.sh); in the default single-user container FETCH_USE_SUDO=false and the venv Python is used directly.
  2. Host cron (bare-metal deployments): the classic crontab from the deployment guide:
30 0 * * * cd /path/to/open-ace && python3 scripts/fetch_claude.py >> logs/cron.log 2>&1
35 0 * * * cd /path/to/open-ace && python3 scripts/fetch_qwen.py >> logs/cron.log 2>&1
40 0 * * * cd /path/to/open-ace && python3 scripts/fetch_openclaw.py >> logs/cron.log 2>&1

Add fetch_codex.py / fetch_zcode.py lines the same way if those tools are in use. Manual runs support --days N to backfill, e.g. python3 scripts/fetch_claude.py --days 7.

Central server plus remote collectors: scripts/upload-to-central/​

For distributed environments, remote machines run the incremental sync daemon instead of cron. Deployment is one command (from scripts/upload-to-central/README.md):

./deploy-remote.sh \
    --host user@remote-host \
    --server http://CENTRAL_SERVER:19888 \
    --auth-key YOUR_AUTH_KEY \
    --hostname remote-hostname

For the local machine use deploy.sh with the same --server / --auth-key / --hostname options. Useful options: --interval (sync interval in seconds, default 300), --user, --install-dir (default ~/upload-to-central), --uninstall.

Operational notes:

  • The daemon installs a systemd unit upload-to-central; check sudo systemctl status upload-to-central and sudo journalctl -u upload-to-central -f
  • Manual sync: ./upload.sh; forced full re-upload: ./upload.sh --full
  • Sync state lives in ~/.open-ace/sync_state.json (per-host last_sync_time, last_upload, total_uploaded); incremental sync only uploads new messages
  • The daemon POSTs to /api/upload/batch on the central server with the X-Auth-Key header — the key must match server.upload_auth_key in the central server's ~/.open-ace/config.json
  • Remote deployment requires SSH access; installing the systemd service requires root

TLS Certificates​

  • Compose / bare metal: TLS terminates at the nginx reverse proxy. NGINX.md shows the full configuration: listen on 443 with ssl_certificate /path/to/ssl/fullchain.pem and ssl_certificate_key /path/to/ssl/privkey.pem, TLSv1.2/1.3 only, and an HTTP→HTTPS redirect on port 80. The repository does not ship a certbot configuration; obtain certificates with certbot (webroot or standalone) for your domain and point nginx at the Let's Encrypt paths. Renewal is automated by certbot's own timer — make sure the renewal hook reloads nginx so the new certificate is picked up.
  • Kubernetes: the Ingress in k8s/service.yaml is pre-configured for cert-manager with the letsencrypt-prod ClusterIssuer (annotation cert-manager.io/cluster-issuer: "letsencrypt-prod", TLS secret open-ace-tls, ssl-redirect: "true"). cert-manager must be installed before applying the manifests:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

After that, issuing and renewing the certificate is handled entirely by cert-manager; there is no manual renewal step.

Data Retention and Compliance​

  • k8s/configmap.yaml declares AUDIT_LOG_RETENTION_DAYS: "90" and DATA_RETENTION_DAYS: "365" (lines 40–41). These values document the intended retention horizon and match the built-in defaults (tenant settings default to 90/365 days; the response-time cleaner defaults to 90 days raw / 365 days aggregated). Note: no background job in the current code base reads these two environment variables directly — runtime retention is driven by the mechanisms below.
  • Retention rules (app/modules/compliance/retention.py, DataRetentionManager), per data type:
Data typeDefaultAction
audit_logs90 daysdelete
quota_alerts30 daysdelete
sessions7 daysdelete
sso_sessions1 daydelete
usage_data365 daysarchive
messages90 daysanonymize
user_activity365 daysarchive
  • Admin API (blueprint /api/compliance, admin only): GET/PUT /api/compliance/retention/rules, POST /api/compliance/retention/cleanup (supports ?dry_run=true), and GET /api/compliance/retention/history|storage|status. There is no built-in scheduler that fires retention automatically — run or schedule cleanup through the API; the compliance status check flags instances whose last cleanup is older than a week. Every run is recorded in the retention_history table.
  • Schema groundwork (Issue #2188 Phase 1): migration 20260803_002_create_retention_tables created retention_policies, retention_executions, legal_holds, retention_evidence, archive_files, and recycle_bin. These persistent policy/execution tables (including legal holds and the recycle bin) are schema preparation; treat legal holds and recycle bin as not yet wired to runtime automation and verify your version's coverage before relying on them operationally.
  • Audit log sync: scripts/openace-ssh-sync prunes its own audit log (/var/log/openace/ssh-sync.log) after 30 days.

Logs and Alerts​

  • Log level: LOG_LEVEL: "INFO" is set in k8s/configmap.yaml. The application factory currently fixes the level in code (logging.basicConfig(level=logging.INFO) in app/__init__.py; the scheduler worker also logs at INFO), so treat INFO as the effective level.
  • Container logs: docker compose logs -f open-ace / docker compose logs -f scheduler. All Compose services use the json-file driver with max-size: 10m, max-file: 3 rotation. The application also writes to /app/logs inside the container, bind-mounted to ./logs on the host in Compose (an emptyDir in Kubernetes).
  • Alert notification history: the alerts_history table records every outgoing notification (alert type, severity, message, channels, recipients, status sent/failed, sent_at).
  • Alert failure queue: when alert creation fails, the alert compensation worker (part of the scheduler service) retries from a queue. Inspect with GET /api/alerts/failure-queue and force a processing pass with POST /api/alerts/failure-queue/retry (admin). Scheduler status endpoints GET /api/schedulers/data-fetch and GET /api/schedulers/quota-enforcement report the corresponding background jobs. The full Alerts API is documented in API.md.
  • SSH sync failures affect readiness (Issue #2328): when secure SSH sync fails, the entrypoint writes /var/log/openace/ssh-sync-failure.warning (human-readable, with remediation hints) and .../ssh-sync-failure.json (structured). While the warning file exists, the ssh_sync check in /readyz fails and the pod reports not ready. Remove the file after fixing the cause; details are in ssh-sync.log in the same directory.