# Platform replicas with shared services `containers/compose.distributed.yaml` adds a platform replica to an existing Kubernetes-backed deployment. Each host runs an AstraBox API process. The replicas use the same PostgreSQL database, OpenSandbox client-pool Redis, OpenSandbox Lifecycle API, LiteLLM and channel gateway. Kubernetes schedules the sandboxes; AstraBox does not select a machine for each user request. This overlay assembles application connections. It does not provision those services, migrate an existing deployment, install a load balancer, or make a single service highly available. Use it for a fresh deployment with explicit shared signing keys, or to add replicas to a deployment that already uses those same explicit keys. Converting automatically derived callback keys requires a maintenance and credential rotation plan; it is not a seamless configuration-only migration. ## Prepare the shared services - Use the same AstraBox database, including SessionStore and encrypted Vault records. Both hosts need private network access to PostgreSQL and Redis; a port bound to the first host's loopback is not reachable from the second. Restrict access to the participating hosts, not the public Internet. - Use one authenticated external OpenSandbox service connected to the same Kubernetes cluster and namespace. Both API hosts must reach the lifecycle endpoint and the sandbox endpoints it returns. - Carry the existing workspace settings into the environment file, because the overlay replaces the base Compose environment. Persistent workspaces use `ASTRABOX_STORAGE_PROVIDER`, `ASTRABOX_SANDBOX_WORKSPACE_VOLUME` and `ASTRABOX_WORKSPACE_MOUNTER_IMAGE`. For AWS EFS, configure `ASTRABOX_STORAGE_PROVIDER=aws_efs` and the same `ASTRABOX_EFS_FILE_SYSTEM_ID`, and install the official CSI driver. The overlay sets `ASTRABOX_WORKSPACE_STORAGE_TOPOLOGY=shared`, so the backing claim must be a shared filesystem that every eligible sandbox node can mount. See [workspace storage](deploy.md#where-conversation-workspaces-live) and [AWS EFS workspace storage](providers/aws-efs.md). Persistent workspaces are optional. - Give both API hosts a kubeconfig readable by container uid 999; the overlay requires it. The platform workspace-mount controller and EFS provider use Kubernetes even when the lifecycle service is external. Use the same cluster and namespace as OpenSandbox. Grant the kubeconfig identity the helper permissions listed in [workspace storage](deploy.md#where-conversation-workspaces-live); the EFS provider also reads the backing claim and its PersistentVolume. - Share one HTTPS LiteLLM service and its existing inference/administration credentials. Share one authenticated HTTPS channel gateway and its token; separate embedded gateways are not a shared messaging runtime. - Put the replicas behind an operator-managed HTTP/WebSocket/SSE ingress. Use the same public origin, identity configuration and Pod-reachable callback URL on every replica. Include the public, callback and private health-check hostnames in `ASTRABOX_ALLOWED_HOSTS`. Drain a replica before removing it from service; set the termination grace period above `ASTRABOX_SHUTDOWN_DRAIN_SECONDS`. ### OpenSandbox version boundary With OpenSandbox Server 0.2.3, use a single external lifecycle service with its own persistent SQLite store. This permits multiple AstraBox replicas and sandbox nodes; it does **not** provide lifecycle-control-plane high availability. Do not run independent SQLite-backed lifecycle replicas and assume that sharing Kubernetes also shares snapshot records. The [0.2.3 configuration reference](https://github.com/opensandbox-group/OpenSandbox/blob/c39b814/server/configuration.md#store) supports only SQLite for server-managed metadata. A different upstream version and its documented shared-store guarantees require separate acceptance. The bundled lifecycle process listens on container loopback. Publishing a host port alone does not expose it. Operate the external service using OpenSandbox's documented server configuration, including API-key authentication. When moving an existing lifecycle service, preserve its persistent metadata and stop the old writer before handing the store to the new service; do not copy a live SQLite database and run both writers. ## Configure each replica Create a protected, absolute-path environment file on each host, for example `/etc/astrabox/replica.env`, readable only by its operator. Use it for both Compose interpolation and container environment injection. Do not commit it or publish the rendered `docker compose config` output: both contain credentials. The shared portion contains these existing application settings: ```dotenv ASTRABOX_DB_URL=postgresql+asyncpg://astrabox:URL_ENCODED_PASSWORD@db.internal:5432/astrabox ASTRABOX_AGENT_PREWARM_REDIS_URL=redis://redis.internal:6379/0 ASTRABOX_SANDBOX_OPENAPI_BASE_URL=https://sandbox-control.example.com ASTRABOX_SANDBOX_API_KEY_SECRET_NAME=opensandbox-api-key OPENSANDBOX_API_KEY=REPLACE_WITH_SHARED_LIFECYCLE_API_KEY ASTRABOX_SANDBOX_SERVER_KUBE_API_SERVER=https://kubernetes.internal:6443 ASTRABOX_SANDBOX_SERVER_KUBE_NAMESPACE=opensandbox ASTRABOX_MCP_PROXY_BASE_URL=https://callbacks.example.com ASTRABOX_ALLOWED_HOSTS=astrabox.example.com,callbacks.example.com,api-a.internal,api-b.internal ASTRABOX_LITELLM_BASE_URL=https://llm.example.com ASTRABOX_CHANNEL_GATEWAY_BASE_URL=https://channels.example.com ASTRABOX_CHANNEL_GATEWAY_TOKEN=REPLACE_WITH_EXISTING_SHARED_GATEWAY_TOKEN ASTRABOX_VAULT_MASTER_KEY=REPLACE_WITH_EXISTING_SHARED_VAULT_KEY ASTRABOX_AUTH_SESSION_SECRET=REPLACE_WITH_EXISTING_SHARED_SESSION_KEY ASTRABOX_TRANSCRIPT_SIGNING_KEY=REPLACE_WITH_EXPLICIT_SHARED_TRANSCRIPT_KEY ``` The lifecycle API key reference uses `astrabox/secrets.py`'s secret provider, not the product's encrypted Vault records. Its environment provider maps a name to uppercase with hyphens replaced by underscores: `opensandbox-api-key` therefore reads `OPENSANDBOX_API_KEY`. Supply the same value to each replica, as shown above; this resolver currently reads only process environment. Sharing the product database alone does not make that named secret available. The container environment file carries this operator-chosen name without adding an application-specific credential variable. Every other setting the existing deployment uses also belongs in this file. Preserve the existing LiteLLM settings, including `ASTRABOX_LITELLM_API_KEY`, `ASTRABOX_LITELLM_SERVER_BASE_URL` and `ASTRABOX_LITELLM_ADMIN_URL` when the deployment sets them. When `ASTRABOX_LITELLM_API_KEY` is empty, each replica's entry point uses `LITELLM_MASTER_KEY` to confirm or create the sandbox model key derived from `ASTRABOX_AUTH_SESSION_SECRET`, and the replica does not start without it. Preserve the existing identity-provider settings, such as `ASTRABOX_WEB_IDENTITY` and the registered OIDC settings. This overlay does not configure an identity provider or install a second Casdoor instance. The overlay selects the local secret-store provider backed by the shared database. Its encryption key must be the existing deployment key, not a newly generated key. The encoded value in `/data/vault.key` can populate `ASTRABOX_VAULT_MASTER_KEY` without changing Vault encryption. Likewise, `/data/auth-session.key` can populate `ASTRABOX_AUTH_SESSION_SECRET` without changing the browser-session signing key. Callback signing has a different boundary. When no explicit transcript key is configured, a file-backed Vault key and its environment representation use different callback-key derivations. Copying the encoded Vault value into the environment therefore does not preserve existing callback credentials. The explicit transcript setting accepts a text secret; it does not import an arbitrary binary derived key. Do not base64-encode a derived key and assume the result reproduces it. An existing deployment without an explicit transcript key needs a coordinated rotation before using this overlay: stop admitting work, drain active work, retire prepared and resident sandbox runtimes carrying old callback credentials, and configure one explicit transcript key across all replicas before preparing replacement runtimes. Preserve the product database, SessionStore and workspace storage. Accept the replacement runtimes against the new callback credentials before reopening traffic. This overlay does not automate that rotation or refresh tokens inside existing sandboxes. Do not copy `/data` wholesale: each replica gets a separate local volume, not a database copy. Add the host-specific Compose inputs to that same file: ```dotenv ASTRABOX_REPLICA_ENV_FILE=/etc/astrabox/replica.env ASTRABOX_SERVER_IMAGE=registry.example.com/astrabox/server@sha256:REPLACE_WITH_IMAGE_DIGEST ASTRABOX_SERVER_BIND_IP=10.0.1.12 ASTRABOX_SERVER_HOST_PORT=8088 ASTRABOX_KUBECONFIG_HOST_PATH=/etc/astrabox/kubeconfig ASTRABOX_STATE_VOLUME=astrabox-api-b-state ``` Use the same immutable server image and sandbox images on every host. Only the bind address, local paths and local volume name vary between replicas. The `ASTRABOX_REPLICA_ENV_FILE` input is a Compose file path, not an application setting. Absolute paths avoid Compose's base-file-relative path resolution. The overlay fixes the container listener at `0.0.0.0:8000`; change only `ASTRABOX_SERVER_BIND_IP` and `ASTRABOX_SERVER_HOST_PORT` for its host publication. ## Start only the platform replica From the repository root, on each host: ```bash docker compose --env-file /etc/astrabox/replica.env \ -p astrabox-api-b \ -f containers/compose.yaml \ -f containers/compose.kubernetes.yaml \ -f containers/compose.distributed.yaml config --quiet docker compose --env-file /etc/astrabox/replica.env \ -p astrabox-api-b \ -f containers/compose.yaml \ -f containers/compose.kubernetes.yaml \ -f containers/compose.distributed.yaml up -d --no-deps --no-build server ``` Use Docker Compose 2.24.4 or newer for the `!override` merge tag. The distributed overlay must be last. Do not use `scripts/compose.sh` for this topology: that wrapper prepares credentials for a local database. Do not enable the `local-data-only` or `docker-runtime-only` profiles. The overlay removes local service dependencies and mounts no Docker socket or local database secrets. The entry point sees the external service URLs and does not start embedded OpenSandbox, LiteLLM or channel-gateway processes. Check `/healthz` and `/readyz` through each replica's private address, then register healthy replicas with the ingress. Validate the deployment with concurrent pool claims from both replicas, unique sandbox assignment, same-session history and streaming through either replica, and cross-replica stop/approval actions. Verify shared workspace persistence on different sandbox nodes separately. These checks exercise shared-service connections that a configuration render or a health check cannot verify.