Skip to main content
This page is for the full inference system. The API Gateway always runs in your AWS, GCP, or Azure account via Ryvn. The remaining choice is where OrangeLine GPUs run, and how you watch and update the stack. For OrangeLine-only, there is no gateway placement to configure: pull a new image tag when you want an update. See Methods.

​
Gateway and GPU placement

The gateway sits close to your identity, networking, and observability stack. OrangeLine can run next to it or on any other GPU host that can run the Docker image. GPUs do not have to be in the same hyperscaler as the gateway.

​
Same cloud as the gateway

OrangeLine GPUs in the same AWS, GCP, or Azure account as the gateway. Best when you already have approved GPU capacity, want simpler private networking, and want inference traffic to stay inside one cloud boundary.

​
Somewhere else

OrangeLine on another hyperscaler, a NeoCloud (Nebius, Lambda, CoreWeave, and others), an inference platform (Together AI, Baseten, Modal), your own local cluster, or bare metal. The gateway calls those workers as model routes. Best when the gateway cloud does not have the GPUs you want, you already operate GPUs elsewhere, or you want to compare GPU homes without moving the gateway. If you have GPUs we have not named, we can work with that.

​
Secrets

Secrets follow customer policy. Common ones include gateway admin credentials, API key material, model-provider credentials, database and cache credentials, observability export credentials, registry access for OrangeLine, and TLS references. On the gateway environment, secrets stay in your cloud (for example a cloud secret manager synced into the cluster). OrangeLine hosts that are not in that account keep their own registry and runtime credentials.

​
Monitoring and logging

Map gateway availability, request rate, latency, error rate, and route health into the stack you already run (Datadog, CloudWatch, Grafana, and similar). Pair that with OrangeLine GPU utilization, queue depth, and token throughput. Keep prompts, completions, secrets, and API keys out of shared logs. Define retention, redaction, and what you are willing to share for support before production. Ryvn shows installation status and logs for the gateway environment. You do not have to send production inference data to Ryvn. See Ryvn.

​
Upgrades

​
Gateway (Ryvn)

  1. Subconscious publishes a gateway release.
  2. You review release notes and evidence.
  3. Ryvn applies the update according to approvals, maintenance windows, and release channels.
  4. Health checks gate the rollout. Rollback is a first-class operation.
Do not maintain a parallel Helm process for the gateway.

​
OrangeLine

On same-cloud GPUs that Ryvn also deploys, follow the same approval flow. On other GPU hosts, pull a new OrangeLine image tag and roll the workers, then confirm gateway routes still succeed.