Skip to main content
TrueFoundry provides different deployment options depending on your cloud provider, existing infrastructure and the components that you want to deploy. Before understanding the different deployment options, its important to read about the overall architecture here: control-plane architecture, compute-plane architecture, gateway-plane architecture.

​
Deployment Options

The key deployment options described above are:
This is a fully managed solution on TrueFoundry’s secure cloud infrastructure with enterprise-grade features. You don’t need to deploy any infrastructure on your end. This includes usage of only the AI Gateway modle and not AI Deployment.

TrueFoundry Managed SAAS

This is ideal for smaller, mid-size or entprises that want to use TrueFoundry AI Gateway without the operational overhead of self-hosting.
The key features and advantages of this deployment option are:
  1. Globally distributed gateway to minimize latency: TrueFoundry gateway is deployed in multiple regions of the world across multiple zones and multiple cloud providers to provide low latency and high availability. Learn more about our globally distributed infrastructure.
  2. Zero Overhead of maintenance: There is no overhead of maintaining infrastructure and you can get access to the latest features and improvements.
  3. Data is encrypted at rest and in transit.
  4. TrueFoundry Infrastructure is SOC2, ISO27001, GDPR, and HIPAA compliant
Installation: No Installation Needed.
This is a deployment of the AI Gateway plane on your own infrastructure. The control-plane is hosted by TrueFoundry and it doesn’t include the AI Deployment module. The AI Gateway exports the request-response data to an ingestion server running on the TrueFoundry control plane. From the ingestion server, the data can be stored in a blob storage bucket either on the TrueFoundry side or on your own infrastructure — depending on your data residency and compliance requirements. The control-plane stores the metrics and has access to the bucket containing the request-response data.

Gateway Plane and Data Storage on your own infrastructure

The key features about this mode of deployment are:
  1. LLM Traffic stays within your own premises: All LLM traffic stays within your own infrastructure and TrueFoundry doesn’t come into the live path of a request to LLM.
  2. Flexible data storage: The request-response data flows from the AI Gateway to the ingestion server on the TrueFoundry control plane, and can then be stored in a bucket on the TrueFoundry side or in a bucket on your own infrastructure. If stored on your end, you retain full ownership of the data. The data is stored in parquet format — so you can use it for analytics, debugging, and evaluation via Spark, DuckDB, Athena, or any tool of your choice.
  3. Management of Gateway: The AI Gateway needs to be managed on your end.
  4. TrueFoundry control plane has access to the bucket containing the data: Whether the data is stored on the TrueFoundry side or on your infrastructure, the control plane needs access to the bucket to let you browse request logs on the TrueFoundry dashboard. The control plane also compacts the logs and creates indexes to make querying faster.
If the bucket is on your infrastructure, you must grant the TrueFoundry control plane access to it. Without this access, you will not be able to browse logs or use the dashboard features.
When the data is stored on your infrastructure and you browse request logs in the TrueFoundry dashboard, the data is fetched from your blob storage — so you might incur egress charges from your cloud provider. The data might be cached temporarily in the control plane for faster queries. Egress charges also apply when the control plane compacts logs and creates indexes.
Installation: Deploy Gateway Plane
Host both the AI Gateway plane and the control plane on your own infrastructure.In this case, everything except the authentication server and analytics server, everything is hosted on your own infrastructure.

Control Plane, Gateway Plane and Data Storage on your own infrastructure

The only data sent to authentication/licensing server are the emails of the employees using the platform and the count of the requests flowing through the AI Gateway. To understand how SSO works with our central authentication server, refer to this page. This helps us keep track of the licenses and billing.
Installation: Deploy Control Plane and Gateway Plane
Deploy models/services (Compute Plane) on your own infrastructure while using managed AI Gateway.

Compute Plane + AI Gateway SAAS

Installation: Deploy Compute Plane.
Host the compute plane for model/services deployment, control-plane hosted by TrueFoundry. This doesn’t include the AI Gateway product.

Compute Plane only

Installation: Deploy Compute Plane (AWS, GCP, Azure, or generic).
Complete self-hosted for AI Deployment, full control over infra, no Gateway product.

Compute Plane + Control Plane

Installation: Deploy Control and Compute Plane (AWS, GCP, Azure, or generic control plane guides).
Self-host the control-plane, gateway plane and compute plane all in your own infrastructure.

Control Plane + Gateway Plane + Compute Plane

Installation: Deploy Control and Compute Plane.

​
Understanding the Installation Process

TrueFoundry software ships as a combination of OpenTofu/Terraform code and helm charts. Your exact deployment process will depend on your current state of infrastructure and the modules you want to deploy. Here’s a brief overview of how the compute-plane, gateway-plane and control-plane are deployed.

​
Compute Plane

This comprises of a Kubernetes cluster and few add-ons on the Kubernetes cluster as described here. If don’t have a Kubernetes cluster, TrueFoundry can also provide the OpenTofu/Terraform code to provision the cluster on AWS, GCP or Azure. The OpenTofu/Terraform code brings up the cluster, installs the add-ons on the cluster, and creates roles to grant the control-plane permission to DockerRegistry, SecretsManager, BlobStorage and the Kubernetes cluster itself. Each of the addons is installed as an argo-application on the Kubernetes cluster. You can find the list of addons in the compute plane overview page. Once the addons are installed on the cluster, the management of the addons (upgrade, value changes) are done from the TrueFoundry dashboard itself and terrraform is not required. OpenTofu/Terraform is only used to upgrade the cluster or make other changes in the infrastructure.

​
Gateway Plane

The AI Gateway plane ships as a single helm-chart that can be deployed on any Kubernetes cluster. The AI Gateway has no external dependencies and only needs to be able to connect to the control-plane via a secure WebSocket connection. The AI Gateway plane is also stateless and has no database or storage attached to it. The TrueFoundry gateway helm chart is available in this Github repository: tfy-llm-gateway

​
Control Plane

The control-plane ships as a single helm-chart that can be deployed on any Kubernetes cluster. It also requires a PostgreSQL database to store the data and connection to a blob storage to store the data. You can either bring your own Kubernetes cluster, Postgres and blobstorage or TrueFoundry can help provision the same using our OpenTofu/Terraform code. The control-plane helm chart is available in this Github repository: truefoundry The control-plane helm chart includes the AI Gateway helm chart as a dependency to make it easier to install both the control-plane and the AI Gateway in one go.

​
Overview of Helm Charts in TrueFoundry

The key helm-charts in TrueFoundry and their composition and usage is as follows: