Key Components
To install the complete control plane on your own infrastructure, you need to install the following components:TrueFoundry Control Plane + Gateway
Shipped as a single Helm chart called
truefoundry that includes both the control plane and AI Gateway.PostgreSQL Database
Managed or self-hosted PostgreSQL (version 13 or higher).
Blob Storage
S3, GCS, Azure Blob, or any other S3-compatible storage for request logs.
Compute Requirements
Truefoundry ships as a helm chart (https://github.com/truefoundry/infra-charts/tree/main/charts/truefoundry) that has configurable options to either deploy both Deployment and AI Gateway feature or just choose the one of them according to your needs. The compute requirements change based on the set of features and the scale of the number of users and requests. Here are a few scenarios that you can choose from based on your needs.Small (Dev)
Small (Dev)
The small tier is recommended for development purposes. Here all the components are deployed on Kubernetes and in non HA mode (single replica). This is suitable if you are just testing out the different features of Truefoundry.
Resource requirements are the same across AWS, GCP, and Azure.
Costs shown are approximate AWS prices
| Component | Resources | Min Nodes | Approx. Cost (AWS) |
|---|---|---|---|
| Helm-Chart (AI Gateway Control Plane components) | CPU: 6 vCPU Memory: 12 GB Storage: 60 GB Persistent Volumes (Block Storage) on Kubernetes | 2 Pods should be spread over min 2 nodes | ~ $220 pm (EC2 and EC2 others) |
| Helm-Chart (AI Gateway component only) | CPU: 1 vCPU Memory: 512 Mi Storage: — | 1 Pods should be spread over min 1 node | ~ $35 pm (EC2 and EC2 others) |
| Postgres (Deployed on Kubernetes) | CPU: 0.5 vCPU Memory: 0.5 GB Storage: 5 GB Persistent Volumes (Block Storage) on Kubernetes | — | ~ $15 pm (RDS compute and storage) |
| Blob Storage (S3 Compatible) | Storage: 20 GB | — | ~ $3 pm (S3 storage) |
Medium (Recommended)
Medium (Recommended)
The medium tier is configured for production and will suffice teams of 10-500 members.
Resource requirements are the same across AWS, GCP, and Azure. Costs shown are approximate AWS prices
The AI Gateway is configured for a minimum 3 replicas (1 vCPU 1GB each) which can handle around 500 requests/second to LLMs.It’s configured to be horizontally scalable and autoscale when the load increases. The Block Storage and S3 are used to store LLM request logs. The size is dependent on the size and number of requests and should be set as per the expected usage.
| Component | Resources | Min Nodes | Approx. Cost (AWS) |
|---|---|---|---|
| Helm-Chart (AI Gateway Control Plane components) | CPU: 14 vCPU Memory: 32 GB Storage: 250 GB | 3 Pods should be spread over min 3 nodes | ~ $600 pm (EC2 and EC2 others) |
| Helm-Chart (AI Gateway component only) | CPU: 3 vCPU Memory: 3 GB Storage: — | 3 Pods should be spread over min 3 nodes | ~ $105 pm (EC2 and EC2 others) |
| Postgres (Managed Database) | CPU: 2 vCPU Memory: 4 GB Storage: 30 GB | — | ~ $60 pm (RDS compute and storage) |
| Blob Storage (S3 Compatible) | Storage: 500 GB | — | ~ $15 pm (S3 storage) |
Large (Prod)
Large (Prod)
The large tier is configured for production and will suffice organizations of 500-50000 members.
Resource requirements are the same across AWS, GCP, and Azure. Costs shown are approximate AWS prices
The AI Gateway is configured for a minimum 10 replicas (1 vCPU 1GB each) which can handle around 2000 requests/second to LLMs.It’s configured to be horizontally scalable and autoscale when the load increases. The Block Storage and S3 are used to store LLM request logs. The size is dependent on the size and number of requests and should be set as per the expected usage.
| Component | Resources | Min Nodes | Approx. Cost (AWS) |
|---|---|---|---|
| Helm-Chart (AI Gateway Control Plane components) | CPU: 32 vCPU Memory: 64 GB Storage: 400 GB | 10 Pods should be spread over min 10 nodes | ~ $1400 pm (EC2 and EC2 others) |
| Helm-Chart (AI Gateway component only) | CPU: 10 vCPU Memory: 10 GB Storage: — | 10 Pods should be spread over min 10 nodes | ~ $350 pm (EC2 and EC2 others) |
| Postgres (Managed Database) | CPU: 2 vCPU Memory: 4 GB Storage: 30 GB | — | ~ $60 pm (RDS compute and storage) |
| Blob Storage (S3 Compatible) | Storage: 1000 GB | — | ~ $30 pm (S3 storage) |
Prerequisites for Installation
Make sure the following are set up in your environment before starting the installation.| Requirement | Details |
|---|---|
| Kubernetes Cluster | Kubernetes version 1.27 or higher. |
| Dynamic Storage Provisioning | Support for dynamic PVC provisioning (e.g., AWS EBS, Azure Disk, GCP Persistent Disk). |
| Ingress | An ingress controller (e.g., NGINX Ingress Controller) or Istio service mesh to expose the control plane dashboard and AI Gateway. |
| Domain & TLS Certificate | A domain to map the ingress of the control plane dashboard and AI Gateway, along with a TLS certificate for the domain. This domain is referred to as the Control Plane URL in our documentation. |
| Egress Access | Outbound HTTPS access to the TrueFoundry endpoints listed below. |
| PostgreSQL Database | PostgreSQL version >= 13. Managed PostgreSQL (AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL) is recommended for production. For PostgreSQL 17+, disable SSL (AWS: set force_ssl=0; Azure: set require_secure_transport=false in the parameter group). For instance sizing, refer to the Compute Requirements section. |
| Blob Storage | S3, GCS, Azure Blob Storage, or any other S3-compatible storage to store AI Gateway request logs. Setup instructions are included in the guide below. |
Egress URLs from your cluster to TrueFoundry
https://auth.truefoundry.com— Central authentication server for licensing and authentication.https://login.truefoundry.com— Login UI for the central authentication server.https://catalogue.truefoundry.com— Central repository for fetching catalogues for latest models, their public cost, MCP servers, etc.https://analytics.truefoundry.com— Analytics server for sending usage analytics to TrueFoundry.
If you do not have a managed PostgreSQL database and just want to test, set
devMode to true in the values file to spin up a local PostgreSQL database on Kubernetes.Things required from TrueFoundry team
The following will be provided by the TrueFoundry team. Make sure your organization is registered on TrueFoundry before requesting them.| Item | Description |
|---|---|
| Tenant Name | Unique identifier for your control plane. |
License Key (TFY_API_KEY) | Authentication key used to license your installation. |
| Image Pull Secret | Docker config JSON used to pull TrueFoundry container images. |
One Tenant Name and License Key must only be used to set up one Control Plane. Switching to a new tenant name and license key later will lead to complete data loss of the existing control plane.
Contents
| Section | Description |
|---|---|
| Overview | Components, compute requirements, prerequisites |
| AWS installation | Install on AWS with S3 and IAM |
| GCP installation | Install on GCP with GCS |
| Azure installation | Install on Azure Blob Storage |
| OpenShift installation | OpenShift-specific Helm values |
| On-prem installation | Minio, self-hosted Postgres, generic setup |
| FAQs | Common questions, troubleshooting, and advanced configuration |