Skip to main content
TrueFoundry AI Deployment is the platform layer for running your AI workloads on your own infrastructure. It lets data scientists and ML engineers deploy models, services, training jobs, LLMs, and workflows on Kubernetes clusters in your cloud account or on-prem data center — without needing to be Kubernetes experts. You describe what you want to deploy through the UI, CLI, or Python SDK, and TrueFoundry handles containerization, autoscaling, GPU scheduling, monitoring, and rollouts on your clusters.

​
How It Works

TrueFoundry follows a split-plane architecture. The control plane is the orchestration layer where you manage and monitor deployments — it can be TrueFoundry-hosted SaaS or self-hosted in your VPC. The compute plane is one or more Kubernetes clusters in your own environment where your workloads actually run. Your code, models, and data never leave your infrastructure.
AI Deployment architecture diagram showing your team using the UI, CLI, or Python SDK to interact with the TrueFoundry control plane, which connects over a secure WebSocket via tfy-agent to Kubernetes clusters in your compute plane running model APIs, services, jobs, LLMs, workflows, and notebooks

TrueFoundry AI Deployment architecture: the control plane orchestrates workloads on Kubernetes clusters in your own infrastructure

The control plane never accesses your clusters directly. A lightweight agent (tfy-agent) runs in each compute plane cluster and connects outbound to the control plane over a secure WebSocket, receiving deployment instructions and streaming back the state of your workloads. See TrueFoundry Architecture for the full picture.

​
Bring Your Own Compute

AI Deployment requires you to bring your own compute. TrueFoundry does not sell or resell GPUs or CPUs — the compute plane always runs in your environment:
  • Your cloud account: AWS EKS, Google Cloud GKE, or Azure AKS clusters. TrueFoundry can provision a new cluster for you using OpenTofu/Terraform, or attach to an existing one.
  • On-prem or other clouds: any standard Kubernetes or OpenShift cluster, including on-prem GPU clusters and neoclouds.
This means you keep full control over data residency, security boundaries, and cloud spend, and you can use your existing cloud commitments and discounts. To set up a cluster, see Deploy Compute Plane.

​
When Should You Use AI Deployment?

AI Deployment is the right fit when you need to run AI workloads on infrastructure you control:
  • Self-host open-source or fine-tuned LLMs on your own GPUs instead of (or alongside) calling provider APIs, for cost, latency, or data privacy reasons.
  • Serve classical ML models as APIs — scikit-learn, XGBoost, PyTorch, TensorFlow, or any framework — with autoscaling and monitoring built in.
  • Run training jobs and pipelines on schedules or on demand, including distributed and GPU workloads.
  • Give your team self-serve infrastructure — notebooks, SSH servers, and deployment workflows on shared clusters, with access control and cost visibility for platform teams.
  • Optimize compute costs with autoscaling, scale-to-zero, spot instances, and fractional GPUs.
If you only need governed access to hosted LLM providers, the AI Gateway alone is sufficient and does not require a compute plane. The two products work together: models you deploy with AI Deployment can be registered on the gateway and consumed through the same unified API.

​
What You Can Deploy

Model APIs

Deploy models from the registry or Hugging Face as real-time APIs.

Services

Deploy web services and APIs from your code or Docker images.

Jobs

Run training and batch jobs manually or on a schedule.

LLMs

Deploy open-source LLMs from the model catalogue on your GPUs.

LLM Fine-tuning

Fine-tune LLMs on your own data and compute.

Workflows

Orchestrate multi-step ML pipelines with Python.

Async Services

Queue-backed services for long-running inference.

Notebooks & SSH

Jupyter notebooks and SSH servers on cluster compute.

MCP Servers

Deploy MCP servers from code or npx/uvx packages.

Volumes

Persistent storage for models, datasets, and caches.

Experiment Tracking

Track runs, metrics, artifacts, and models.

Helm & Manifests

Deploy Helm charts and raw Kubernetes manifests.

​
Next Steps

  1. Understand the core abstractions — clusters, workspaces, repositories, and environments — in Key Concepts.
  2. Set up a compute plane cluster by following Deploy Compute Plane.
  3. Install and authenticate the CLI using Setup For CLI.
  4. Deploy your first workload with the service deployment guide or the model deployment guide.