> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Moving to External Secrets Operator for Secrets Management — v0.151.5

> TrueFoundry now uses External Secrets Operator (ESO) to sync deployment secrets and image pull credentials on your compute plane.

<Warning>
  **Applicable to:** Customers using the **TrueFoundry AI Engineering** module who deploy applications (Service, Job, Notebook, Workflow, Spark Job, Volume, and related workloads) on a compute plane cluster.

  **You must upgrade tfy-agent to version 0.2.95 or later on all compute plane clusters attached to your control plane *before* upgrading the control plane.** If the control plane is upgraded first, new deployments will fail because the cluster lacks the ESO components needed to reconcile secrets.

  If you use only the AI Gateway module and not AI Engineering, this change does not apply to you.
</Warning>

## What Is Changing

### Before

Secret references (`tfy-secret://...`) in deployment specs were resolved to their actual values on the control plane, and those resolved values were embedded directly in the Kubernetes Secret objects within the ArgoCD Application manifests applied to your cluster. This meant the actual secret value ended up materialized in the Application YAML on the cluster — not ideal from a security standpoint. Updating a secret also required redeploying or re-syncing the application.

### After

Starting with **v0.151.5**, TrueFoundry no longer puts secret values in the ArgoCD manifests. Instead, it adds an `ExternalSecret` custom resource to the deployment manifest. The [External Secrets Operator (ESO)](https://external-secrets.io/) running on your cluster reads this resource, fetches the actual secret value from the control plane, and creates the Kubernetes Secret object locally.

Here's a simplified example of what the `ExternalSecret` resource looks like:

```yaml theme={"dark"}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-image-pull-creds-external-secret
  labels:
    truefoundry.com/application: my-app
    truefoundry.com/component-type: service
spec:
  data:
    - remoteRef:
        key: tfy-image-pull-creds://<cluster>:<registry>
      secretKey: .dockerconfigjson
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: tfy-agent-secret-store
  target:
    name: my-app-image-pull-secret
    template:
      type: kubernetes.io/dockerconfigjson
```

ESO understands this resource and fetches the actual secret from the control plane to create the Kubernetes Secret. Secrets are refreshed every **1 hour** by default.

<Note>
  Automatic secret refresh does **not** restart running pods. Pods will pick up the new secret values on their next restart. Automated pod restarts on secret rotation will be supported in a future release.
</Note>

<Accordion title="How External Secrets Operator (ESO) works">
  ESO is a Kubernetes operator that reads `ExternalSecret` custom resources and materializes them as standard K8s secret objects. TrueFoundry installs and configures ESO on your compute plane cluster through the [tfy-agent Helm chart](https://github.com/truefoundry/infra-charts/tree/main/charts/tfy-agent) (version 0.2.95+).

  | Component                                   | Scope           | Namespace          | Role                                                                                  |
  | ------------------------------------------- | --------------- | ------------------ | ------------------------------------------------------------------------------------- |
  | **External Secrets Operator controller**    | Cluster         | `tfy-agent`        | Watches `ExternalSecret` resources and creates or updates K8s secret objects          |
  | **ClusterSecretStore** (`tfy-secret-store`) | Cluster         | —                  | Tells ESO how to reach the TrueFoundry control plane to resolve secret references     |
  | **ExternalSecret**                          | Per application | Workload namespace | Declares which TrueFoundry secret reference maps to which K8s secret object           |
  | **K8s secret object**                       | Per application | Workload namespace | The in-cluster secret your Pods mount as env vars, volumes, or image pull credentials |

  The `ClusterSecretStore` uses a webhook provider that calls the TrueFoundry control plane:

  ```
  {controlPlaneURL}/api/svc/v1/control-plane/secret?secret_ref={{ .remoteRef.key }}
  ```

  The webhook authenticates using the cluster token stored in the tfy-agent token K8s secret object (`CLUSTER_TOKEN` key).
</Accordion>

### Affected workloads and secret types synced by ESO

| Workload type                        | What is auto-synced                                                                           |
| ------------------------------------ | --------------------------------------------------------------------------------------------- |
| **Service / Async Service**          | Environment secrets, volume mounts, image pull credentials, and SQS autoscaling auth secrets  |
| **Job / Cron Job**                   | Environment and mount secrets, plus image pull credentials                                    |
| **Spark Job**                        | Environment and mount secrets, plus image pull credentials                                    |
| **Workflow**                         | Task-level secrets and mounts                                                                 |
| **Volume**                           | Secrets attached to volume operations                                                         |
| **Workbench** (Notebook, SSH Server) | Environment and mount secrets                                                                 |
| **Alert Manager**                    | Notification channel credentials (email, Slack, PagerDuty, and similar provider integrations) |

## Why This Change

* **Image pull secrets refresh automatically** — Private registry credentials are also managed by ESO and refresh on the same interval, so updated registry tokens propagate without a full redeploy.
* **Single source of truth** — Secret values are always resolved from the TrueFoundry control plane at runtime, matching the `tfy-secret://...` references in your deployment specs.
* **Standard Kubernetes pattern** — ESO is the widely adopted operator for syncing external secrets into K8s. TrueFoundry manages its installation and configuration for you through tfy-agent.

## What You Need to Do

<Steps>
  <Step title="Upgrade tfy-agent on every compute plane cluster">
    Update the [tfy-agent Helm chart](https://github.com/truefoundry/infra-charts/tree/main/charts/tfy-agent) to version **0.2.95 or later** on each compute plane cluster. This version includes the External Secrets Operator subchart and the `ClusterSecretStore` configuration. See the [values reference](https://github.com/truefoundry/infra-charts/blob/main/charts/tfy-agent/values.yaml#L794) for all available options.

    Verify ESO is running after the upgrade:

    ```bash theme={"dark"}
    kubectl get pods -n tfy-agent -l app.kubernetes.io/name=external-secrets
    kubectl get clustersecretstore tfy-secret-store
    ```

    The `ClusterSecretStore` should report a `Ready` status once the cluster token K8s secret object is configured correctly.
  </Step>

  <Step title="Add the webhook label if you use an external cluster token secret">
    If you pass the cluster token inline via `config.clusterToken` in the tfy-agent values file, the chart creates the token K8s secret object and adds the required label automatically when ESO is enabled. **No action needed.**

    If you store the cluster token in an **existing K8s secret object** referenced by `config.clusterTokenSecret`, you must add this label so ESO can use it as a webhook authentication source:

    ```yaml theme={"dark"}
    metadata:
      labels:
        external-secrets.io/type: webhook
    ```

    The K8s secret object must contain a key named `CLUSTER_TOKEN`. Find the secret name in your tfy-agent values file under [`config.clusterTokenSecret`](https://github.com/truefoundry/infra-charts/blob/main/charts/tfy-agent/values.yaml).

    Example using `kubectl`:

    ```bash theme={"dark"}
    kubectl label secret <your-cluster-token-secret-name> \
      -n tfy-agent \
      external-secrets.io/type=webhook
    ```

    Replace `<your-cluster-token-secret-name>` with the value of `config.clusterTokenSecret` from your tfy-agent Helm values.
  </Step>

  <Step title="Confirm ESO is enabled in tfy-agent values">
    ESO is enabled by default. Confirm these settings in your tfy-agent values file (or rely on the defaults):

    ```yaml theme={"dark"}
    external-secrets-operator:
      enabled: true
      clusterSecretStore:
        create: true
        name: tfy-secret-store
    ```

    <Note>
      **If you already run External Secrets Operator in your cluster**, set `external-secrets-operator.enabled: false` to avoid installing a second ESO instance (running two operators can cause them to fight over the same resources). With the subchart disabled, tfy-agent still creates the `ClusterSecretStore` (`tfy-secret-store`) that TrueFoundry's `ExternalSecret` resources reference, and your existing operator reconciles them.

      ```yaml theme={"dark"}
      external-secrets-operator:
        enabled: false
        clusterSecretStore:
          create: true
          name: tfy-secret-store
      ```

      Your existing ESO installation must be a compatible version that supports the `external-secrets.io/v1` API and the `webhook` provider.
    </Note>

    See the full [tfy-agent values reference](https://github.com/truefoundry/infra-charts/blob/main/charts/tfy-agent/values.yaml#L794) for all available options.
  </Step>

  <Step title="Redeploy or sync affected applications">
    After tfy-agent is upgraded and the cluster token K8s secret object is labeled (if applicable), redeploy or sync applications that use secrets. New manifest generation produces `ExternalSecret` resources that ESO uses to manage K8s secret objects.

    You can verify on a running application:

    ```bash theme={"dark"}
    kubectl get externalsecret -n <workspace-namespace>

    kubectl get secret -n <workspace-namespace> -l app.kubernetes.io/managed-by=external-secrets
    ```
  </Step>
</Steps>

### Configuration reference

For the complete list of ESO-related settings and their defaults, see the [external-secrets-operator section of the tfy-agent values file](https://github.com/truefoundry/infra-charts/blob/main/charts/tfy-agent/values.yaml#L794).

***

If you have questions or run into issues during the migration, reach out to [**support@truefoundry.com**](mailto:support@truefoundry.com) — we're happy to help.