← Back to Blog

Production-ready Kubernetes Part 8 - Secrets, Vaults, and the Real Cost of Leaked Credentials

Secrets, Vaults, and the Real Cost of Leaked Credentials

3/19/2026

Kubernetes makes it deceptively easy to store sensitive data.

A simple command like:

echo -n "password" | base64

can quickly become a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: cGFzc3dvcmQ=

And suddenly, your application can access credentials without hardcoding them.

Simple, right?

Unfortunately, secret management in production is rarely that simple.

If you run a large enough system long enough, credentials will eventually leak through:

  • logs
  • debugging output
  • CI pipelines
  • crash dumps
  • misconfigured RBAC
  • compromised nodes

At that point, your secret management architecture determines the blast radius.

In Kubernetes, teams typically evolve through three patterns:

  • 1️⃣ Environment variables
  • 2️⃣ Mounted secret files
  • 3️⃣ External vault systems

Each approach has different tradeoffs in security, operational complexity, and developer experience.

Understanding when each pattern is acceptable — and when it becomes dangerous — is key to running Kubernetes in production.

1️⃣ Kubernetes Secrets Are Not Really "Secret"

Before discussing patterns, it's important to understand something many engineers misunderstand.

Kubernetes Secrets are not encrypted by default.

They are simply base64-encoded values stored in etcd.

Example:

data:
  password: cGFzc3dvcmQ=

This is encoding, not encryption.

Anyone with access to the cluster API can decode it instantly:

kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 --decode

If a malicious actor gains:

  • access to the API server
  • permissions via RBAC
  • or access to etcd backups

they can retrieve every secret stored in the cluster.

Mitigations exist:

  • Encryption at rest for etcd
  • RBAC restrictions
  • audit logging
  • external secret managers

But it's important to understand:

Kubernetes Secrets are a distribution mechanism, not a full secret management system.

2️⃣ Environment Variables — The Quick & Dirty Pattern

The simplest way to inject secrets into an application is through environment variables.

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
        - name: api
          image: my-api:latest
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password

The application simply reads the environment variable:

db_password = os.getenv("DB_PASSWORD")

This pattern is popular because:

  • easy for developers
  • supported by every language
  • easy to configure in Kubernetes

But environment variables have serious leakage risks.

Secrets can appear in:

  • process listings
  • crash dumps
  • debugging logs
  • application error output
  • CI/CD pipeline logs
  • container inspection tools

Example:

kubectl exec -it pod -- printenv

This can expose all environment variables.

When environment variables are acceptable

Environment variables are fine for:

  • non-sensitive configuration
  • feature flags
  • internal service endpoints
  • low-risk tokens

Example:

CACHE_TTL=60
FEATURE_FLAG_NEW_UI=true
SERVICE_ENDPOINT=https://internal-api

When they should be avoided

Avoid environment variables for:

  • database passwords
  • API tokens
  • private keys
  • encryption keys

If a credential leak would trigger a security incident, environment variables are not the right solution.

3️⃣ Mounted Secrets — A Safer Default

A more secure approach is mounting secrets as files.

Example Kubernetes configuration:

volumes:
  - name: db-secret
    secret:
      secretName: db-credentials


containers:
  - name: api
    image: my-api:latest
    volumeMounts:
      - name: db-secret
        mountPath: /secrets
        readOnly: true

Inside the container, Kubernetes creates files:

/secrets/password

The application reads the file:

with open("/secrets/password") as f:
    db_password = f.read().strip()

Kubernetes mounts these secrets via tmpfs, meaning:

  • stored in memory
  • not written to disk

Why mounted secrets are safer

Mounted secrets reduce several risks:

  • not exposed in process trees
  • less likely to appear in logs
  • easier to restrict filesystem access

Another benefit:

Applications can watch files for changes.

This allows secret rotation without restarting pods.

Example patterns:

  • Nginx reloading TLS certificates
  • apps watching configuration files
  • sidecars updating credentials

Limitations

Mounted secrets still have limitations:

  • secrets still exist in etcd
  • secrets still accessible via Kubernetes API
  • rotation is often manual
  • auditing is limited

For many organizations, this is good enough.

But for regulated environments, it may not be.

4️⃣ External Vaults — Centralized Secret Management

For organizations with strong security requirements, external vault systems provide a more robust solution.

Examples include:

  • HashiCorp Vault
  • AWS Secrets Manager
  • Azure Key Vault
  • Google Secret Manager

These systems provide capabilities Kubernetes lacks:

  • automatic secret rotation
  • dynamic credentials
  • centralized access policies
  • full audit trails
  • secret leasing and expiration

Example: dynamic database credentials with Vault.

Instead of storing a static password, Vault generates temporary credentials:

username: app-2981
password: xF32kQ91
ttl: 1 hour

When the credential expires, it becomes invalid automatically.

This drastically reduces the blast radius of credential leaks.

The tradeoff: operational complexity

Vault systems introduce new operational responsibilities:

  • running or integrating the vault
  • configuring authentication
  • managing policies
  • integrating with Kubernetes

For small systems, this complexity may not be worth it.

But for large platforms, it often becomes necessary.

5️⃣ The Bridge — Kubernetes Operators

External vaults are powerful, but they need integration with Kubernetes.

This is where operators come in.

Popular tools include:

  • External Secrets Operator (ESO)
  • Bank-Vaults
  • Secrets Store CSI Driver

Example using External Secrets Operator:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: production/db/password

The operator:

  1. fetches secrets from the external vault
  2. creates or updates Kubernetes Secrets
  3. keeps them synchronized

This allows teams to combine:

  • vault security
  • Kubernetes-native consumption

without manually copying secrets.

Conclusion

There is no single correct way to manage secrets in Kubernetes.

Instead, the right solution depends on:

  • the sensitivity of your credentials
  • your security requirements
  • your operational maturity
  • your compliance constraints

A common maturity path looks like this:

Early stage systems

Environment variables for simple configuration.

Growing platforms

Mounted secrets for better isolation.

Security-critical environments

External vaults with automated rotation.

The key insight is that secret management defines your blast radius.

When a credential eventually leaks — and one eventually will — your architecture determines how painful that incident becomes.

Actionable Steps

Step 1 — Audit your current secrets

Identify:

  • where secrets are stored
  • how they are injected into applications
  • which credentials are most sensitive

Step 2 — Reduce environment variable usage

Move sensitive credentials away from environment variables and into mounted secrets.

Step 3 — Enable encryption at rest

Ensure Kubernetes etcd encryption is enabled for secrets.

Step 4 — Implement RBAC restrictions

Restrict who can:

  • list secrets
  • read secrets
  • modify secrets.

Step 5 — Evaluate external vaults

If your system requires:

  • strict auditing
  • automated rotation
  • compliance controls

introduce a dedicated secret management platform.

Related Posts

Production-ready Kubernetes Series: