How to Use HashiCorp Vault for Credential Management

Store and resolve pipeline secrets using HashiCorp Vault via the vault-kv relation

Overview

Concourse CI supports looking up secrets from HashiCorp Vault at pipeline runtime. Instead of embedding credentials in your pipeline YAML, you reference them with the ((my-secret)) syntax — Concourse fetches the value from Vault automatically when the pipeline runs.

This is useful for:

Passwords, API tokens, and SSH keys used by pipeline tasks
Credentials that rotate regularly and should not be hard-coded
Secrets that need to be scoped per-team or per-pipeline

Two integration methods are available:

Juju relation (recommended) — uses the vault-kv relation between the vault charm (2.0/stable) and the Concourse web unit. The charm handles all configuration automatically.
Manual configuration — set Vault connection parameters directly via juju config when using an externally managed Vault instance.

Method 1: Juju Relation (Recommended)

Prerequisites

Vault charm deployed from channel 2.0/stable
Concourse web unit running and related to PostgreSQL
Juju 3.1 or later

Step 1 — Deploy Vault

juju deploy vault --channel 2.0/stable

Step 2 — Integrate Vault with Concourse

Integrate the vault-kv endpoint exposed by the Vault charm with the web unit:

juju integrate vault:vault-kv web:vault-kv

Note: Replace web with your actual Concourse application name if you deployed it under a different name (e.g. concourse or concourse-web).

Step 3 — Initialize Vault

A fresh Vault deployment is sealed and uninitialized. Run the following command and save the output — you will need the unseal key and root token in later steps:

juju ssh vault/leader -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json

The JSON output contains unseal_keys_b64 and root_token. Store them securely.

Step 4 — Unseal Vault

juju ssh vault/leader -- vault operator unseal <unseal-key>

Replace <unseal-key> with the base-64 unseal key from the previous step.

Step 5 — Store the Root Token as a Juju Secret

The Vault charm needs the root token to authorize the charm credential. Store it as a Juju secret and grant access:

# Create the Juju secret
juju add-secret vault-root-token token="<root-token>"

# Grant the vault application access to this secret
juju grant-secret vault-root-token vault

Step 6 — Authorize the Vault Charm

Use the secret ID returned by juju add-secret to authorize the charm:

juju run vault/leader authorize-charm secret-id=<secret-id> --wait 2m

After this step the Vault charm will provision an AppRole credential for Concourse and write it into the relation data. The Concourse charm will pick up the credential and restart the web service automatically.

Step 7 — Verify

juju ssh web/0 -- sudo grep VAULT /var/lib/concourse/config.env

✅ Success: You should see CONCOURSE_VAULT_URL, CONCOURSE_VAULT_AUTH_BACKEND, and related variables populated in the config file.

Method 2: Manual Configuration

If you manage Vault outside of Juju (e.g. a pre-existing production Vault cluster), configure the connection parameters directly with juju config:

juju config web \
    vault-url=https://vault.example.com:8200 \
    vault-auth-backend=approle \
    vault-auth-param="role-id=<role-id>,secret-id=<secret-id>" \
    vault-ca-cert="$(cat /path/to/vault-ca.pem)"

Available configuration options:

vault-url — Full URL of the Vault server including port
vault-auth-backend — Auth backend to use (e.g. approle, token)
vault-auth-param — Comma-separated key=value auth parameters
vault-ca-cert — PEM-encoded CA certificate for TLS verification (leave empty to use system trust store)

Note: Manual configuration takes precedence over relation-provided settings. Remove the vault-url config to revert to relation-managed credentials.

Using Vault Secrets in Pipelines

Once Vault is configured, reference secrets in your pipeline YAML using double-parentheses syntax. Concourse resolves secrets at runtime from the KV mount charm-<app>-concourse (e.g. charm-web-concourse when your application is named web).

Path structure: CONCOURSE_VAULT_PATH_PREFIX is the full prefix. Concourse appends /<team>/<pipeline>/<secret> directly — there is no extra segment inserted automatically.

Write a Secret to Vault

# Write a pipeline-scoped secret
# Path format: <vault-path-prefix>/<team>/<pipeline>/<secret-name>
vault kv put charm-web-concourse/main/my-pipeline/db-password \
    value=s3cr3t

# Write a team-scoped secret (shared across all pipelines in the team)
vault kv put charm-web-concourse/main/db-password \
    value=s3cr3t

Reference the Secret in a Pipeline

resources:
  - name: app-source
    type: git
    source:
      uri: https://github.com/example/app.git
      private_key: ((github-deploy-key))

jobs:
  - name: deploy
    plan:
      - get: app-source
      - task: run-deploy
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: ubuntu
          params:
            DB_PASSWORD: ((my-team/db-password))
            API_TOKEN: ((my-pipeline/api-token))
          run:
            path: bash
            args:
              - -c
              - |
                echo "Deploying with resolved credentials..."
                ./deploy.sh

Concourse resolves secrets in the following lookup order:

Pipeline-scoped: <vault-path-prefix>/<team>/<pipeline>/<name>
Team-scoped: <vault-path-prefix>/<team>/<name>

Example with vault-path-prefix=/charm-web-concourse, team main, pipeline my-pipeline, secret db-password: Concourse checks charm-web-concourse/main/my-pipeline/db-password then charm-web-concourse/main/db-password.

Troubleshooting

Symptom	Cause	Solution
Vault unit shows blocked or waiting	Vault has not been initialized	Run `vault operator init` as shown in Step 3
Pipeline fails with secret not found	Vault is sealed	Run `vault operator unseal <key>` on the Vault leader
Concourse does not start after relation	Charm credentials not provisioned (authorize-charm not run)	Run `juju run vault/leader authorize-charm secret-id=<id> --wait 2m`
`VAULT` vars missing from `config.env`	Relation data not yet applied or charm hook error	Check `juju debug-log --include web` for hook errors

Useful Debug Commands

# Check charm logs for vault-related events
juju debug-log --include web | grep -i vault

# Inspect the config file on the web unit
juju ssh web/0 -- sudo cat /var/lib/concourse/config.env

# Check Vault seal status
juju ssh vault/leader -- vault status

Removing Vault

To disconnect Vault from Concourse, remove the relation:

juju remove-relation vault:vault-kv web:vault-kv

Warning: After removing the relation, Concourse will no longer be able to resolve ((...))) secret references. Any pipelines that depend on Vault-backed credentials will fail until you either restore the relation or supply the secrets another way.

Concourse CI Machine Charm