Understanding Key Distribution

Why TSA uses SSH keys, how peer and TSA relations work, and the security model

The Challenge: Workers Must Connect to Web

Concourse CI's architecture is fundamentally distributed: workers execute tasks remotely while the web server (ATC) schedules and coordinates. But how do workers authenticate to the web server securely?

Traditional approaches have problems:

Shared passwords: Difficult to rotate, vulnerable if leaked, no per-worker identity
OAuth tokens: Requires external identity provider, adds complexity
Mutual TLS: Requires PKI infrastructure, certificate rotation challenges

Concourse chose a simpler, more elegant solution: SSH public key authentication.

Why SSH Keys?

SSH keys provide several advantages for Concourse's worker-to-web connection:

Advantage	Why It Matters
Asymmetric cryptography	Public key can be freely distributed; private key never leaves the worker
Per-worker identity	Each worker has unique keypair, enabling granular access control
Battle-tested protocol	SSH is mature, well-understood, widely supported
No external dependencies	No need for external CA, LDAP, or OAuth providers
Secure key exchange	SSH host key verification prevents man-in-the-middle attacks

TSA: The "SSH Gateway" for Workers

TSA (Transportation Security Administration) is Concourse's SSH server component. It runs on the web server and acts as the authentication gateway for workers:

The TSA doesn't just authenticate workers—it also establishes a reverse SSH tunnel that ATC uses to send commands back to workers.

💡 Key Insight: Workers initiate the SSH connection to TSA (outbound from worker's perspective). The web server never needs to connect to workers—it uses the reverse tunnel. This makes firewall configuration much simpler.

The Three Types of Keys

The charm manages three categories of SSH keys:

Key Type	Purpose	Owner	Distribution
TSA Host Key	Identifies the TSA server (prevents MITM attacks)	Web/Leader	Shared with all workers
TSA Public Key	Used by TSA to verify worker connections	Web/Leader	Shared with all workers
Worker Private/Public Key	Worker authenticates itself to TSA	Each worker	Public key sent to web, private stays on worker

Key Storage Locations

# On web/leader unit:
/var/lib/concourse/keys/
├── tsa_host_key          # TSA's SSH server private key
├── tsa_host_key.pub      # TSA's SSH server public key
├── session_signing_key   # For signing session tokens
└── authorized_worker_keys  # Worker public keys (authorized_keys format)

# On worker unit:
/var/lib/concourse/keys/
├── worker_key            # Worker's SSH private key
├── worker_key.pub        # Worker's SSH public key
└── tsa_host_key.pub      # TSA's public key (for host verification)

Distribution Mechanisms: Peer vs TSA Relations

The charm uses two different Juju relations to distribute keys, depending on the deployment mode:

Peer Relation (mode=auto)

In mode=auto deployments, all units belong to the same Juju application. They automatically share data via the peer relation:

How it works:

Leader generates TSA keys on first install
Leader stores keys in peer relation data (encrypted by Juju)
Workers read keys from peer relation automatically
Workers generate their own worker keypair
Workers store their public key in peer relation
Leader collects all worker public keys and writes to authorized_worker_keys

Zero Configuration: In mode=auto, all key distribution is automatic. No manual key copying, no configuration needed. Just deploy and scale.

TSA + Flight Relations (mode=web + mode=worker)

When deploying separate web and worker applications, they use explicit tsa and flight relations:

How it works:

Web application generates TSA keys on install
When tsa relation is created, web shares keys with worker application
Worker application receives TSA keys via flight relation
Workers generate their own worker keypair
Workers send their public key back via flight relation
Web application collects worker public keys and updates authorized_worker_keys

💡 Benefit of Separate Applications: You can scale workers independently from the web server. Want 10 more workers? juju add-unit worker -n 10—keys are exchanged automatically via the existing relation.

Security Model

Threat Model: What We're Protecting Against

Threat	Protection Mechanism
Rogue worker registration	Worker public key must be in `authorized_worker_keys`
Man-in-the-middle attack	Workers verify TSA host key (prevents impersonation)
Key theft from worker	Private keys stored with `0600` permissions, owned by `root`
Relation data interception	Juju encrypts relation data in transit and at rest
Unauthorized task execution	ATC controls task assignment; workers cannot self-assign

Why Workers Don't Need Mutual TLS

You might wonder: why not use mutual TLS instead of SSH keys? Several reasons:

SSH is simpler: No certificate chain validation, no CA management
Reverse tunnel architecture: Workers initiate connection; web doesn't connect to workers
Public key as identity: Worker's public key is its identity—no need for separate certificate
Ecosystem compatibility: Concourse's design is built around SSH; changing to TLS would break existing tooling

Attack Scenario: Compromised Worker

What happens if a worker's private key is compromised?

Attacker gains worker access: Can connect to TSA as that worker
Limited blast radius: Attacker can only execute tasks assigned to that worker (cannot access other workers' tasks)
No ATC access: Worker keys don't grant access to the web UI or API
Revocation: Remove worker's public key from authorized_worker_keys to block access

⚠️ Security Best Practice: Treat worker private keys as sensitive credentials. They're stored as files on disk, not in Juju secrets, so ensure workers run in isolated environments (VMs/containers with restricted access).

Key Rotation

The charm does not currently support automatic key rotation. Keys are generated once on initial deployment and reused indefinitely. To rotate keys:

Rotating TSA Keys (Requires Downtime)

Stop all workers: juju run concourse-ci/1,2,3 -- systemctl stop concourse-worker
SSH to leader: juju ssh concourse-ci/0
Delete old keys: sudo rm /var/lib/concourse/keys/tsa_*
Regenerate: sudo ssh-keygen -t rsa -b 4096 -f /var/lib/concourse/keys/tsa_host_key -N ''
Update peer relation data with new keys (manual Juju operation)
Restart web: sudo systemctl restart concourse-server
Restart workers: juju run concourse-ci/1,2,3 -- systemctl start concourse-worker

Rotating Worker Keys (Per-Worker)

Stop worker: juju ssh concourse-ci/1 -- sudo systemctl stop concourse-worker
Delete old key: sudo rm /var/lib/concourse/keys/worker_key*
Regenerate: sudo ssh-keygen -t rsa -b 4096 -f /var/lib/concourse/keys/worker_key -N ''
Update peer relation with new public key
Restart worker: sudo systemctl start concourse-worker

Future Enhancement: Automated key rotation with zero downtime is planned for a future version. It would involve generating new keys in parallel, updating TSA to accept both old and new keys temporarily, then phasing out old keys.

Debugging Key Issues

Worker Can't Connect to TSA

Symptom: Worker logs show "Permission denied (publickey)"

Check:

# On web/leader, verify worker's public key is authorized:
juju ssh concourse-ci/0
sudo cat /var/lib/concourse/keys/authorized_worker_keys
# Should contain: ssh-rsa AAAA...worker/1's key...

# On worker, verify it has TSA host key:
juju ssh concourse-ci/1
sudo cat /var/lib/concourse/keys/tsa_host_key.pub
# Should match web's tsa_host_key.pub

Worker Shows "Host Key Verification Failed"

Cause: Worker's tsa_host_key.pub doesn't match web's actual host key

Fix:

# Compare keys:
juju ssh concourse-ci/0 -- sudo cat /var/lib/concourse/keys/tsa_host_key.pub
juju ssh concourse-ci/1 -- sudo cat /var/lib/concourse/keys/tsa_host_key.pub
# They must match exactly

Concourse CI Machine Charm - Documentation