tooolbox/blog/Vault Secrets Management Setup: Transit Encryption and AppRole Authentication
security2024-11-01Β·9Β·256/348

Vault Secrets Management Setup: Transit Encryption and AppRole Authentication

Setting up HashiCorp Vault for secrets management with AppRole authentication, transit encryption, and dynamic database credentials.

Introduction

Secrets management is one of those things you know you should do but keep postponing. For months, I was storing database passwords and API keys in environment variables and .env files. It worked, but it was not secure β€” any developer with access to the server could read them, and they were visible in process listings.

I decided to set up HashiCorp Vault to centralize secrets management. Vault provides dynamic secrets (database credentials that are generated on-demand and automatically rotated), encryption as a service, and fine-grained access control. The setup was more involved than I expected, particularly around authentication methods and policies. Here is how I configured Vault with AppRole authentication, transit encryption, and dynamic database credentials.

Environment

  • Vault version: 1.15.4 (community edition)
  • Deployment: Docker on Ubuntu 22.04
  • Backend storage: Raft integrated storage
  • Services: Go API server, Python ML pipeline, Node.js admin dashboard
  • Database: PostgreSQL 16
  • Goal: Centralized secrets, dynamic DB credentials, transit encryption

Problem

The initial Vault setup was straightforward β€” I initialized and unsealed the Vault server, then tried to configure AppRole authentication for my services. AppRole is a method that allows machines to authenticate with Vault using a role ID and secret ID, which is suitable for automated services.

But when I tried to enable AppRole, I got an error:

vault auth enable approle
# Error: approle: cannot enable approle auth method: path is already in use

AppRole was already enabled. This happened because I had enabled it during an earlier test and forgot to clean up. After clearing the old configuration, I re-enabled AppRole and created a role:

vault write auth/approle/role/api-server \
    token_policies="api-server-policy" \
    token_ttl=1h \
    token_max_ttl=4h

But when I tried to log in from the API server using the role ID and secret ID:

vault write auth/approle/login \
    role_id="abc123" \
    secret_id="def456"
# Error: permission denied

The "permission denied" error was vague β€” it could mean anything from a wrong role ID to a policy issue. This turned out to be one of several configuration issues that needed to be resolved.

Analysis

I debugged the AppRole issue step by step:

1. Verify role ID and secret ID:

# Get the role ID
vault read auth/approle/role/api-server/role-id
# role_id: abc123

# Generate a new secret ID
vault write -f auth/approle/role/api-server/secret-id
# secret_id: xyz789

I was using an old secret ID. After using the newly generated one, I got a different error.

2. Check token policies:

vault policy read api-server-policy
# Error: permission denied

The policy did not exist. I had referenced it in the role configuration but never created it.

3. Check Vault audit logging:

vault audit list -detailed
# path        type    description
# file/       file    Audit log to a file

cat /vault/audit/audit.log | tail -20
# {"type":"request","auth":{"token_type":"approle"},"error":"permission denied"}

The audit log showed that Vault was rejecting the request because the policy referenced in the AppRole role did not exist.

This was a chain of issues:

  1. Old secret ID β€” fixed by generating a new one
  2. Missing policy β€” the api-server-policy was never created
  3. Incorrect policy paths β€” even after creating the policy, the paths were wrong

Solution

I built the complete Vault configuration from scratch:

1. Initialize and configure Vault:

# Initialize Vault (5 keys, 3 threshold)
vault operator init \
    -key-shares=5 \
    -key-threshold=3 \
    -format=json > /vault/init-keys.json

# Unseal (run 3 times with different keys)
vault operator unseal 
vault operator unseal 
vault operator unseal

2. Enable and configure secrets engines:

# Enable AppRole auth
vault auth enable approle

# Enable KV v2 secrets engine
vault secrets enable -path=secret kv-v2

# Enable transit secrets engine
vault secrets enable -path=transit transit

# Enable database secrets engine
vault secrets enable -path=database database

3. Create the database connection and role:

# Configure PostgreSQL connection
vault write database/config/postgres \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/trading?sslmode=disable" \
    allowed_roles="api-server-role" \
    username="vault_admin" \
    password="secure_password"

# Create a role for dynamic credentials
vault write database/roles/api-server-role \
    db_name=postgres \
    default_ttl="1h" \
    max_ttl="24h" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

4. Create policies:

# api-server-policy.hcl
path "secret/data/api-server/*" {
  capabilities = ["read", "list"]
}

path "transit/encrypt/api-key" {
  capabilities = ["update"]
}

path "transit/decrypt/api-key" {
  capabilities = ["update"]
}

path "database/creds/api-server-role" {
  capabilities = ["read"]
}
vault policy write api-server-policy api-server-policy.hcl

5. Configure AppRole:

# Create the role with the correct policy
vault write auth/approle/role/api-server-role \
    token_policies="api-server-policy" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=10m \
    secret_id_num_uses=40

# Get role ID
vault read auth/approle/role/api-server-role/role-id
# role_id: 

# Generate secret ID
vault write -f auth/approle/role/api-server-role/secret-id
# secret_id:

6. Client application integration:

# vault_client.py
import hvac
import os

class VaultClient:
    def __init__(self):
        self.client = hvac.Client(
            url=os.environ.get("VAULT_ADDR", "http://vault:8200")
        )
        self._authenticate()

    def _authenticate(self):
        role_id = os.environ["VAULT_ROLE_ID"]
        secret_id = os.environ["VAULT_SECRET_ID"]
        self.client.auth.approle.login(
            role_id=role_id,
            secret_id=secret_id
        )

    def get_secret(self, path):
        response = self.client.secrets.kv.v2.read_secret_version(path=path)
        return response["data"]["data"]

    def get_db_credentials(self):
        response = self.client.read("database/creds/api-server-role")
        return {
            "username": response["data"]["username"],
            "password": response["data"]["password"]
        }

    def encrypt(self, plaintext, key="api-key"):
        response = self.client.secrets.transit.encrypt_data(
            name=key,
            plaintext=plaintext.encode("utf-8").hex()
        )
        return response["data"]["ciphertext"]

    def decrypt(self, ciphertext, key="api-key"):
        response = self.client.secrets.transit.decrypt_data(
            name=key,
            ciphertext=ciphertext
        )
        return bytes.fromhex(response["data"]["plaintext"]).decode("utf-8")

After deploying the corrected configuration:

# Test login
vault write auth/approle/login \
    role_id="abc123" \
    secret_id="xyz789"

# Success! Token created:
# token: hvs.CAESI...
# token_accessor: abc...
# token_policies: ["api-server-policy"]

# Test database credential generation
vault read database/creds/api-server-role
# username: v-appserver-role-abc123
# password: A1b2-C3d4-E5f6
# ttl: 1h

Dynamic credentials were being generated correctly, and the API server could authenticate and access secrets.

Lessons Learned

  1. Vault policies must exist before referencing them β€” AppRole roles reference policies by name. If the policy does not exist, Vault silently fails with a generic permission denied error.
  2. Use Vault audit logging β€” Enable audit logging from the start. It is the only way to debug authentication and authorization issues.
  3. Dynamic credentials are powerful β€” Database credentials generated by Vault are unique per session, short-lived, and automatically revoked. This eliminates the risk of leaked credentials.
  4. Store init keys securely β€” The unseal keys from vault operator init are the only way to unseal Vault. Store them in separate, secure locations.
  5. Test with small policies first β€” Start with permissive policies for debugging, then tighten them once everything works. Vault's default deny behavior can make debugging very difficult.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.