Files

T

igodwin abe7b6beee Refactor auth and authz

2025-10-26 02:25:24 -07:00

17 KiB

Raw Blame History

API Key Management Guide

This guide explains how to generate, manage, and use API keys with the Notifier service. The system uses a hybrid approach combining an in-memory cache for performance with PostgreSQL for persistence.

Overview

The API key management system provides:

Secure Key Generation: Cryptographically random 32-byte keys with nk_ prefix
Persistent Storage: PostgreSQL backend with full audit trail
Fast Lookups: In-memory cache with write-through consistency
Rate Limiting: Per-key request rate limits (configurable per minute)
Expiration: Optional key expiration dates
Key Rotation: Ability to revoke and create new keys
Audit Logging: Track who created/revoked keys and when
Role-Based Access: Control which notifiers each key can access

Architecture

Hybrid Cache Strategy

The system uses a write-through hybrid approach:

Request Flow:
  1. Check in-memory cache (O(1) lookup) ← Fast path
  2. If hit and valid, use immediately
  3. If miss, create from database (fallback)
  4. Update cache and return

Write Flow:
  1. Write to PostgreSQL database first
  2. If successful, update in-memory cache
  3. If DB fails, cache is not updated (consistency)

Benefits:

Fast authentication checks (cache lookup in microseconds)
Persistent storage for durability
Consistent state across restarts
Multi-instance support (all instances read from same DB)

Setup

Prerequisites

PostgreSQL 12+ database
Network access from notifier to PostgreSQL

Configuration

Add database configuration to config.yaml:

auth:
  enabled: true
  default_rate_limit: 100  # requests per minute
  database:
    url: "postgresql://user:password@localhost:5432/notifier"
    # Or use environment variable: NOTIFIER_AUTH_DATABASE_URL

Environment Variable:

export NOTIFIER_AUTH_DATABASE_URL="postgresql://user:password@localhost:5432/notifier"
export NOTIFIER_AUTH_ENABLED=true
export NOTIFIER_AUTH_DEFAULT_RATE_LIMIT=100

Database Setup

The schema is automatically created on first startup:

-- Tables created automatically:
-- api_keys: Stores API key metadata
-- api_key_audit_log: Tracks all key operations

To manually initialize the database:

psql postgresql://user:password@localhost:5432/notifier < schema.sql

Bootstrap: Creating Initial Admin Key

On first deployment, you need to create an initial admin key to bootstrap the system.

Option 1: Environment Variable (Recommended for CI/CD)

export NOTIFIER_BOOTSTRAP_ADMIN_KEY=true
./notifier serve

The service will:

Check if bootstrap has already been done
Create a random admin key with all permissions
Save it to ./notifier-admin-key.txt
Print to stdout (make sure to capture and secure this!)

Output:

============================================================
NOTIFIER BOOTSTRAP: ADMIN KEY CREATED
============================================================
Key: nk_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6
...
============================================================

Option 2: Programmatic (Docker/Kubernetes)

In your startup script:

import "github.com/igodwin/notifier/internal/auth"

bootstrapCfg := &auth.BootstrapConfig{
    Enabled:          true,
    AdminKeyFileName: "/var/run/notifier/admin-key.txt",
    PrintToStdout:    false,
}

adminKey, err := auth.BootstrapAdminKey(ctx, keyStore, bootstrapCfg, logger)
if err != nil && err.Error() != "bootstrap already completed" {
    logger.Fatalf("Bootstrap failed: %v", err)
}

Option 3: Docker Environment

FROM notifier:latest

ENV NOTIFIER_AUTH_ENABLED=true
ENV NOTIFIER_BOOTSTRAP_ADMIN_KEY=true
ENV NOTIFIER_AUTH_DATABASE_URL=postgresql://user:pass@db:5432/notifier

ENTRYPOINT ["/app/notifier", "serve"]

Capture the key:

docker logs <container-id> | grep "Key:"

Option 4: Kubernetes

apiVersion: batch/v1
kind: Job
metadata:
  name: notifier-bootstrap
spec:
  template:
    spec:
      containers:
      - name: notifier
        image: notifier:latest
        env:
        - name: NOTIFIER_AUTH_ENABLED
          value: "true"
        - name: NOTIFIER_BOOTSTRAP_ADMIN_KEY
          value: "true"
        - name: NOTIFIER_AUTH_DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: notifier-db-secret
              key: url
        volumeMounts:
        - name: keys
          mountPath: /var/run/notifier
      volumes:
      - name: keys
        secret:
          secretName: notifier-keys
      restartPolicy: Never

After the job completes, extract the key from the secret or logs.

Managing API Keys

Creating New Keys

Use the admin key to create additional keys via REST API:

# Create a key for sending emails only
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer nk_admin_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app-email",
    "roles": ["notify-email"],
    "rate_limit": 1000,
    "expires_in": "8760h"
  }'

Response:

{
  "key": "nk_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6",
  "name": "my-app-email-1698297600",
  "client_id": "my-app-email",
  "roles": ["notify-email"],
  "created_at": "2024-10-26T12:00:00Z",
  "expires_at": "2025-10-26T12:00:00Z",
  "rate_limit": 1000
}

Listing Keys

List all keys for your client:

curl -X GET http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer nk_your_key"

List keys for specific client (admin only):

curl -X GET "http://localhost:8080/api/v1/admin/keys?client_id=other-app" \
  -H "Authorization: Bearer nk_admin_key"

Revoking Keys

Disable a key (cannot be undone, but you can create a new one):

curl -X DELETE http://localhost:8080/api/v1/admin/keys/nk_key_to_revoke \
  -H "Authorization: Bearer nk_admin_key"

Rotating Keys

Create a new key with same permissions, then revoke the old one:

# 1. Create new key with same roles
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer nk_admin_key" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app",
    "roles": ["notify-email", "notify-slack"],
    "rate_limit": 1000
  }'

# 2. Update your application to use the new key

# 3. Verify everything works

# 4. Revoke the old key
curl -X DELETE http://localhost:8080/api/v1/admin/keys/nk_old_key \
  -H "Authorization: Bearer nk_admin_key"

Viewing Audit Log

See who created/revoked a key and when:

curl -X GET "http://localhost:8080/api/v1/admin/keys/nk_key/audit?limit=50" \
  -H "Authorization: Bearer nk_admin_key"

Response:

{
  "key_preview": "nk_o5p6",
  "audit_log": [
    {
      "action": "created",
      "performed_by": "admin-bootstrap",
      "performed_at": "2024-10-26T12:00:00Z",
      "details": {
        "client_id": "my-app",
        "roles": ["notify-email"]
      }
    },
    {
      "action": "deactivated",
      "performed_by": "admin-user",
      "performed_at": "2024-10-26T14:30:00Z",
      "details": null
    }
  ]
}

Using API Keys

With REST API

Include the key in the Authorization: Bearer header:

curl -X POST http://localhost:8080/api/v1/notifications \
  -H "Authorization: Bearer nk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "email",
    "subject": "Hello",
    "body": "World",
    "recipients": ["user@example.com"]
  }'

Or use the X-API-Key header:

curl -X POST http://localhost:8080/api/v1/notifications \
  -H "X-API-Key: nk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "email",
    "subject": "Hello",
    "body": "World",
    "recipients": ["user@example.com"]
  }'

With gRPC

Include the key in gRPC metadata:

Go Client:

import "google.golang.org/grpc/metadata"

ctx := metadata.AppendToOutgoingContext(context.Background(),
    "authorization", "Bearer nk_your_api_key")

client := pb.NewNotificationServiceClient(conn)
resp, err := client.SendNotification(ctx, &pb.SendNotificationRequest{
    // ...
})

Python Client:

import grpc

metadata = [('authorization', 'Bearer nk_your_api_key')]
channel = grpc.secure_channel('localhost:50051', grpc.ssl_channel_credentials())
client = pb.NotificationServiceStub(channel)
response = client.SendNotification(request, metadata=metadata)

Key Naming and Organization

Naming Convention

Keys are generated with format: nk_<32-hex-chars>

The system auto-generates a name based on client_id and timestamp:

my-app-email-1698297600
my-slack-integration-1698297700

You can customize via the name field when creating keys.

Recommended Organization

By Application:

client_id: api-gateway
client_id: worker-service
client_id: monitoring-system

By Permission:

notify-email: Email notifications only
notify-slack: Slack notifications only
notify-all: All notification types
admin: Key management + all notifications

By Environment:

my-app-prod-email
my-app-staging-email
my-app-dev-email

Security Best Practices

Do's

✅ Rotate keys regularly - Create new keys every 90 days, revoke old ones

✅ Use unique keys per service - Don't share keys between different apps

✅ Limit permissions - Only grant roles needed (e.g., notify-email not admin)

✅ Use reasonable rate limits - Prevent accidental DoS from misconfiguration

✅ Store in secure vaults - Use Kubernetes Secrets, AWS Secrets Manager, HashiCorp Vault

✅ Set expiration dates - Keys should expire after a period

✅ Monitor audit logs - Regularly check who's creating/revoking keys

✅ Use short-lived keys for CI/CD - Rotate automatically during deployments

Don'ts

❌ Don't commit keys to version control - Even private repos

❌ Don't use admin key in production - Create limited permission keys

❌ Don't share keys between teams - Each team/app gets its own

❌ Don't set unlimited rate limits - Prevents accidental overload

❌ Don't use expired keys - Revoke and create new ones

❌ Don't store plaintext in logs - Only last 4 characters should be visible

Rate Limiting

Each API key has a configurable rate limit (requests per minute):

# Create key with 1000 req/min limit
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer nk_admin_key" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app",
    "roles": ["notify-email"],
    "rate_limit": 1000
  }'

Rate Limit Errors:

Exceeding limit returns HTTP 429 Too Many Requests
Limit resets every minute
Set rate_limit: 0 for unlimited (not recommended)

Recommended Limits

Admin/Testing: 10,000 req/min
Production Email: 1,000-5,000 req/min
Production Slack: 500-1,000 req/min
Production Ntfy: 500-1,000 req/min
Development: 100-500 req/min

Expiration Dates

Keys can optionally expire:

# Create key that expires in 24 hours
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer nk_admin_key" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app",
    "roles": ["notify-email"],
    "expires_in": "24h"
  }'

# Expires in 30 days
"expires_in": "720h"

# Expires in 1 year
"expires_in": "8760h"

Duration Format: Go duration format

s - seconds (30s)
m - minutes (30m)
h - hours (24h)
d - days (not supported, use 24h instead)

Expired keys are automatically filtered when loading cache at startup.

Troubleshooting

Key Creation Fails

Error: Failed to create API key

Causes:

Database connection issue
PostgreSQL not running
Network connectivity problem

Solution:

# Test PostgreSQL connection
psql postgresql://user:password@localhost:5432/notifier -c "SELECT 1"

Authentication Fails

Error: 401 Unauthorized or API key not found

Causes:

Wrong key format
Key is revoked/expired
Key not in cache (distributed setup issue)

Solutions:

# Verify key format
echo $API_KEY | grep "^nk_"

# List your keys
curl -X GET http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer $API_KEY"

# Check if key is active
curl -X GET "http://localhost:8080/api/v1/admin/keys?client_id=your-app" \
  -H "Authorization: Bearer $ADMIN_KEY"

Rate Limit Exceeded

Error: 429 Too Many Requests

Causes:

Key rate limit exceeded in current minute
Misconfiguration sending too many requests

Solutions:

Wait a minute for limit window to reset
Check your request volume

Increase rate limit for the key:

# Create new key with higher limit
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer nk_admin_key" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app",
    "roles": ["notify-email"],
    "rate_limit": 5000
  }'

Bootstrap Key Lost

If you lose the bootstrap admin key:

Create a new database admin user with direct SQL access

Insert a new admin key into the database:

INSERT INTO api_keys (
  key, name, client_id, roles, is_active, rate_limit, created_by
) VALUES (
  'nk_your_new_key_here',
  'recovery-admin',
  'admin-recovery',
  ARRAY['admin', 'notify-email', 'notify-slack', 'notify-ntfy'],
  true,
  0,
  'system-recovery'
);

API Reference

POST /api/v1/admin/keys

Create a new API key.

Headers:

Authorization: Bearer <admin-key> (required, admin role)
Content-Type: application/json

Request Body:

{
  "client_id": "string",          // Required: client identifier
  "roles": ["string"],            // Required: array of role names
  "rate_limit": 100,              // Optional: requests per minute
  "expires_in": "24h"             // Optional: expiration duration
}

Response: 201 Created

{
  "key": "string",
  "name": "string",
  "client_id": "string",
  "roles": ["string"],
  "created_at": "RFC3339",
  "expires_at": "RFC3339",
  "rate_limit": 100
}

GET /api/v1/admin/keys

List API keys.

Headers:

Authorization: Bearer <key> (required)

Query Parameters:

client_id: Filter by client (requires admin role if different from authenticated client)

Response: 200 OK

{
  "keys": [
    {
      "key_preview": "nk_xxxx",
      "name": "string",
      "client_id": "string",
      "roles": ["string"],
      "created_at": "RFC3339",
      "last_used_at": "RFC3339",
      "expires_at": "RFC3339",
      "is_active": true,
      "rate_limit": 100
    }
  ]
}

DELETE /api/v1/admin/keys/{key}

Revoke an API key.

Headers:

Authorization: Bearer <admin-key> (required, admin role)

Request Body (optional):

{
  "reason": "Compromised key"
}

Response: 204 No Content

GET /api/v1/admin/keys/{key}/audit

Get audit log for a key.

Headers:

Authorization: Bearer <admin-key> (required, admin role)

Query Parameters:

limit: Number of log entries to return (default: 100, max: 1000)

Response: 200 OK

{
  "key_preview": "nk_xxxx",
  "audit_log": [
    {
      "action": "created|deactivated",
      "performed_by": "string",
      "performed_at": "RFC3339",
      "details": {}
    }
  ]
}

Complete Example

Step 1: Bootstrap

export NOTIFIER_BOOTSTRAP_ADMIN_KEY=true
export NOTIFIER_AUTH_ENABLED=true
export NOTIFIER_AUTH_DATABASE_URL="postgresql://user:pass@localhost:5432/notifier"

./notifier serve

# Output:
# NOTIFIER BOOTSTRAP: ADMIN KEY CREATED
# Key: nk_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

Step 2: Save Admin Key

export ADMIN_KEY="nk_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"
echo $ADMIN_KEY > ~/.notifier-admin-key
chmod 600 ~/.notifier-admin-key

Step 3: Create Service Keys

# Email service key
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app-email",
    "roles": ["notify-email"],
    "rate_limit": 1000,
    "expires_in": "8760h"
  }' | jq -r '.key' > ~/.notifier-email-key

# Slack service key
curl -X POST http://localhost:8080/api/v1/admin/keys \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-app-slack",
    "roles": ["notify-slack"],
    "rate_limit": 500
  }' | jq -r '.key' > ~/.notifier-slack-key

Step 4: Use Keys in Application

export NOTIFIER_EMAIL_KEY=$(cat ~/.notifier-email-key)
export NOTIFIER_SLACK_KEY=$(cat ~/.notifier-slack-key)

# Send email
curl -X POST http://localhost:8080/api/v1/notifications \
  -H "Authorization: Bearer $NOTIFIER_EMAIL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "email",
    "subject": "Hello",
    "body": "World",
    "recipients": ["user@example.com"]
  }'

# Send Slack
curl -X POST http://localhost:8080/api/v1/notifications \
  -H "Authorization: Bearer $NOTIFIER_SLACK_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "slack",
    "subject": "Hello",
    "body": "World",
    "recipients": ["#general"]
  }'

17 KiB Raw Blame History

API Key Management Guide

Overview

Architecture

Hybrid Cache Strategy

Setup

Prerequisites

Configuration

Database Setup

Bootstrap: Creating Initial Admin Key

Option 1: Environment Variable (Recommended for CI/CD)

Option 2: Programmatic (Docker/Kubernetes)

Option 3: Docker Environment

Option 4: Kubernetes

Managing API Keys

Creating New Keys

Listing Keys

Revoking Keys

Rotating Keys

Viewing Audit Log

Using API Keys

With REST API

With gRPC

Key Naming and Organization

Naming Convention

Recommended Organization

Security Best Practices

Do's

Don'ts

Rate Limiting

Recommended Limits

Expiration Dates

Troubleshooting

Key Creation Fails

Authentication Fails

Rate Limit Exceeded

Bootstrap Key Lost

API Reference

POST /api/v1/admin/keys

GET /api/v1/admin/keys

DELETE /api/v1/admin/keys/{key}

GET /api/v1/admin/keys/{key}/audit

Complete Example

Step 1: Bootstrap

Step 2: Save Admin Key

Step 3: Create Service Keys

Step 4: Use Keys in Application

References

17 KiB

Raw Blame History