From 52734efdec048cb8eda5ef6b0785f2a946a77993 Mon Sep 17 00:00:00 2001 From: Ivan Godwin Date: Thu, 30 Oct 2025 22:57:38 -0700 Subject: [PATCH] New doc supporting CORS, gRCP, and general client integration --- CLIENT_INTEGRATION_README.md | 336 +++++++++++++++ docs/CLIENT_INTEGRATION_PROMPT.md | 344 +++++++++++++++ docs/CORS_IMPLEMENTATION.md | 255 +++++++++++ docs/CORS_KUBERNETES_GUIDE.md | 357 ++++++++++++++++ docs/GRPC_CONNECTION_OPTIMIZATION.md | 607 +++++++++++++++++++++++++++ docs/QUICK_START_CLIENT.md | 319 ++++++++++++++ k8s/configmap.yaml | 29 ++ 7 files changed, 2247 insertions(+) create mode 100644 CLIENT_INTEGRATION_README.md create mode 100644 docs/CLIENT_INTEGRATION_PROMPT.md create mode 100644 docs/CORS_IMPLEMENTATION.md create mode 100644 docs/CORS_KUBERNETES_GUIDE.md create mode 100644 docs/GRPC_CONNECTION_OPTIMIZATION.md create mode 100644 docs/QUICK_START_CLIENT.md diff --git a/CLIENT_INTEGRATION_README.md b/CLIENT_INTEGRATION_README.md new file mode 100644 index 0000000..da77492 --- /dev/null +++ b/CLIENT_INTEGRATION_README.md @@ -0,0 +1,336 @@ +# Client Integration Guide - Notifier Service + +**Quick links for different use cases:** + +## 📚 Documentation Index + +### For Developers Building New Clients + +1. **Start Here:** [`docs/QUICK_START_CLIENT.md`](docs/QUICK_START_CLIENT.md) + - 5-minute quick start guide + - Complete minimal working example + - Essential requirements checklist + +2. **AI Assistant Prompt:** [`docs/CLIENT_INTEGRATION_PROMPT.md`](docs/CLIENT_INTEGRATION_PROMPT.md) + - Copy-paste prompt for Claude/ChatGPT/etc. + - Generates production-ready client code + - Includes all requirements and best practices + +3. **Performance Deep Dive:** [`docs/GRPC_CONNECTION_OPTIMIZATION.md`](docs/GRPC_CONNECTION_OPTIMIZATION.md) + - Why long-lived connections matter (10x faster!) + - Connection pooling strategies + - Keepalive configuration + - Resilience patterns + +4. **Best Practices:** [`docs/CLIENT_RECOMMENDATIONS.md`](docs/CLIENT_RECOMMENDATIONS.md) + - Advanced patterns + - Security hardening + - Observability and monitoring + - Testing strategies + +### For Kubernetes Deployments + +5. **CORS and Kubernetes:** [`docs/CORS_KUBERNETES_GUIDE.md`](docs/CORS_KUBERNETES_GUIDE.md) + - Do you need CORS? (Spoiler: Not for gRPC!) + - Service-to-service communication + - When CORS matters (web frontends) + +6. **CORS Implementation:** [`docs/CORS_IMPLEMENTATION.md`](docs/CORS_IMPLEMENTATION.md) + - Technical details of CORS security + - Configuration examples + - Migration guide + +### For Authentication & Security + +7. **Authentication Guide:** [`docs/AUTH.md`](docs/AUTH.md) + - Complete auth system documentation + - API key management + - Role-based access control (RBAC) + +## 🎯 Quick Decision Tree + +### "I need to build a new client application" + +**Option A: Using AI Assistant (Recommended)** +→ Use [`CLIENT_INTEGRATION_PROMPT.md`](docs/CLIENT_INTEGRATION_PROMPT.md) +→ Paste the prompt into Claude/ChatGPT +→ Review generated code against checklist below + +**Option B: Manual Implementation** +→ Start with [`QUICK_START_CLIENT.md`](docs/QUICK_START_CLIENT.md) +→ Copy the complete example +→ Customize for your needs +→ Reference [`GRPC_CONNECTION_OPTIMIZATION.md`](docs/GRPC_CONNECTION_OPTIMIZATION.md) for performance + +### "I'm deploying to Kubernetes" + +→ Read [`CORS_KUBERNETES_GUIDE.md`](docs/CORS_KUBERNETES_GUIDE.md) first +→ No CORS needed for backend gRPC communication! ✅ +→ Configure Kubernetes Secret for API key +→ Use service DNS: `notifier-grpc:50051` + +### "My client is too slow" + +→ Check [`GRPC_CONNECTION_OPTIMIZATION.md`](docs/GRPC_CONNECTION_OPTIMIZATION.md) +→ Verify you're using long-lived connections (not creating per request) +→ Add keepalive configuration +→ Consider connection pooling for high throughput + +### "Authentication isn't working" + +→ See [`AUTH.md`](docs/AUTH.md) for troubleshooting +→ Verify API key format: `nk_` + 64 hex characters +→ Check you're sending it in metadata with every request +→ Confirm key is active and has required roles + +## ✅ Client Implementation Checklist + +Use this checklist to verify your client implementation: + +### Connection Management +- [ ] Using single long-lived connection (created at startup) +- [ ] Connection reused for all requests (not creating per request) +- [ ] Keepalive configured (10s ping, 3s timeout) +- [ ] Connection closed on application shutdown +- [ ] Health checks implemented (every 30-60s) + +### Authentication +- [ ] API key loaded from environment variable (`NOTIFIER_API_KEY`) +- [ ] API key sent in metadata with every request +- [ ] Using correct format: `authorization: Bearer nk_...` +- [ ] No hardcoded credentials in source code +- [ ] API key masked in logs (never logged in full) + +### Error Handling +- [ ] Rate limit errors retry with exponential backoff +- [ ] Authentication errors logged and not retried +- [ ] Invalid argument errors logged and not retried +- [ ] Unavailable errors retry with backoff +- [ ] Maximum retry limit enforced (3-5 attempts) + +### Context & Timeouts +- [ ] Every request uses context with timeout (5-30s) +- [ ] Contexts properly cancelled with defer +- [ ] Deadline exceeded errors handled gracefully + +### Security +- [ ] Credentials in Kubernetes Secret (not ConfigMap) +- [ ] No credentials in Docker image layers +- [ ] `.gitignore` excludes credential files +- [ ] No sensitive data in logs or metrics labels + +### Observability +- [ ] Connection establishment logged (info level) +- [ ] Errors logged with appropriate levels +- [ ] Success responses logged (debug level) +- [ ] Metrics instrumented (if using Prometheus) +- [ ] Distributed tracing integrated (if using OpenTelemetry) + +### Testing +- [ ] Unit tests with mocked client +- [ ] Retry logic tested +- [ ] Error handling tested +- [ ] Integration tests (if service available) + +### Production Readiness +- [ ] Graceful shutdown implemented +- [ ] Circuit breaker pattern (optional, for resilience) +- [ ] Notification failures don't crash application +- [ ] Health check endpoint exposed (if needed) + +## 🔐 API Key Management + +### Getting an API Key + +Contact your Notifier service administrator to create a key: + +```bash +# Using the notifier CLI (if available) +notifier-cli keys create \ + --client-id "my-application" \ + --roles "email,slack" \ + --rate-limit 1000 + +# Output: +# Created API key: nk_a1b2c3d4e5f6... +# Save this key securely - it won't be shown again! +``` + +### API Key Format + +``` +nk_<64 hex characters> + +Example: +nk_a1b2c3d4e5f6789012345678901234567890123456789012345678901234 +│ └─────────────────────────────────────────────────────────┘ +│ 64 characters +└── Prefix (always "nk_") +``` + +### Storing API Keys + +**Development:** +```bash +export NOTIFIER_API_KEY="nk_your_dev_key_here" +``` + +**Production (Kubernetes):** +```yaml +# 1. Create secret +kubectl create secret generic notifier-credentials \ + --from-literal=api-key=nk_your_prod_key_here + +# 2. Reference in deployment +env: +- name: NOTIFIER_API_KEY + valueFrom: + secretKeyRef: + name: notifier-credentials + key: api-key +``` + +### Rate Limits + +Default: **100 requests per minute** (configurable per key) + +When rate limit is exceeded: +- Error code: `codes.ResourceExhausted` +- Retry after: 60 seconds (check `Retry-After` header) +- Strategy: Implement exponential backoff + +To request higher rate limit, contact your administrator. + +## 🚀 Example Implementations + +### Minimal Example (50 lines) + +See [`QUICK_START_CLIENT.md`](docs/QUICK_START_CLIENT.md) for complete code. + +### Production-Ready Example + +Use the AI assistant prompt in [`CLIENT_INTEGRATION_PROMPT.md`](docs/CLIENT_INTEGRATION_PROMPT.md) to generate a complete implementation with: +- Connection pooling +- Retry logic +- Health checks +- Metrics +- Graceful shutdown +- Unit tests + +### Language-Specific Examples + +**Go:** See `QUICK_START_CLIENT.md` (complete example included) + +**Python/Java/Node.js:** Use the prompt in `CLIENT_INTEGRATION_PROMPT.md` with your AI assistant and specify your language: + +``` +[Include the prompt from CLIENT_INTEGRATION_PROMPT.md] + +Please implement this in Python using the grpcio library. +``` + +## 🔧 Troubleshooting + +### Common Issues and Solutions + +| Issue | Symptom | Solution | +|-------|---------|----------| +| **Slow performance** | 10-15ms per request | Verify using long-lived connection, not creating new connection per request | +| **Authentication failed** | `codes.Unauthenticated` | Check API key format, confirm key is active, verify metadata header | +| **Rate limit exceeded** | `codes.ResourceExhausted` | Implement exponential backoff retry, consider batching requests | +| **Connection timeout** | Requests hang indefinitely | Add keepalive configuration, set context timeout | +| **Connection dropped** | Errors after idle period | Configure keepalive: 10s ping interval | +| **Health checks failing** | Service appears down | Check network connectivity, verify service is running, check DNS resolution | + +### Debug Checklist + +```bash +# 1. Verify service is running +kubectl get pods -l app=notifier +kubectl get svc notifier-grpc + +# 2. Check service is reachable +kubectl run -it --rm debug --image=busybox --restart=Never -- \ + nslookup notifier-grpc + +# 3. Test gRPC connectivity (if grpcurl installed) +grpcurl -plaintext notifier-grpc:50051 list + +# 4. Verify API key +echo $NOTIFIER_API_KEY | wc -c # Should be 67 (nk_ + 64 chars + newline) + +# 5. Check application logs +kubectl logs -l app=your-app --tail=100 +``` + +## 📊 Performance Expectations + +**With Long-Lived Connection (Recommended):** +- First request: ~15ms (connection setup + auth) +- Subsequent requests: ~1-2ms (auth only) +- Throughput: ~500-1000 req/s per connection +- Latency p99: <5ms + +**Without Connection Reuse (Bad):** +- Every request: ~15ms (repeated connection setup) +- Throughput: ~60 req/s per connection +- Latency p99: ~20ms + +**Connection Pool (High Throughput):** +- 4 connections: ~2000-4000 req/s +- Linear scaling with connection count +- Use for >1000 req/s sustained load + +## 🎓 Learn More + +- **gRPC Documentation:** https://grpc.io/docs/ +- **Protocol Buffers:** https://protobuf.dev/ +- **gRPC Best Practices:** https://grpc.io/docs/guides/performance/ +- **Kubernetes Secrets:** https://kubernetes.io/docs/concepts/configuration/secret/ + +## 📞 Support + +**Questions about:** +- API keys, rate limits, permissions → Contact Notifier admin +- Client implementation, best practices → See documentation above +- Service issues, downtime → Check service health dashboard +- Feature requests → Open issue in Notifier repository + +## 📝 Quick Reference Card + +``` +Service: notifier-grpc:50051 +Protocol: gRPC (HTTP/2) +Auth: authorization: Bearer nk_<64-hex> +Rate Limit: 100 req/min (default) +Timeout: 10s recommended +Keepalive: 10s ping, 3s timeout +Connection: Long-lived (reuse) +``` + +**Essential imports (Go):** +```go +import ( + "google.golang.org/grpc" + "google.golang.org/grpc/keepalive" + "google.golang.org/grpc/metadata" + pb "github.com/igodwin/notifier/api/grpc/pb" +) +``` + +**Minimal client setup:** +```go +// 1. Create connection with keepalive +conn, _ := grpc.Dial(address, grpc.WithKeepaliveParams(...)) + +// 2. Create client +client := pb.NewNotifierServiceClient(conn) + +// 3. Add auth to context +ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+apiKey) + +// 4. Make request +resp, err := client.SendNotification(ctx, req) +``` + +That's it! You're ready to integrate with the Notifier service. 🚀 diff --git a/docs/CLIENT_INTEGRATION_PROMPT.md b/docs/CLIENT_INTEGRATION_PROMPT.md new file mode 100644 index 0000000..047b4ff --- /dev/null +++ b/docs/CLIENT_INTEGRATION_PROMPT.md @@ -0,0 +1,344 @@ +# Notifier Service Client Integration Prompt + +**Use this prompt with Claude or another AI assistant when building a client application that integrates with the Notifier service.** + +--- + +## Prompt for AI Assistant + +I need to build a client application that integrates with a notification service using gRPC. Please help me implement a production-ready client that follows best practices for performance, security, and reliability. + +### Service Details + +**Protocol:** gRPC (HTTP/2) +**Service Address:** `notifier-grpc:50051` (Kubernetes internal DNS) +**Authentication:** API Key via gRPC metadata +**API Key Format:** `nk_` prefix followed by 64 hex characters (e.g., `nk_abc123...`) + +### Authentication Requirements + +1. **API Key Delivery:** + - Send API key in gRPC metadata with EVERY request + - Two accepted header formats: + - `authorization: Bearer nk_your_api_key_here` (preferred) + - `x-api-key: nk_your_api_key_here` + +2. **Per-Request Authentication:** + - The service validates the API key on every RPC call + - Authentication is NOT done at connection time + - This means you can (and should) use long-lived connections + +3. **Rate Limiting:** + - Rate limits are enforced per API key + - Default: 100 requests per minute (configurable per key) + - Rate limit exceeded returns: `codes.ResourceExhausted` + - Implement exponential backoff retry for rate limit errors + +4. **Credential Management:** + - Load API key from environment variable: `NOTIFIER_API_KEY` + - NEVER hardcode API keys in source code + - NEVER log the full API key (mask it: `nk_abc...xyz`) + - Store in Kubernetes Secret if deploying to k8s + +### Connection Management Requirements + +**CRITICAL:** Use long-lived gRPC connections for optimal performance. + +1. **Connection Lifecycle:** + - Create ONE connection at application startup + - Reuse the connection for ALL requests + - Close the connection only when application exits + - Do NOT create a new connection for each request + +2. **Keepalive Configuration:** + - Configure client keepalive to prevent connection timeouts + - Recommended settings: + - Time: 10 seconds (ping every 10s of inactivity) + - Timeout: 3 seconds (wait for ping response) + - PermitWithoutStream: true (allow pings when idle) + +3. **Why This Matters:** + - Connection reuse: 10x faster (1.5ms vs 15ms per request) + - Eliminates TCP + TLS handshake overhead (10-15ms per request) + - HTTP/2 multiplexing: handle multiple concurrent requests on one connection + - Lower resource usage: one connection vs thousands + +### Error Handling Requirements + +1. **Handle These Error Codes:** + - `codes.Unauthenticated` - Invalid or missing API key → Log error, don't retry + - `codes.ResourceExhausted` - Rate limit exceeded → Retry with exponential backoff + - `codes.InvalidArgument` - Bad request payload → Log error, don't retry + - `codes.Unavailable` - Service temporarily unavailable → Retry with backoff + - `codes.DeadlineExceeded` - Request timeout → Retry (may be transient) + +2. **Retry Strategy:** + - Implement exponential backoff: 100ms, 200ms, 400ms, 800ms + - Maximum 3-5 retry attempts for retryable errors + - Only retry: rate limits, unavailable, deadline exceeded + - Do NOT retry: authentication errors, invalid arguments + +3. **Context Timeouts:** + - Set reasonable timeout for each request (5-30 seconds) + - Use `context.WithTimeout()` for every RPC call + - Allow timeout to be configurable + +### Code Structure Requirements + +1. **Client Structure:** + ``` + NotifierClient struct: + - conn: *grpc.ClientConn (long-lived) + - client: pb.NotifierServiceClient + - apiKey: string (loaded from env) + - logger: logging interface + ``` + +2. **Required Methods:** + - `NewNotifierClient(address, apiKey string) (*NotifierClient, error)` - Initialize + - `SendNotification(ctx, request) (response, error)` - Send single notification + - `SendBatchNotifications(ctx, requests) (responses, error)` - Send batch + - `HealthCheck(ctx) error` - Verify connection health + - `Close() error` - Clean up connection + +3. **Initialization Pattern:** + - Create client as singleton at application startup + - Use `sync.Once` to ensure only one instance + - Defer `Close()` in main() to ensure cleanup + +### Observability Requirements + +1. **Logging:** + - Log connection establishment (info level) + - Log authentication failures (warn level) + - Log rate limit exceeded (warn level) + - Log successful sends (debug level) + - NEVER log the full API key (mask it) + +2. **Metrics (if using Prometheus):** + - Counter: `notifier_requests_total` (labels: status, method) + - Counter: `notifier_requests_failed_total` (labels: error_code) + - Histogram: `notifier_request_duration_seconds` + - Counter: `notifier_rate_limit_errors_total` + +3. **Health Checks:** + - Implement background health check goroutine + - Check every 30-60 seconds + - Call the service's HealthCheck RPC + - Log warnings if health checks fail + +### Security Requirements + +1. **Credential Security:** + - Load API key ONLY from environment variables + - Never commit API keys to version control + - Use `.gitignore` to exclude any files with credentials + - Implement a `String()` method that masks the API key for logging + +2. **TLS Configuration:** + - For local development: `grpc.WithInsecure()` is acceptable + - For production: Use TLS with proper certificate validation + - For Kubernetes: Internal communication may use insecure (cluster network is trusted) + +3. **Graceful Degradation:** + - If notifier service is unavailable, application should continue + - Log notification failures but don't crash the application + - Consider implementing a circuit breaker pattern for resilience + +### Testing Requirements + +1. **Unit Tests:** + - Mock the gRPC client interface + - Test retry logic with rate limit errors + - Test context timeout handling + - Test credential masking in logs + +2. **Integration Tests:** + - Test actual connection to notifier service (if available) + - Test authentication with valid and invalid keys + - Test rate limiting behavior + +### Kubernetes Deployment Requirements (if applicable) + +1. **Environment Variables:** + ```yaml + env: + - name: NOTIFIER_API_KEY + valueFrom: + secretKeyRef: + name: notifier-credentials + key: api-key + - name: NOTIFIER_ADDRESS + value: "notifier-grpc:50051" + ``` + +2. **Secrets Configuration:** + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: notifier-credentials + type: Opaque + stringData: + api-key: nk_your_actual_api_key_here + ``` + +### Available gRPC Methods + +The NotifierService provides these RPC methods: + +1. **SendNotification** - Send a single notification + - Request: `SendNotificationRequest` + - Response: `SendNotificationResponse` (contains notification_id) + +2. **SendBatchNotifications** - Send multiple notifications at once + - Request: `SendBatchNotificationsRequest` + - Response: `SendBatchNotificationsResponse` (contains notification_ids) + +3. **GetNotification** - Retrieve notification status by ID + - Request: `GetNotificationRequest` (notification_id) + - Response: `GetNotificationResponse` + +4. **ListNotifications** - List recent notifications + - Request: `ListNotificationsRequest` (pagination) + - Response: `ListNotificationsResponse` + +5. **CancelNotification** - Cancel a pending notification + - Request: `CancelNotificationRequest` (notification_id) + - Response: `CancelNotificationResponse` + +6. **RetryNotification** - Retry a failed notification + - Request: `RetryNotificationRequest` (notification_id) + - Response: `RetryNotificationResponse` + +7. **GetStats** - Get service statistics + - Request: `GetStatsRequest` + - Response: `GetStatsResponse` + +8. **GetNotifiers** - List available notifier types and accounts + - Request: `GetNotifiersRequest` + - Response: `GetNotifiersResponse` + +9. **HealthCheck** - Verify service health + - Request: `HealthCheckRequest` + - Response: `HealthCheckResponse` + +### Notification Types Supported + +- `NOTIFICATION_TYPE_EMAIL` - Email notifications +- `NOTIFICATION_TYPE_SLACK` - Slack messages +- `NOTIFICATION_TYPE_NTFY` - Ntfy.sh push notifications +- `NOTIFICATION_TYPE_STDOUT` - Console output (development only) + +### Request Example Structure + +```protobuf +message SendNotificationRequest { + NotificationType type = 1; // Required: email, slack, ntfy, stdout + string subject = 2; // Required: notification subject/title + string body = 3; // Required: notification body/content + repeated string recipients = 4; // Required: email addresses, slack channels, etc. + Priority priority = 5; // Optional: normal, high, critical + string account = 6; // Optional: specific notifier account to use + map metadata = 7; // Optional: additional metadata +} +``` + +### Implementation Goals + +Please implement a client that: + +1. ✅ Uses a single long-lived gRPC connection +2. ✅ Sends API key in metadata with every request +3. ✅ Implements exponential backoff retry for rate limits +4. ✅ Loads credentials from environment variables +5. ✅ Uses context with timeout for all requests +6. ✅ Implements keepalive to prevent connection timeouts +7. ✅ Includes proper error handling for all error codes +8. ✅ Logs important events without exposing credentials +9. ✅ Provides a health check mechanism +10. ✅ Includes graceful shutdown (connection cleanup) +11. ✅ Is production-ready with proper error handling and logging +12. ✅ Includes basic unit tests + +### Example Usage Pattern + +The client should be usable like this: + +```go +// Initialize once at startup +client, err := notifier.NewClient( + os.Getenv("NOTIFIER_ADDRESS"), + os.Getenv("NOTIFIER_API_KEY"), +) +if err != nil { + log.Fatal(err) +} +defer client.Close() + +// Use throughout application lifetime +ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) +defer cancel() + +resp, err := client.SendNotification(ctx, ¬ifier.SendRequest{ + Type: notifier.TypeEmail, + Subject: "Important Alert", + Body: "This is a critical notification", + Recipients: []string{"admin@example.com"}, + Priority: notifier.PriorityHigh, +}) + +if err != nil { + log.Printf("Failed to send notification: %v", err) + // Application continues - notification failure is not fatal + return +} + +log.Printf("Notification sent successfully: %s", resp.NotificationID) +``` + +### Additional Context + +- This is a microservices architecture running in Kubernetes +- The notifier service handles routing to multiple notification backends (SMTP, Slack, Ntfy) +- The client may need to send notifications from multiple goroutines concurrently +- High reliability is important, but notification failures should not crash the application +- The API key has a rate limit of 100 requests per minute (may vary per key) + +### Code Generation Note + +If you need the protobuf definitions, they can be generated from the service's proto files located at `api/grpc/proto/notifier.proto` in the notifier service repository. + +--- + +## Usage Instructions + +**Copy the prompt above and provide it to your AI assistant when building the client. The prompt includes:** + +- All authentication requirements and formats +- Connection management best practices +- Error handling strategies +- Security requirements +- Observability guidelines +- Complete API method listing +- Production-ready patterns +- Kubernetes deployment configuration + +**The AI assistant should generate:** +- Complete, production-ready client code +- Proper connection pooling and keepalive +- Comprehensive error handling +- Secure credential management +- Health check implementation +- Unit tests +- Usage documentation + +**Review the generated code for:** +- ✅ Single long-lived connection (not creating connections per request) +- ✅ API key sent in metadata with every RPC +- ✅ Keepalive configuration present +- ✅ Exponential backoff retry logic +- ✅ No hardcoded credentials +- ✅ Proper context timeouts +- ✅ Error handling for all gRPC error codes +- ✅ Graceful shutdown (Close() method) diff --git a/docs/CORS_IMPLEMENTATION.md b/docs/CORS_IMPLEMENTATION.md new file mode 100644 index 0000000..12866e9 --- /dev/null +++ b/docs/CORS_IMPLEMENTATION.md @@ -0,0 +1,255 @@ +# CORS Implementation - Security Enhancement + +## Overview + +This document describes the implementation of a secure CORS (Cross-Origin Resource Sharing) configuration system that replaces the previous wildcard (`*`) configuration with an explicit origin whitelist to prevent CSRF attacks. + +## Implementation Summary + +### 1. CORSConfig Structure (api/rest/router.go) + +Created a comprehensive `CORSConfig` struct with the following fields: + +```go +type CORSConfig struct { + AllowedOrigins []string // Whitelist of allowed origins + AllowedMethods []string // Allowed HTTP methods + AllowedHeaders []string // Allowed HTTP headers + AllowCredentials bool // Whether to allow credentials + MaxAge int // Cache duration for preflight responses +} +``` + +**Key Security Features:** +- Wildcards (`*`) are explicitly NOT supported +- Origins must be explicitly whitelisted +- Credentials can only be enabled with specific origins + +### 2. CORS Middleware Implementation + +The `newCORSMiddleware()` function implements: + +- **Origin Validation**: Checks incoming `Origin` header against whitelist +- **Exact Match Required**: Only sets CORS headers if origin is in whitelist +- **Never Uses Wildcard**: Always returns the exact origin, never `*` +- **Preflight Handling**: Properly handles OPTIONS requests +- **Configurable Headers**: All CORS headers are configurable + +**Code Location:** `api/rest/router.go:98-149` + +### 3. Configuration System + +#### Config Structure (internal/config/config.go) + +Added `CORSConfig` to the main application configuration: + +```go +type CORSConfig struct { + AllowedOrigins []string `mapstructure:"allowed_origins"` + AllowedMethods []string `mapstructure:"allowed_methods"` + AllowedHeaders []string `mapstructure:"allowed_headers"` + AllowCredentials bool `mapstructure:"allow_credentials"` + MaxAge int `mapstructure:"max_age"` +} +``` + +#### Default Values (internal/config/config.go:205-210) + +```go +AllowedOrigins: []string{} // Empty by default +AllowedMethods: []string{"GET", "POST", "OPTIONS", "DELETE"} // Standard REST methods +AllowedHeaders: []string{"Content-Type", "Authorization"} // Common headers +AllowCredentials: false // Disabled by default +MaxAge: 3600 // 1 hour +``` + +### 4. Configuration Validation (internal/config/config.go:263-282) + +Implemented `validateCORS()` that enforces: + +1. **Wildcard Rejection**: `*` is not allowed in `allowed_origins` +2. **Origin Format Validation**: Origins must start with `http://` or `https://` +3. **Credentials Validation**: `allow_credentials` requires at least one origin + +**Example Error Messages:** +- `"wildcard (*) is not allowed in CORS allowed_origins for security reasons - specify exact origins instead"` +- `"invalid origin format: example.com - origins must start with http:// or https://"` +- `"allow_credentials is enabled but no origins are allowed - this configuration is ineffective"` + +### 5. Updated Server Initialization (cmd/server/main.go:288-326) + +The `startRESTServer()` function now: + +1. Converts config CORS to `rest.CORSConfig` +2. Logs CORS configuration on startup +3. Warns if no origins are configured +4. Passes CORS config to router + +**Example Log Output:** +``` +CORS enabled for origins: [http://localhost:3000 http://localhost:8080] +``` + +Or: +``` +CORS has no allowed origins configured - all cross-origin requests will be blocked +``` + +### 6. Configuration Example (config.yaml:112-146) + +#### Development Configuration +```yaml +cors: + allowed_origins: + - "http://localhost:3000" # React/Next.js dev port + - "http://localhost:8080" # Vue/Angular dev port + - "http://localhost:5173" # Vite dev server + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: false + max_age: 3600 +``` + +#### Production Configuration (Example) +```yaml +cors: + allowed_origins: + - "https://app.example.com" + - "https://dashboard.example.com" + - "https://api-docs.example.com" + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: true # Enable for auth tokens + max_age: 3600 +``` + +## Test Coverage + +### 1. CORS Middleware Tests (api/rest/cors_test.go) + +**Test Cases:** +- ✅ `TestCORSMiddleware_AllowedOrigin`: Verifies allowed origins are accepted +- ✅ `TestCORSMiddleware_BlockedOrigin`: Verifies non-whitelisted origins are rejected +- ✅ `TestCORSMiddleware_PreflightRequest`: Tests OPTIONS preflight handling +- ✅ `TestCORSMiddleware_Credentials`: Verifies credential header handling +- ✅ `TestCORSMiddleware_NoWildcard`: Ensures wildcard is never returned +- ✅ `TestCORSMiddleware_EmptyConfig`: Tests secure default (no origins) +- ✅ `TestCORSMiddleware_MaxAge`: Tests cache duration configuration +- ✅ `TestDefaultCORSConfig`: Verifies default configuration values + +**Total: 8 test functions, 21 test cases** + +### 2. CORS Validation Tests (internal/config/cors_test.go) + +**Test Cases:** +- ✅ `TestValidateCORS_WildcardRejection`: Wildcard origins are rejected +- ✅ `TestValidateCORS_InvalidOriginFormat`: Invalid origin formats are rejected +- ✅ `TestValidateCORS_CredentialsWithoutOrigins`: Credentials require origins +- ✅ `TestValidateCORS_ValidConfigurations`: Valid configs are accepted +- ✅ `TestValidateCORS_MultipleOrigins`: Multiple origins with wildcard rejected +- ✅ `TestValidateCORS_EdgeCases`: Edge cases handled correctly + +**Total: 6 test functions, 16 test cases** + +## Security Improvements + +### Before +```go +w.Header().Set("Access-Control-Allow-Origin", "*") // ❌ CSRF Vulnerability +``` + +### After +```go +// Only set CORS headers if origin is in whitelist +if allowed { + w.Header().Set("Access-Control-Allow-Origin", origin) // ✅ Exact origin +} +// Otherwise, no CORS headers are set (browser blocks the response) +``` + +## Security Guarantees + +1. **No Wildcard**: The system makes it impossible to configure wildcard CORS +2. **Explicit Whitelist**: All allowed origins must be explicitly configured +3. **Validation at Startup**: Invalid configurations are rejected before the server starts +4. **Default Secure**: Empty origin list by default (most restrictive) +5. **Environment-Specific**: Different configs for dev/staging/production + +## Migration Guide + +### For Development + +Update your `config.yaml` to include localhost origins: + +```yaml +cors: + allowed_origins: + - "http://localhost:3000" + - "http://localhost:8080" +``` + +### For Production + +Configure your production origins: + +```yaml +cors: + allowed_origins: + - "https://yourdomain.com" + - "https://app.yourdomain.com" + allow_credentials: true +``` + +### Environment Variables + +You can also configure CORS via environment variables: + +```bash +export NOTIFIER_CORS_ALLOWED_ORIGINS="https://app.example.com,https://dashboard.example.com" +export NOTIFIER_CORS_ALLOW_CREDENTIALS=true +``` + +## Acceptance Criteria + +✅ **CORS whitelist fully configurable** - via config.yaml or environment variables +✅ **Wildcard configuration is impossible** - validation rejects `*` at startup +✅ **Security headers properly set** - only for whitelisted origins +✅ **Environment-specific configs work** - examples provided for dev/prod +✅ **No CSRF vulnerability** - wildcard eliminated, exact origins only +✅ **Comprehensive tests** - 37 test cases covering all scenarios + +## Files Modified + +1. `api/rest/router.go` - CORS config struct and middleware +2. `api/rest/cors_test.go` - CORS middleware tests (NEW) +3. `internal/config/config.go` - CORS configuration structure +4. `internal/config/cors_test.go` - CORS validation tests (NEW) +5. `cmd/server/main.go` - Server initialization with CORS config +6. `config.yaml` - Example CORS configuration +7. `internal/auth/bootstrap.go` - Fixed unrelated string formatting bug +8. `internal/auth/keystore_hybrid.go` - Fixed unrelated rate limiter signature + +## Additional Fixes + +While implementing CORS, also fixed pre-existing build errors: +- String multiplication in `bootstrap.go` (changed to `strings.Repeat`) +- Rate limiter signature mismatch in `keystore_hybrid.go` +- Missing parameter in `service_retention_test.go` + +## References + +- **OWASP CORS**: https://owasp.org/www-community/attacks/csrf +- **MDN CORS**: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS +- **RFC 6454**: The Web Origin Concept diff --git a/docs/CORS_KUBERNETES_GUIDE.md b/docs/CORS_KUBERNETES_GUIDE.md new file mode 100644 index 0000000..507087e --- /dev/null +++ b/docs/CORS_KUBERNETES_GUIDE.md @@ -0,0 +1,357 @@ +# CORS Configuration for Kubernetes Deployments + +## Important: CORS Only Affects REST API, Not gRPC + +**Key Point:** CORS (Cross-Origin Resource Sharing) is a **browser security mechanism** that only applies to HTTP/REST APIs accessed from web browsers. It does **NOT** affect gRPC communication. + +## Service-to-Service Communication (gRPC) + +### ✅ No CORS Configuration Needed + +For services running in the same Kubernetes cluster communicating via gRPC: + +**You don't need to configure CORS at all!** + +Here's why: + +1. **gRPC uses HTTP/2** - CORS is not enforced +2. **Server-to-Server** - CORS only applies to browser-based requests +3. **No Origin header** - Backend services don't send Origin headers + +### gRPC Client Connection Example + +```go +// Service A connecting to notifier service via gRPC +conn, err := grpc.Dial("notifier-grpc.default.svc.cluster.local:50051", + grpc.WithInsecure()) +if err != nil { + log.Fatalf("Failed to connect: %v", err) +} +defer conn.Close() + +client := pb.NewNotifierServiceClient(conn) +``` + +**No CORS configuration required** ✅ + +The gRPC service runs on port `50051` (as defined in your k8s/service.yaml) and is accessible at: +- From same namespace: `notifier-grpc:50051` +- From other namespace: `notifier-grpc.default.svc.cluster.local:50051` +- Full DNS: `notifier-grpc.default.svc.cluster.local:50051` + +## When You DO Need CORS Configuration + +CORS configuration is **only** needed when: + +1. **Web browsers** access the REST API (port 8080) +2. The web app is served from a **different origin** than the API +3. The request goes through the **REST API**, not gRPC + +### Common Scenarios + +| Scenario | Needs CORS? | Reason | +|----------|-------------|---------| +| Backend service → gRPC API | ❌ No | Not a browser, uses gRPC | +| Backend service → REST API | ❌ No | Not a browser | +| Web app → gRPC (grpc-web) | ✅ Yes | Browser-based, needs CORS | +| Web app → REST API | ✅ Yes | Browser-based, needs CORS | +| CLI tool → REST API | ❌ No | Not a browser | +| Postman/curl → REST API | ❌ No | Not a browser | + +## CORS Configuration for Web Applications + +If you have a web frontend that calls the REST API, you need to configure CORS. + +### Scenario 1: Frontend in Same Cluster + +**Example:** Frontend at `https://app.example.com`, API at `https://notifier.example.com` + +Update your ConfigMap (k8s/configmap.yaml): + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: notifier-config +data: + config.yaml: | + # ... existing config ... + + cors: + allowed_origins: + - "https://app.example.com" # Your frontend domain + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: true # If your frontend sends auth tokens + max_age: 3600 +``` + +### Scenario 2: Multiple Frontends + +```yaml +cors: + allowed_origins: + - "https://app.example.com" # Main app + - "https://dashboard.example.com" # Admin dashboard + - "https://mobile.example.com" # Mobile web app + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: true + max_age: 3600 +``` + +### Scenario 3: Development Environment + +For local development (frontend at http://localhost:3000): + +```yaml +cors: + allowed_origins: + - "http://localhost:3000" # React/Next.js dev server + - "http://localhost:8080" # Vue/Angular dev server + - "http://localhost:5173" # Vite dev server + - "https://app.example.com" # Production frontend + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: false + max_age: 3600 +``` + +### Scenario 4: No Web Frontend (Backend Services Only) + +If you only have backend services using gRPC: + +```yaml +cors: + # Empty or minimal config - no origins needed + allowed_origins: [] # No browser clients + allowed_methods: + - "GET" + - "POST" + allowed_headers: + - "Content-Type" +``` + +The REST API will still work for non-browser clients (like curl, Postman, or backend HTTP clients), but browsers will be blocked unless their origin is whitelisted. + +## Complete Kubernetes Configuration Example + +Here's your updated k8s/configmap.yaml with CORS: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: notifier-config + labels: + app: notifier +data: + config.yaml: | + server: + grpc_port: 50051 + rest_port: 8080 + host: "0.0.0.0" + mode: "both" + + queue: + type: "local" + max_size: 10000 + worker_count: 10 + retry_attempts: 3 + retry_backoff: "exponential" + local: + buffer_size: 1000 + persist_to_disk: false + + notifiers: + stdout: true + + logging: + level: "info" + format: "json" + output_path: "stdout" + + metrics: + enabled: true + port: 9090 + path: "/metrics" + prometheus_enabled: true + + health_check: + enabled: true + port: 8081 + path: "/health" + interval: 30 + + # CORS configuration for REST API + # Only needed if web browsers will access the REST API + cors: + # Add your frontend domains here + # For backend-only services, leave empty + allowed_origins: [] + + # Or if you have a web frontend: + # allowed_origins: + # - "https://app.example.com" + + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: false + max_age: 3600 +``` + +## Environment-Specific Configuration + +You can also use environment variables for different environments: + +### Development ConfigMap + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: notifier-config-dev + namespace: development +data: + NOTIFIER_CORS_ALLOWED_ORIGINS: "http://localhost:3000,http://localhost:8080" + NOTIFIER_CORS_ALLOW_CREDENTIALS: "false" +``` + +### Production ConfigMap + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: notifier-config-prod + namespace: production +data: + NOTIFIER_CORS_ALLOWED_ORIGINS: "https://app.example.com,https://dashboard.example.com" + NOTIFIER_CORS_ALLOW_CREDENTIALS: "true" +``` + +## Troubleshooting + +### "CORS error" in Browser Console + +**Symptom:** +``` +Access to fetch at 'https://notifier.example.com/api/v1/notifications' +from origin 'https://app.example.com' has been blocked by CORS policy +``` + +**Solution:** +Add `https://app.example.com` to `allowed_origins` in your ConfigMap. + +### Backend Service Can't Connect + +**Symptom:** +```go +// Service in cluster trying to connect +conn, err := grpc.Dial("notifier-grpc:50051", grpc.WithInsecure()) +// Error: connection refused +``` + +**Solution:** +This is NOT a CORS issue. Check: +1. Service name is correct (`notifier-grpc`) +2. Port is correct (`50051`) +3. Service is running: `kubectl get pods -l app=notifier` +4. Service endpoints: `kubectl get endpoints notifier-grpc` + +### REST API Returns 200 but No CORS Headers + +**Symptom:** +Browser blocks the response even though the API returns 200 OK. + +**Solution:** +The origin is not in the whitelist. Check: +1. Origin is exactly matching (including protocol and port) +2. ConfigMap has been updated +3. Pod has been restarted to pick up new config: + ```bash + kubectl rollout restart deployment notifier + ``` + +## Quick Reference + +### For Backend Services Only (Recommended) +```yaml +cors: + allowed_origins: [] # Empty - no browser access needed +``` + +### For Web Frontend + Backend Services +```yaml +cors: + allowed_origins: + - "https://your-frontend-domain.com" + allow_credentials: true # If using auth tokens +``` + +### Applying Changes + +After updating the ConfigMap: + +```bash +# Update ConfigMap +kubectl apply -f k8s/configmap.yaml + +# Restart pods to pick up new config +kubectl rollout restart deployment notifier + +# Verify pods are running +kubectl get pods -l app=notifier + +# Check logs for CORS configuration +kubectl logs -l app=notifier | grep CORS +``` + +You should see: +``` +CORS enabled for origins: [https://app.example.com] +``` + +Or if no origins configured: +``` +CORS has no allowed origins configured - all cross-origin requests will be blocked +``` + +## Summary + +**For your use case (services in Kubernetes using gRPC):** + +✅ **You don't need to configure CORS at all!** + +CORS only applies to web browsers accessing the REST API. Your backend services communicating via gRPC are completely unaffected by CORS configuration. + +**Only add CORS configuration if:** +- You have a web frontend (React, Vue, Angular, etc.) +- That web frontend calls the REST API (not gRPC) +- The frontend is served from a different domain than the API + +For purely backend service-to-service communication in Kubernetes, CORS is irrelevant. diff --git a/docs/GRPC_CONNECTION_OPTIMIZATION.md b/docs/GRPC_CONNECTION_OPTIMIZATION.md new file mode 100644 index 0000000..fbb2b9c --- /dev/null +++ b/docs/GRPC_CONNECTION_OPTIMIZATION.md @@ -0,0 +1,607 @@ +# gRPC Connection Optimization Guide + +## TL;DR + +**Yes, you can and should use long-lived gRPC connections!** + +- ✅ Single connection can handle thousands of concurrent requests +- ✅ Authentication happens **per-request**, not per-connection +- ✅ Connection reuse eliminates TCP/TLS handshake overhead +- ✅ HTTP/2 multiplexing allows concurrent RPCs on one connection +- ✅ Built-in keepalive prevents connection timeouts + +## How gRPC Authentication Works + +### Key Point: Auth is Per-Request, Not Per-Connection + +Your current implementation authenticates **each RPC call**, not the connection itself. This means: + +1. Client establishes a **long-lived connection** (once) +2. Client sends **API key in metadata** with each request +3. Server validates the key for **every RPC call** +4. Connection stays open for multiple requests + +``` +Connection Lifecycle: +┌─────────────────────────────────────────────────────────────┐ +│ TCP Connection (persistent) │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ TLS Handshake (once) │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Request 1: Authorization: Bearer nk_abc... → Validated │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Request 2: Authorization: Bearer nk_abc... → Validated │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Request 3: Authorization: Bearer nk_abc... → Validated │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ ... (connection stays open) │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Authentication Overhead Analysis + +**Per-Connection (One-Time):** +- TCP handshake: ~1-2ms (3-way handshake) +- TLS handshake: ~5-10ms (certificate exchange, key agreement) +- **Total: ~10-15ms once** + +**Per-Request (Every Call):** +- API key validation: ~0.1-1ms (in-memory lookup) +- Rate limit check: ~0.1ms (in-memory counter) +- **Total: ~0.2-1ms per request** + +**With Connection Reuse:** +- First request: 10-15ms (connection) + 1ms (auth) = **11-16ms** +- Subsequent requests: **1ms** (only auth, no connection setup) + +**Without Connection Reuse (reconnecting each time):** +- Every request: 10-15ms (connection) + 1ms (auth) = **11-16ms** + +**Savings: 10-15ms per request after the first one!** + +## Recommended Client Pattern + +### Basic Long-Lived Connection + +```go +package main + +import ( + "context" + "log" + "time" + + pb "github.com/igodwin/notifier/api/grpc/pb" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/metadata" +) + +type NotifierClient struct { + conn *grpc.ClientConn + client pb.NotifierServiceClient + apiKey string +} + +func NewNotifierClient(address, apiKey string) (*NotifierClient, error) { + // Establish long-lived connection + conn, err := grpc.Dial(address, + grpc.WithTransportCredentials(insecure.NewCredentials()), + + // Connection pool settings + grpc.WithDefaultCallOptions( + grpc.MaxCallRecvMsgSize(4*1024*1024), // 4MB + grpc.MaxCallSendMsgSize(4*1024*1024), + ), + + // Keepalive settings + grpc.WithKeepaliveParams(keepalive.ClientParameters{ + Time: 10 * time.Second, // Send keepalive ping every 10s + Timeout: 3 * time.Second, // Wait 3s for ping ack + PermitWithoutStream: true, // Allow pings when no streams + }), + ) + if err != nil { + return nil, err + } + + return &NotifierClient{ + conn: conn, + client: pb.NewNotifierServiceClient(conn), + apiKey: apiKey, + }, nil +} + +func (nc *NotifierClient) SendNotification(ctx context.Context, req *pb.SendNotificationRequest) (*pb.SendNotificationResponse, error) { + // Add API key to metadata for THIS request + ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+nc.apiKey) + + // Make RPC call - reuses existing connection + return nc.client.SendNotification(ctx, req) +} + +func (nc *NotifierClient) Close() error { + return nc.conn.Close() +} + +func main() { + // Create client with long-lived connection + client, err := NewNotifierClient("notifier-grpc:50051", "nk_your_api_key") + if err != nil { + log.Fatal(err) + } + defer client.Close() // Close when application exits + + // Reuse client for multiple requests + for i := 0; i < 1000; i++ { + ctx := context.Background() + resp, err := client.SendNotification(ctx, &pb.SendNotificationRequest{ + Type: pb.NotificationType_NOTIFICATION_TYPE_EMAIL, + Subject: "Test notification", + Body: "This is a test", + Recipients: []string{"user@example.com"}, + }) + + if err != nil { + log.Printf("Request %d failed: %v", i, err) + continue + } + + log.Printf("Request %d succeeded: %s", i, resp.NotificationId) + } + + // Connection is closed when main() exits +} +``` + +### Advanced: Connection Pool for High Concurrency + +For extremely high throughput, you can create multiple connections: + +```go +package main + +import ( + "context" + "sync" + + pb "github.com/igodwin/notifier/api/grpc/pb" + "google.golang.org/grpc" + "google.golang.org/grpc/keepalive" +) + +type NotifierPool struct { + connections []*grpc.ClientConn + clients []pb.NotifierServiceClient + apiKey string + current uint32 + mu sync.Mutex +} + +func NewNotifierPool(address, apiKey string, poolSize int) (*NotifierPool, error) { + pool := &NotifierPool{ + connections: make([]*grpc.ClientConn, poolSize), + clients: make([]pb.NotifierServiceClient, poolSize), + apiKey: apiKey, + } + + // Create multiple connections + for i := 0; i < poolSize; i++ { + conn, err := grpc.Dial(address, + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithKeepaliveParams(keepalive.ClientParameters{ + Time: 10 * time.Second, + Timeout: 3 * time.Second, + PermitWithoutStream: true, + }), + ) + if err != nil { + // Clean up any connections already created + pool.Close() + return nil, err + } + + pool.connections[i] = conn + pool.clients[i] = pb.NewNotifierServiceClient(conn) + } + + return pool, nil +} + +func (np *NotifierPool) getClient() pb.NotifierServiceClient { + // Round-robin connection selection + np.mu.Lock() + defer np.mu.Unlock() + + idx := np.current % uint32(len(np.clients)) + np.current++ + return np.clients[idx] +} + +func (np *NotifierPool) SendNotification(ctx context.Context, req *pb.SendNotificationRequest) (*pb.SendNotificationResponse, error) { + // Add API key to metadata + ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+np.apiKey) + + // Get a client from pool (round-robin) + client := np.getClient() + + return client.SendNotification(ctx, req) +} + +func (np *NotifierPool) Close() error { + var firstErr error + for _, conn := range np.connections { + if conn != nil { + if err := conn.Close(); err != nil && firstErr == nil { + firstErr = err + } + } + } + return firstErr +} + +// Usage +func main() { + // Create pool with 4 connections + pool, err := NewNotifierPool("notifier-grpc:50051", "nk_your_api_key", 4) + if err != nil { + log.Fatal(err) + } + defer pool.Close() + + // Use pool concurrently from multiple goroutines + var wg sync.WaitGroup + for i := 0; i < 1000; i++ { + wg.Add(1) + go func(id int) { + defer wg.Done() + + resp, err := pool.SendNotification(context.Background(), &pb.SendNotificationRequest{ + Type: pb.NotificationType_NOTIFICATION_TYPE_EMAIL, + Subject: fmt.Sprintf("Notification %d", id), + Body: "Test", + Recipients: []string{"user@example.com"}, + }) + + if err != nil { + log.Printf("Request %d failed: %v", id, err) + return + } + + log.Printf("Request %d succeeded: %s", id, resp.NotificationId) + }(i) + } + + wg.Wait() +} +``` + +## Keepalive Configuration + +### Why Keepalive Matters + +In Kubernetes, idle connections may be terminated by: +- Load balancers (after 60-600 seconds) +- Network proxies +- Firewalls with connection tracking + +**Solution:** Send periodic keepalive pings + +### Client-Side Keepalive + +```go +grpc.WithKeepaliveParams(keepalive.ClientParameters{ + Time: 10 * time.Second, // Send ping every 10s of inactivity + Timeout: 3 * time.Second, // Wait 3s for ping response + PermitWithoutStream: true, // Send pings even when no active RPCs +}) +``` + +### Server-Side Keepalive (Already Configured in Your Server) + +Add to `cmd/server/main.go` in `startGRPCServer()`: + +```go +serverOpts = append(serverOpts, + grpc.KeepaliveParams(keepalive.ServerParameters{ + MaxConnectionIdle: 15 * time.Minute, // Close idle connections after 15m + MaxConnectionAge: 30 * time.Minute, // Force close after 30m + MaxConnectionAgeGrace: 5 * time.Second, // Allow 5s for RPCs to complete + Time: 5 * time.Second, // Send ping if idle for 5s + Timeout: 1 * time.Second, // Wait 1s for ping response + }), + grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{ + MinTime: 5 * time.Second, // Don't allow pings more often than 5s + PermitWithoutStream: true, // Allow pings when no streams + }), +) +``` + +## Performance Comparison + +### Scenario 1: Short-Lived Connections (Creating new connection for each request) + +``` +Request 1: 15ms (10ms connect + 5ms TLS + 1ms auth + 0.5ms RPC) +Request 2: 15ms (10ms connect + 5ms TLS + 1ms auth + 0.5ms RPC) +Request 3: 15ms (10ms connect + 5ms TLS + 1ms auth + 0.5ms RPC) +... +1000 requests: ~15,000ms (15 seconds) +``` + +### Scenario 2: Long-Lived Connection (Recommended) + +``` +Request 1: 15ms (10ms connect + 5ms TLS + 1ms auth + 0.5ms RPC) +Request 2: 1.5ms (1ms auth + 0.5ms RPC) +Request 3: 1.5ms (1ms auth + 0.5ms RPC) +... +1000 requests: ~1,515ms (1.5 seconds) +``` + +**Performance Improvement: 10x faster! (15s → 1.5s)** + +### Scenario 3: Connection Pool with 4 Connections + +``` +First 4 requests: 15ms each (connection setup) +Remaining 996: 1.5ms each (reuse connections) +1000 requests: ~1,554ms (1.5 seconds) +Handles concurrent load better +``` + +## Best Practices + +### 1. Connection Lifecycle Management + +```go +// Application-scoped client (singleton) +var ( + notifierClient *NotifierClient + once sync.Once +) + +func GetNotifierClient() *NotifierClient { + once.Do(func() { + client, err := NewNotifierClient( + os.Getenv("NOTIFIER_ADDRESS"), + os.Getenv("NOTIFIER_API_KEY"), + ) + if err != nil { + log.Fatalf("Failed to create notifier client: %v", err) + } + notifierClient = client + }) + return notifierClient +} + +// In main(): +func main() { + client := GetNotifierClient() + defer client.Close() + + // ... run application ... +} +``` + +### 2. Context with Timeout + +Always use context with timeout to prevent hanging requests: + +```go +func (nc *NotifierClient) SendNotification(ctx context.Context, req *pb.SendNotificationRequest) (*pb.SendNotificationResponse, error) { + // Set timeout for this request + ctx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + + // Add API key + ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+nc.apiKey) + + return nc.client.SendNotification(ctx, req) +} +``` + +### 3. Health Checks + +Periodically verify the connection is healthy: + +```go +func (nc *NotifierClient) HealthCheck(ctx context.Context) error { + ctx, cancel := context.WithTimeout(ctx, 3*time.Second) + defer cancel() + + // Add API key + ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+nc.apiKey) + + _, err := nc.client.HealthCheck(ctx, &pb.HealthCheckRequest{}) + return err +} + +// In background goroutine: +go func() { + ticker := time.NewTicker(30 * time.Second) + defer ticker.Stop() + + for range ticker.C { + if err := client.HealthCheck(context.Background()); err != nil { + log.Printf("Health check failed: %v", err) + // Consider reconnecting or alerting + } + } +}() +``` + +### 4. Graceful Reconnection + +Handle connection failures gracefully: + +```go +type ResilientNotifierClient struct { + address string + apiKey string + client *NotifierClient + mu sync.RWMutex +} + +func (rnc *ResilientNotifierClient) SendNotification(ctx context.Context, req *pb.SendNotificationRequest) (*pb.SendNotificationResponse, error) { + rnc.mu.RLock() + client := rnc.client + rnc.mu.RUnlock() + + resp, err := client.SendNotification(ctx, req) + if err != nil && isConnectionError(err) { + // Try to reconnect + log.Printf("Connection error, attempting reconnect: %v", err) + if err := rnc.reconnect(); err != nil { + return nil, fmt.Errorf("reconnection failed: %w", err) + } + + // Retry request with new connection + rnc.mu.RLock() + client = rnc.client + rnc.mu.RUnlock() + + return client.SendNotification(ctx, req) + } + + return resp, err +} + +func (rnc *ResilientNotifierClient) reconnect() error { + rnc.mu.Lock() + defer rnc.mu.Unlock() + + // Close old connection + if rnc.client != nil { + rnc.client.Close() + } + + // Create new connection + client, err := NewNotifierClient(rnc.address, rnc.apiKey) + if err != nil { + return err + } + + rnc.client = client + return nil +} +``` + +## Kubernetes Deployment Considerations + +### 1. Service Configuration + +Your services are already correctly configured for long-lived connections: + +```yaml +# k8s/service.yaml +apiVersion: v1 +kind: Service +metadata: + name: notifier-grpc +spec: + type: ClusterIP # ✅ Good: Stable internal endpoint + ports: + - port: 50051 + targetPort: grpc + protocol: TCP +``` + +### 2. Client Connection String + +```go +// Within same namespace +client, _ := NewNotifierClient("notifier-grpc:50051", apiKey) + +// From different namespace +client, _ := NewNotifierClient("notifier-grpc.default.svc.cluster.local:50051", apiKey) +``` + +### 3. Load Balancing + +Kubernetes service provides **connection-level** load balancing. For better **request-level** load balancing with long-lived connections, consider: + +**Option A: Client-Side Load Balancing** + +```go +import "google.golang.org/grpc/resolver" + +conn, err := grpc.Dial( + "dns:///notifier-grpc:50051", // DNS resolver + grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`), + grpc.WithTransportCredentials(insecure.NewCredentials()), +) +``` + +**Option B: Connection Pool** (shown earlier) + +## Rate Limiting Considerations + +### Your Current Implementation + +The rate limiter checks on **every request** (in `grpc_middleware.go:46-50`): + +```go +allowed, err := m.store.CheckRateLimit(apiKey) +if err != nil || !allowed { + return nil, status.Error(codes.ResourceExhausted, "Rate limit exceeded") +} +``` + +### Impact with Long-Lived Connections + +**No negative impact!** Rate limiting works the same: +- Each RPC call is checked independently +- Connection reuse doesn't bypass rate limits +- Rate limit is per API key, not per connection + +## Summary + +### ✅ DO: Use Long-Lived Connections + +```go +// Create once at application startup +client, _ := NewNotifierClient("notifier-grpc:50051", apiKey) +defer client.Close() + +// Reuse for all requests +for { + client.SendNotification(ctx, req) +} +``` + +**Benefits:** +- 10x faster (eliminates connection setup overhead) +- Lower latency (1.5ms vs 15ms per request) +- Fewer resources (one connection vs. thousands) +- Better throughput (HTTP/2 multiplexing) +- Automatic keepalive prevents timeouts + +### ❌ DON'T: Create Connection Per Request + +```go +// BAD: Don't do this! +for { + client, _ := NewNotifierClient("notifier-grpc:50051", apiKey) + client.SendNotification(ctx, req) + client.Close() +} +``` + +**Problems:** +- Slow (15ms per request) +- Wasteful (repeated TCP/TLS handshakes) +- Resource-intensive (thousands of connections) + +### Authentication Still Happens Per-Request + +- API key sent in metadata with every RPC +- Server validates on every call +- No security trade-off +- Just eliminates connection setup overhead + +**You get both: Maximum performance AND full security!** diff --git a/docs/QUICK_START_CLIENT.md b/docs/QUICK_START_CLIENT.md new file mode 100644 index 0000000..948133c --- /dev/null +++ b/docs/QUICK_START_CLIENT.md @@ -0,0 +1,319 @@ +# Quick Start: Notifier Client Integration + +**5-minute guide to integrating with the Notifier service** + +## Essential Requirements + +### 1. Connection: Use Long-Lived gRPC Connection ⚡ + +```go +// ✅ DO THIS: Create once, reuse forever +client, _ := NewNotifierClient("notifier-grpc:50051", apiKey) +defer client.Close() + +for i := 0; i < 10000; i++ { + client.SendNotification(ctx, req) // Reuse connection +} + +// ❌ DON'T DO THIS: Creating connection per request +for i := 0; i < 10000; i++ { + client, _ := NewNotifierClient("notifier-grpc:50051", apiKey) + client.SendNotification(ctx, req) + client.Close() // SLOW: 10x slower! +} +``` + +**Why?** Connection reuse is 10x faster (1.5ms vs 15ms per request) + +### 2. Authentication: Send API Key with Every Request 🔐 + +```go +import "google.golang.org/grpc/metadata" + +// Add API key to metadata for each request +ctx = metadata.AppendToOutgoingContext(ctx, + "authorization", "Bearer "+apiKey) + +resp, err := client.SendNotification(ctx, req) +``` + +**Two accepted formats:** +- `authorization: Bearer nk_your_api_key` (preferred) +- `x-api-key: nk_your_api_key` + +**API Key Format:** `nk_` + 64 hex characters (e.g., `nk_a1b2c3d4...`) + +### 3. Credentials: Load from Environment 🔒 + +```go +// ✅ DO THIS +apiKey := os.Getenv("NOTIFIER_API_KEY") + +// ❌ DON'T DO THIS +apiKey := "nk_abc123..." // NEVER hardcode! +``` + +**Kubernetes Secret:** +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: notifier-credentials +stringData: + api-key: nk_your_actual_key_here +--- +# In your deployment: +env: +- name: NOTIFIER_API_KEY + valueFrom: + secretKeyRef: + name: notifier-credentials + key: api-key +``` + +### 4. Keepalive: Prevent Connection Timeouts ⏰ + +```go +import "google.golang.org/grpc/keepalive" + +conn, err := grpc.Dial(address, + grpc.WithKeepaliveParams(keepalive.ClientParameters{ + Time: 10 * time.Second, // Ping every 10s + Timeout: 3 * time.Second, // Wait 3s for response + PermitWithoutStream: true, // Ping when idle + }), +) +``` + +### 5. Error Handling: Retry Only Specific Errors 🔄 + +```go +err := client.SendNotification(ctx, req) + +switch status.Code(err) { +case codes.ResourceExhausted: + // Rate limit exceeded - RETRY with backoff + time.Sleep(100 * time.Millisecond) + // Try again... + +case codes.Unauthenticated: + // Invalid API key - DON'T RETRY, log error + log.Errorf("Authentication failed: %v", err) + +case codes.InvalidArgument: + // Bad request - DON'T RETRY, fix request + log.Errorf("Invalid request: %v", err) + +case codes.Unavailable: + // Service down - RETRY with backoff + time.Sleep(100 * time.Millisecond) + // Try again... +} +``` + +**Retry Strategy:** +- Exponential backoff: 100ms → 200ms → 400ms → 800ms +- Max 3-5 attempts +- Only retry: `ResourceExhausted`, `Unavailable`, `DeadlineExceeded` + +### 6. Context Timeout: Always Set Timeout ⏱️ + +```go +// ✅ DO THIS: Set timeout for each request +ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) +defer cancel() + +resp, err := client.SendNotification(ctx, req) + +// ❌ DON'T DO THIS: No timeout +ctx := context.Background() +resp, err := client.SendNotification(ctx, req) // Could hang forever! +``` + +## Complete Minimal Example + +```go +package main + +import ( + "context" + "log" + "os" + "time" + + pb "github.com/igodwin/notifier/api/grpc/pb" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/keepalive" + "google.golang.org/grpc/metadata" +) + +type NotifierClient struct { + conn *grpc.ClientConn + client pb.NotifierServiceClient + apiKey string +} + +func NewNotifierClient(address, apiKey string) (*NotifierClient, error) { + // Create long-lived connection + conn, err := grpc.Dial(address, + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithKeepaliveParams(keepalive.ClientParameters{ + Time: 10 * time.Second, + Timeout: 3 * time.Second, + PermitWithoutStream: true, + }), + ) + if err != nil { + return nil, err + } + + return &NotifierClient{ + conn: conn, + client: pb.NewNotifierServiceClient(conn), + apiKey: apiKey, + }, nil +} + +func (nc *NotifierClient) SendNotification(ctx context.Context, req *pb.SendNotificationRequest) (*pb.SendNotificationResponse, error) { + // Add timeout + ctx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + + // Add API key to metadata + ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+nc.apiKey) + + // Make RPC call + return nc.client.SendNotification(ctx, req) +} + +func (nc *NotifierClient) Close() error { + return nc.conn.Close() +} + +func main() { + // Load credentials + address := os.Getenv("NOTIFIER_ADDRESS") + apiKey := os.Getenv("NOTIFIER_API_KEY") + + if address == "" { + address = "notifier-grpc:50051" + } + if apiKey == "" { + log.Fatal("NOTIFIER_API_KEY environment variable required") + } + + // Create client (once) + client, err := NewNotifierClient(address, apiKey) + if err != nil { + log.Fatalf("Failed to create client: %v", err) + } + defer client.Close() + + // Send notification + resp, err := client.SendNotification(context.Background(), &pb.SendNotificationRequest{ + Type: pb.NotificationType_NOTIFICATION_TYPE_EMAIL, + Subject: "Test Notification", + Body: "This is a test from the notifier client", + Recipients: []string{"user@example.com"}, + Priority: pb.Priority_PRIORITY_NORMAL, + }) + + if err != nil { + log.Fatalf("Failed to send notification: %v", err) + } + + log.Printf("Notification sent successfully: %s", resp.NotificationId) +} +``` + +## Service Connection Details + +**Address:** +- Same namespace: `notifier-grpc:50051` +- Different namespace: `notifier-grpc.default.svc.cluster.local:50051` + +**Rate Limit:** 100 requests/minute (default, configurable per key) + +**Available Methods:** +- `SendNotification` - Send single notification +- `SendBatchNotifications` - Send multiple notifications +- `GetNotification` - Get notification status +- `ListNotifications` - List recent notifications +- `CancelNotification` - Cancel pending notification +- `RetryNotification` - Retry failed notification +- `GetStats` - Get service statistics +- `GetNotifiers` - List available notifiers +- `HealthCheck` - Check service health + +**Notification Types:** +- `NOTIFICATION_TYPE_EMAIL` - Email via SMTP +- `NOTIFICATION_TYPE_SLACK` - Slack webhook +- `NOTIFICATION_TYPE_NTFY` - Ntfy.sh push notifications +- `NOTIFICATION_TYPE_STDOUT` - Console output (dev only) + +## Security Checklist + +- [ ] API key loaded from environment variable +- [ ] No hardcoded credentials in source code +- [ ] API key never logged in full (mask it: `nk_abc...xyz`) +- [ ] Kubernetes Secret created for API key +- [ ] Deployment configured to load secret as env var +- [ ] `.gitignore` excludes any credential files + +## Testing Your Integration + +```bash +# Set environment variables +export NOTIFIER_ADDRESS="notifier-grpc:50051" +export NOTIFIER_API_KEY="nk_your_api_key_here" + +# Run your application +go run main.go + +# Check logs for: +# ✅ "Notification sent successfully: " +# ❌ "Authentication failed" - check API key +# ❌ "Rate limit exceeded" - slow down requests +``` + +## Common Issues + +**"Unauthenticated" error:** +- Check API key format starts with `nk_` +- Verify key is 66 characters total (nk_ + 64 hex) +- Confirm key is active in notifier service +- Check you're adding it to metadata correctly + +**"Rate limit exceeded" error:** +- Implement exponential backoff retry +- Consider batching notifications +- Request higher rate limit for your key + +**Connection timeouts:** +- Add keepalive configuration +- Check network connectivity to `notifier-grpc:50051` +- Verify notifier service is running: `kubectl get pods -l app=notifier` + +**Slow performance:** +- Confirm you're reusing connection (not creating per request) +- Check keepalive is configured +- Verify you're using connection pooling for high concurrency + +## Next Steps + +For more details, see: +- `CLIENT_INTEGRATION_PROMPT.md` - Complete prompt for AI assistants +- `GRPC_CONNECTION_OPTIMIZATION.md` - Deep dive on performance +- `CLIENT_RECOMMENDATIONS.md` - Advanced patterns and best practices +- `AUTH.md` - Authentication system details + +## Getting Your API Key + +Contact your Notifier service administrator to: +1. Create an API key for your application +2. Set appropriate rate limits +3. Assign required roles for notifier access +4. Get the key in format: `nk_<64-hex-characters>` + +Store it securely and never commit it to version control! diff --git a/k8s/configmap.yaml b/k8s/configmap.yaml index 3fe1d54..1a2598f 100644 --- a/k8s/configmap.yaml +++ b/k8s/configmap.yaml @@ -66,3 +66,32 @@ data: port: 8081 path: "/health" interval: 30 + + # CORS (Cross-Origin Resource Sharing) configuration + # Only needed if web browsers will access the REST API + # For backend-only services using gRPC, CORS is not required + cors: + # Whitelist of allowed origins - wildcards (*) are NOT supported + # Leave empty if you don't have web browser clients + allowed_origins: [] + + # If you have a web frontend, add its domain(s) here: + # allowed_origins: + # - "https://app.example.com" + # - "https://dashboard.example.com" + + # For local development: + # allowed_origins: + # - "http://localhost:3000" + # - "http://localhost:8080" + + allowed_methods: + - "GET" + - "POST" + - "OPTIONS" + - "DELETE" + allowed_headers: + - "Content-Type" + - "Authorization" + allow_credentials: false + max_age: 3600