notifier/tests/e2e/README.md

E2E Tests for Notifier Service

This directory contains end-to-end (E2E) tests for the Notifier service using testcontainers for isolated Docker-based testing.

Overview

The E2E tests validate real-world scenarios using:

• testcontainers-go: Isolated containerized service instances
• REST client library: Type-safe API client (pkg/client)
• Real notifications: Actual in-memory notification processing

CRITICAL-1 Test Scenarios

The following tests validate the notification retention policy implementation:

1. TTL-Based Cleanup (TestCRITICAL1_TTLBasedCleanup)

• Purpose: Verify old notifications are removed after TTL expires
• Configuration: TTL=2s, check frequency=500ms
• Steps:
  1. Send notification
  2. Wait for TTL + cleanup interval
  3. Verify notification count decreased
• Expected Result: ✅ Old notifications cleaned up

2. Max Size Enforcement (TestCRITICAL1_MaxSizeEnforcement)

• Purpose: Verify max_size limit is enforced
• Configuration: max_size=5, TTL=24h
• Steps:
  1. Send 10 notifications
  2. Wait for cleanup
  3. Verify only 5 remain
• Expected Result: ✅ Excess notifications removed

3. Cleanup Disabled (TestCRITICAL1_CleanupDisabled)

• Purpose: Verify cleanup doesn't run when disabled
• Configuration: enabled=false
• Steps:
  1. Send 10 notifications
  2. Wait for time cleanup would run
  3. Verify all 10 still exist
• Expected Result: ✅ No notifications removed

4. Concurrent Sends (TestCRITICAL1_ConcurrentSends)

• Purpose: Verify concurrent client access is safe
• Configuration: Default retention
• Steps:
  1. Send 10 notifications concurrently
  2. Verify all succeeded
  3. Check stats
• Expected Result: ✅ All 10 notifications processed

5. Oldest Removed First (TestCRITICAL1_OldestRemovedFirst)

• Purpose: Verify oldest are removed when exceeding max_size
• Configuration: max_size=3
• Steps:
  1. Send 5 notifications with delays
  2. Wait for cleanup
  3. Verify oldest 2 removed
• Expected Result: ✅ Newest 3 remain

6. Memory Bounded (TestCRITICAL1_MemoryBounded)

• Purpose: Verify memory stays bounded over time
• Configuration: max_size=50, TTL=5s
• Steps:
  1. Send 30 notifications in batches
  2. Wait for cleanup between batches
  3. Verify count stays under max_size
• Expected Result: ✅ Memory usage stays bounded

7. Service Health (TestCRITICAL1_ServiceHealthy)

• Purpose: Verify service stays responsive during cleanup
• Configuration: Default retention
• Steps:
  1. Send 20 notifications, check health
  2. Repeat 5 times
  3. Verify health check always succeeds
• Expected Result: ✅ Service remains responsive

Prerequisites

• Docker: For running containerized tests
• Go 1.21+: For building and running tests
• Docker daemon: Must be running

Running Tests

All E2E Tests

go test -v ./tests/e2e -timeout 600s

Single Test

go test -v ./tests/e2e -timeout 120s -run TestCRITICAL1_TTLBasedCleanup

Skip Long-Running Tests

go test -v ./tests/e2e -short

With Debug Output

go test -v ./tests/e2e -timeout 600s -count=1 -race

Test Execution Flow

1. Build Docker Image

   • Uses existing Dockerfile to build notifier:test image
   • Compiles service with all retention features

2. Create Container

   • testcontainers spins up isolated container
   • Exposes port 8080 internally
   • Sets environment variables for retention config

3. Wait for Readiness

   • Polls /health endpoint until ready (max 30s)
   • Waits for service to fully initialize

4. Run Test Scenarios

   • Send notifications via REST client
   • Wait for cleanup to run
   • Verify expected behavior

5. Cleanup

   • Stops and removes container
   • Cleans up Docker resources

Configuration

Each test can customize retention settings via environment variables:

retention := "NOTIFIER_RETENTION_ENABLED=true"
retention += "|NOTIFIER_RETENTION_TTL=2s"
retention += "|NOTIFIER_RETENTION_CHECK_FREQUENCY=500ms"
retention += "|NOTIFIER_RETENTION_MAX_SIZE=50"

suite := SetupSuite(t, retention)

Available Settings

• NOTIFIER_RETENTION_ENABLED: Turn cleanup on/off
• NOTIFIER_RETENTION_TTL: Time-to-live (e.g., "2s", "24h")
• NOTIFIER_RETENTION_CHECK_FREQUENCY: Cleanup interval (e.g., "500ms", "1h")
• NOTIFIER_RETENTION_MAX_SIZE: Max notifications (e.g., 50, 100000)

Client Library

Tests use the REST client library in pkg/client/:

// Create client
cfg := client.ClientConfig{
    BaseURL: "http://localhost:8080",
    Timeout: 30 * time.Second,
}
c := client.NewRESTClient(cfg)

// Send notification
resp, err := c.Send(ctx, client.NotificationRequest{
    Type: "stdout",
    Subject: "Test",
    Body: "Message",
    Recipients: []string{"test@example.com"},
})

// Get stats
stats, err := c.GetStats(ctx)

Debugging

View Container Logs

Tests automatically capture logs on failure. Access via:

logs := suite.GetLogs(context.Background())
fmt.Println(logs)

Inspect Running Container

# Find container ID
docker ps | grep notifier:test

# View logs
docker logs <container-id>

# Connect to container
docker exec -it <container-id> /bin/sh

Common Issues

Issue: Container fails to build

Error: Failed to build docker image

Solution: Ensure Dockerfile exists and Docker daemon is running

Issue: Port already in use

Error: Failed to listen on

Solution: Stop other services on port 8080 or retry

Issue: Timeout waiting for service

Error: Service failed to become ready

Solution: Check Docker logs, may need more CPU/memory

Performance Expectations

Cleanup Performance

• Cleanup 5000 notifications: ~1.4ms
• Per-item overhead: <1μs
• No impact on normal operations

Test Duration

• Single test: 5-10 seconds
• Full suite: 45-60 seconds (running sequentially)
• CI/CD recommendation: Run with -timeout 600s

CI/CD Integration

GitHub Actions Example

- name: Run E2E Tests
  run: |
    go test -v ./tests/e2e -timeout 600s

Docker-in-Docker

services:
  docker:
    image: docker:dind
    options: --privileged

Test Matrix

Scenario	TTL	Frequency	Max Size	Expected
TTL Cleanup	2s	500ms	10k	Old removed
Max Size	24h	500ms	5	Size capped
Disabled	-	-	-	No cleanup
Concurrent	24h	1s	1k	All succeed
Oldest First	24h	500ms	3	Newest kept
Bounded	5s	500ms	50	Size bounded
Health	5s	500ms	1k	Always healthy

Future Enhancements

• gRPC client library + tests
• Load testing scenarios (10k+ notifications)
• Memory profiling validation
• Performance benchmarking
• Stress testing (high throughput)
• Multi-container orchestration tests
• Kubernetes integration tests

References

• testcontainers-go
• Client Library
• CRITICAL-1 Implementation
• Retention Configuration