diff --git a/CLAUDE.md b/CLAUDE.md index e43a487c..dba9f7d9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -293,9 +293,11 @@ UI redesign reverts with the env flag and no web redeploy. Two independent gates route a precomputed client-computed embedding to the biometric-processor's embedding endpoints instead of uploading the raw -image/audio (privacy + GPU-less; the raw media never leaves the device). Both -default OFF (legacy server-side path, byte-identical), flip WITHOUT a redeploy, -and support per-tenant canary. **Both MUST be passed via the compose +image/audio (privacy + GPU-less; the raw media never leaves the device). The CODE +default for both is OFF, BUT production `.env.prod` sets `APP_AUTH_CLIENT_SIDE_EMBEDDING=true` +→ **FACE client-side embedding IS the production default** (enroll + verify, verified +2026-06-14); VOICE stays OFF (legacy server-side path, byte-identical). Both flip WITHOUT a +redeploy, and support per-tenant canary. **Both MUST be passed via the compose `environment:` block** (the service uses an explicit block, NOT `env_file:` — a var only in `.env.prod` is silently dropped; this exact gap once broke the face flag). diff --git a/docs/RUNBOOK_FLYWAY_V29_REPAIR.md b/docs/RUNBOOK_FLYWAY_V29_REPAIR.md deleted file mode 100644 index a644d21d..00000000 --- a/docs/RUNBOOK_FLYWAY_V29_REPAIR.md +++ /dev/null @@ -1,115 +0,0 @@ -# Runbook — Flyway repair after the V29/V40/V41 DR-safety edits - -**Status:** REQUIRED before the next prod boot that ships this change. -**Owner action.** Do NOT skip — `spring.flyway.validate-on-migrate=true` in prod -(`application-prod.yml`, enforced since 2026-05-11) will otherwise crash-loop the -app on boot with a checksum-mismatch on V29, V40, and V41. - -## Why a repair is needed - -P1-5 (disaster-recovery safety) required making the Flyway chain apply cleanly from -a **fresh** database. Three already-applied migrations were edited: - -| Version | File | Edit | Runtime effect on prod | -|---|---|---|---| -| V29 | `V29__add_email_otp_to_default_login_flow.sql` | Resolve the system "Default Login" flow + EMAIL_OTP method by **natural key** instead of two hardcoded prod-only UUIDs; keep the idempotent `WHERE NOT EXISTS (step_order = 2)` guard. | **None.** The step_order=2 row already exists on prod, so 0 rows are written (verified by a ROLLBACK transaction against prod). | -| V40 | `V40__partition_audit_logs.sql` | (a) Rename the legacy PK index `audit_logs_pkey` → `audit_logs_legacy_pkey` (one `ALTER INDEX IF EXISTS`) so the new composite PK can be created on a from-zero run. (b) Fix the trailing `COMMENT ON TABLE … IS 'a' \|\| 'b'` (a SQL syntax error) to a single string literal. | **None.** V40 is already `success=t` and never re-executes; the rename only fires inside V40's own from-scratch run. Prod's `audit_logs_pkey` is owned by the partitioned root, not a legacy table, so the rename would be a no-op there anyway. | -| V41 | `V41__audit_logs_partition_maintenance.sql` | Fix the same invalid `COMMENT … IS 'a' \|\| 'b'` concatenation to a single literal. | **None.** Already `success=t`; never re-executes. | - -These edits change the **checksums** Flyway computes for V29/V40/V41. On a DB that -already applied them, `validate-on-migrate` compares the new file checksum to the -stored one and fails. `flyway repair` re-stamps the stored checksums to match the -edited files **without re-running any migration** (descriptions and versions are -unchanged, so nothing else in the history row moves). - -> Note: V40 and V41 contained a `COMMENT … IS 'a' || 'b'` concatenation that is a -> PostgreSQL **syntax error** — i.e. the committed text never actually executed on -> any database (prod's `audit_logs` comment is still the old V5 text). Prod's V40/V41 -> rows are nonetheless `success=t`, so prod's live schema is correct and untouched; -> the repair only reconciles the stored checksums with the now-valid files. - -## Prod checksums BEFORE the repair (for reference / rollback) - -``` -version | description | checksum (pre-edit, stored in prod) ---------+-------------------------------------+------------------------------------ - 29 | add email otp to default login flow | -1799823743 - 40 | partition audit logs | 1442904668 - 41 | audit logs partition maintenance | -1174003222 -``` - -After the repair these three rows hold the checksums of the **edited** files. - -## Procedure (prod) - -Run this AFTER pulling the new image but BEFORE/INSTEAD of letting the app run -`validate`. There is no Flyway CLI on the VPS — use the one-shot container that -ships with the app image, or run `repair` via the app with validate temporarily -relaxed. Two equivalent options: - -### Option A — one-time relaxed-validate boot (simplest, no extra tooling) - -```bash -cd /opt/projects/fivucsas/identity-core-api -# 1. Temporarily relax validation so the app can boot and run `repair`-equivalent. -# Spring Boot's Flyway auto-config calls migrate(); with validate-on-migrate=false -# it will re-stamp on the next migrate without failing on the checksum diff IS NOT -# automatic — so use the explicit repair in Option B. Prefer Option B. -``` - -> Spring's `migrate()` does **not** auto-repair a checksum mismatch even with -> `validate-on-migrate=false` — it just skips validation. To actually re-stamp the -> stored checksums, run an explicit `flyway repair`. Use Option B. - -### Option B — explicit `flyway repair` (recommended) - -Run the Flyway image against prod's DB on the Docker network (`shared-postgres`), -pointing at the **edited** migration files from the new image/checkout: - -```bash -cd /opt/projects/fivucsas/identity-core-api - -# Sanity: show the mismatched rows first (optional) -docker exec shared-postgres psql -U postgres -d identity_core \ - -c "SELECT version, description, checksum FROM flyway_schema_history WHERE version IN ('29','40','41') ORDER BY version::int;" - -# Repair: re-stamp stored checksums from the migration files in this checkout. -# (Use the same DB creds as .env.prod; identity_core DB on shared-postgres.) -docker run --rm \ - --network \ - -v "$PWD/src/main/resources/db/migration:/flyway/sql:ro" \ - flyway/flyway:10 \ - -url="jdbc:postgresql://shared-postgres:5432/identity_core" \ - -user="$DB_USERNAME" -password="$DB_PASSWORD" \ - -baselineOnMigrate=true -baselineVersion=0 \ - repair - -# Verify the three checksums now match the edited files, then deploy/boot normally. -``` - -> `` = the compose network `shared-postgres` is attached to -> (find via `docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' shared-postgres`). -> `flyway repair` ONLY: (1) re-stamps checksums of applied migrations to match the -> files, and (2) removes failed-migration rows. It runs **no** DDL/DML. - -### After repair — normal deploy - -```bash -docker compose -f docker-compose.prod.yml --env-file .env.prod build --no-cache identity-core-api -docker compose -f docker-compose.prod.yml --env-file .env.prod up -d identity-core-api -# App boots; Flyway validate passes (checksums match); migrate is a no-op (already at head). -curl -s http://127.0.0.1:8080/actuator/health -``` - -## Verification performed (2026-05-30) - -- **BEFORE:** a from-zero psql replay of V0..V71 failed at - `V29` → `auth_flow_steps_auth_flow_id_fkey` on key `e986943a-…` (the exact DR - symptom). Also latent: V40 PK-index collision and the V40/V41 invalid-COMMENT - syntax errors. -- **AFTER:** the full V0..V71 chain applied on a throwaway `flyway_dr_test` DB - (`CREATE EXTENSION vector`) with **zero errors**; the system "Default Login" flow - ends with PASSWORD(1)+EMAIL_OTP(2, required, 300s, 3 attempts) — byte-identical to - prod's row — and `audit_logs` is range-partitioned. Throwaway DB dropped after. -- **Prod no-op:** the rewritten V29 body run inside a `BEGIN … ROLLBACK` against the - live `identity_core` DB inserted **0 rows**. diff --git a/docs/migrations/MIGRATION_V7_V8_V9_SUMMARY.md b/docs/migrations/MIGRATION_V7_V8_V9_SUMMARY.md deleted file mode 100644 index b3e86429..00000000 --- a/docs/migrations/MIGRATION_V7_V8_V9_SUMMARY.md +++ /dev/null @@ -1,463 +0,0 @@ -# Database Migrations V7, V8, V9 - Implementation Summary - -## Overview - -Three new Flyway migrations have been added to the identity-core-api project to enhance performance, audit capabilities, and rate limiting functionality. - -## Migration Files - -### V7: Performance Indexes (`V7__add_performance_indexes.sql`) - -**Purpose**: Optimize database query performance for frequently accessed data patterns. - -**Key Indexes Added**: - -1. **Users Table** - - `idx_users_email_unique`: Unique index for email lookups (login, registration) - - `idx_users_tenant_status`: Composite index for tenant-scoped user status queries - -2. **Biometric Data Table** - - `idx_biometric_user_tenant`: Composite index for user-tenant biometric lookups - - `idx_biometric_primary_lookup`: Partial index for primary biometric selection - -3. **Audit Logs Table** - - `idx_audit_tenant_created`: Time-based tenant audit queries - - `idx_audit_user_action`: User activity tracking - - `idx_audit_failed_operations`: Security monitoring for failed operations - -4. **Active Sessions Table** - - `idx_sessions_user_expires`: Session validation and cleanup - - `idx_sessions_expired`: Expired session cleanup - -5. **Refresh Tokens Table** - - `idx_refresh_tokens_hash_lookup`: Token validation during refresh - - `idx_refresh_tokens_user_expires`: User token cleanup - - `idx_refresh_tokens_expired`: Expired token cleanup - -6. **Security Events Table** - - `idx_security_events_tenant_severity`: Real-time security monitoring - - `idx_security_events_critical`: Critical unresolved events - -7. **Additional Indexes** - - Liveness attempts: User history tracking - - Biometric verification logs: Audit trail - - Password history: Prevent password reuse - -**Performance Impact**: -- Login queries: ~50-80% faster -- Audit log queries: ~70-90% faster -- Session validation: ~60-80% faster -- Token refresh: ~40-60% faster - ---- - -### V8: Audit Log Enhancements (`V8__add_audit_log_enhancements.sql`) - -**Purpose**: Extend audit logging with distributed tracing, performance monitoring, and enhanced analytics. - -**New Columns**: -- `user_agent_v2` (TEXT): Enhanced user agent tracking -- `request_id` (UUID): Distributed request tracing -- `duration_ms` (INTEGER): Operation duration for performance analysis -- `enhanced_metadata` (JSONB): Flexible additional data storage - -**New Indexes**: -- `idx_audit_request_id`: Distributed tracing lookup -- `idx_audit_duration_slow`: Slow operation identification (>1 second) -- `idx_audit_request_timing`: Request timeline analysis -- `idx_audit_enhanced_metadata_gin`: JSON field queries - -**Database Functions**: - -1. `populate_audit_request_id()`: Automatically extracts request_id and duration from metadata -2. `apply_audit_retention_policy()`: Tiered data retention - - Detailed logs: 90 days - - Summary logs: 1 year -3. `refresh_audit_statistics()`: Refreshes materialized view for analytics - -**Views**: - -1. `v_recent_audit_logs`: Fast access to last 30 days of logs -2. `v_slow_operations`: Operations exceeding 1 second (last 7 days) -3. `mv_audit_statistics`: Pre-aggregated daily statistics (materialized) - -**Features**: -- Distributed tracing across microservices -- Performance monitoring and optimization -- Automated data retention policy -- Pre-aggregated analytics for dashboards - ---- - -### V9: Rate Limiting Table (`V9__add_rate_limiting_table.sql`) - -**Purpose**: Persistent rate limiting using token bucket algorithm for API throttling. - -**Table**: `rate_limit_buckets` - -**Columns**: -- `limit_key` (VARCHAR): Unique identifier (IP, user, endpoint, tenant) -- `limit_type` (VARCHAR): Category (IP, USER, ENDPOINT, TENANT, GLOBAL) -- `bucket_tokens` (DECIMAL): Current available tokens -- `max_tokens` (DECIMAL): Maximum capacity (burst limit) -- `refill_rate` (DECIMAL): Tokens per second (sustained rate) -- `last_refill_at` (TIMESTAMP): Last refill time -- `expires_at` (TIMESTAMP): Optional expiration -- `request_count` (BIGINT): Total requests processed -- `blocked_count` (BIGINT): Total requests blocked -- `metadata` (JSONB): Additional context - -**Indexes**: -- `idx_rate_limit_key`: Fast bucket lookup -- `idx_rate_limit_type`: Category queries -- `idx_rate_limit_expired`: Cleanup operations -- `idx_rate_limit_refill`: Refill operations -- `idx_rate_limit_hot_buckets`: Active buckets -- `idx_rate_limit_metadata_gin`: JSON queries - -**Database Functions**: - -1. `consume_rate_limit_tokens()`: Atomically consume tokens from bucket - - Returns: allowed (boolean), remaining_tokens, retry_after_seconds - - Implements token bucket algorithm - - Thread-safe and distributed-system compatible - -2. `reset_rate_limit_bucket()`: Reset bucket to full capacity - -3. `cleanup_rate_limit_buckets()`: Remove expired and stale buckets - - Deletes explicitly expired buckets - - Deletes stale buckets (not accessed in 24 hours) - -4. `get_rate_limit_status()`: Get current status without consuming tokens - -**View**: -- `v_rate_limit_monitoring`: Real-time monitoring with status indicators - -**Default Rate Limits**: -- Global API: 10,000 burst, 100/sec sustained -- Login endpoint: 100 burst, 5/sec sustained -- Registration endpoint: 50 burst, 2/sec sustained - -**Token Bucket Algorithm**: -``` -1. Each request checks available tokens -2. If tokens available: consume and allow request -3. If no tokens: block request, return retry-after -4. Tokens refill continuously at refill_rate -5. Maximum capacity is max_tokens (burst limit) -``` - ---- - -## Java Entity Classes - -### `AuditLog.java` - -**Location**: `src/main/java/com/fivucsas/identity/entity/AuditLog.java` - -**Key Features**: -- Immutable audit logs (no updates) -- Complete request context capture -- JSONB support for flexible metadata -- Helper methods for security event detection - -**Methods**: -- `isSecurityEvent()`: Identifies security-sensitive actions -- `isFailed()`: Checks for failed operations -- `isSlowOperation()`: Detects operations > 1 second -- `getEffectiveUserAgent()`: Prioritizes V8 field -- `getEffectiveDuration()`: Unified duration access - ---- - -### `RateLimitBucket.java` - -**Location**: `src/main/java/com/fivucsas/identity/entity/RateLimitBucket.java` - -**Key Features**: -- Token bucket algorithm implementation -- Support for multiple rate limit types -- Rich metadata and tracking metrics - -**Methods**: -- `isExpired()`: Check expiration -- `isEmpty()`: No tokens available -- `isFull()`: At maximum capacity -- `getCapacityPercent()`: Usage percentage -- `getBlockRatePercent()`: Blocking statistics -- `isHot()`: Frequently accessed (< 1 hour) -- `isStale()`: Not accessed (> 24 hours) -- `reset()`: Reset to full capacity - ---- - -## Java Repository Classes - -### `AuditLogRepository.java` - -**Location**: `src/main/java/com/fivucsas/identity/repository/AuditLogRepository.java` - -**Key Methods**: -- `findByTenantIdOrderByCreatedAtDesc()`: Tenant audit logs -- `findByRequestIdOrderByCreatedAtAsc()`: Distributed tracing -- `findSlowOperations()`: Performance monitoring -- `countFailedLoginsByUserAndTimeRange()`: Security monitoring -- `countFailedOperationsByIpAndTimeRange()`: Brute force detection -- `deleteOldAuditLogs()`: Data retention cleanup -- `archiveDetailedAuditData()`: Tiered retention - ---- - -### `RateLimitBucketRepository.java` - -**Location**: `src/main/java/com/fivucsas/identity/repository/RateLimitBucketRepository.java` - -**Key Methods**: -- `findByLimitKey()`: Get bucket by key -- `findExpiredBuckets()`: Cleanup operations -- `findStaleBuckets()`: Identify inactive buckets -- `findHotBuckets()`: Active rate limits -- `findHighBlockRateBuckets()`: Security monitoring -- `resetBucket()`: Reset specific bucket -- `getTotalRequestCount()`: System-wide statistics -- `getStatisticsByType()`: Type-specific metrics - ---- - -## Testing Recommendations - -### 1. Migration Testing - -This project uses **Maven**, not Gradle, and there is no `flyway-maven-plugin` -configured in `pom.xml`. Flyway runs automatically on application startup via -Spring Boot auto-configuration. To apply migrations, start the application: - -```bash -# Start the application — Flyway runs automatically on boot -./mvnw spring-boot:run -Dspring-boot.run.profiles=dev - -# To check migration history after startup, query the DB directly: -# SELECT * FROM flyway_schema_history ORDER BY installed_rank DESC; -``` - -### 2. Performance Testing - -Test index effectiveness: - -```sql --- Before: Full table scan -EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com'; - --- After V7: Index scan -EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com'; -``` - -### 3. Audit Log Testing - -```sql --- Test distributed tracing -SELECT * FROM audit_logs WHERE request_id = 'some-uuid' ORDER BY created_at; - --- Test slow operation detection -SELECT * FROM v_slow_operations LIMIT 10; - --- Test audit statistics -SELECT * FROM mv_audit_statistics WHERE audit_date = CURRENT_DATE; -``` - -### 4. Rate Limiting Testing - -```sql --- Test token consumption -SELECT * FROM consume_rate_limit_tokens('ip:192.168.1.1', 1.0); - --- Test rate limit status -SELECT * FROM get_rate_limit_status('ip:192.168.1.1'); - --- Test monitoring -SELECT * FROM v_rate_limit_monitoring; -``` - -### 5. Integration Testing - -Create integration tests for: -- AuditLog entity CRUD operations -- RateLimitBucket entity operations -- Repository query performance -- Database function behavior - ---- - -## Maintenance Tasks - -### Scheduled Jobs - -Recommended scheduled tasks: - -1. **Daily**: Refresh audit statistics - ```sql - SELECT refresh_audit_statistics(); - ``` - -2. **Daily**: Cleanup rate limit buckets - ```sql - SELECT * FROM cleanup_rate_limit_buckets(); - ``` - -3. **Weekly**: Apply audit retention policy - ```sql - SELECT apply_audit_retention_policy(); - ``` - -4. **Monthly**: Cleanup expired sessions and tokens - ```sql - SELECT cleanup_expired_sessions(); - ``` - ---- - -## Monitoring Queries - -### Performance Monitoring - -```sql --- Top slow endpoints -SELECT endpoint, AVG(duration_ms) as avg_duration, COUNT(*) as count -FROM audit_logs -WHERE duration_ms > 1000 AND created_at >= NOW() - INTERVAL '7 days' -GROUP BY endpoint -ORDER BY avg_duration DESC -LIMIT 10; -``` - -### Security Monitoring - -```sql --- Failed login attempts by IP -SELECT ip_address, COUNT(*) as failed_attempts -FROM audit_logs -WHERE action LIKE '%LOGIN%' AND success = false - AND created_at >= NOW() - INTERVAL '1 hour' -GROUP BY ip_address -HAVING COUNT(*) > 5 -ORDER BY failed_attempts DESC; -``` - -### Rate Limit Monitoring - -```sql --- High block rate buckets -SELECT limit_key, limit_type, - request_count, blocked_count, - ROUND((blocked_count::DECIMAL / request_count) * 100, 2) as block_rate_percent -FROM rate_limit_buckets -WHERE request_count > 0 -ORDER BY block_rate_percent DESC -LIMIT 10; -``` - ---- - -## Rollback Instructions - -If needed, rollback migrations: - -### Rollback V9 -```sql -DROP VIEW IF EXISTS v_rate_limit_monitoring; -DROP FUNCTION IF EXISTS get_rate_limit_status(VARCHAR); -DROP FUNCTION IF EXISTS cleanup_rate_limit_buckets(); -DROP FUNCTION IF EXISTS reset_rate_limit_bucket(VARCHAR); -DROP FUNCTION IF EXISTS consume_rate_limit_tokens(VARCHAR, DECIMAL, VARCHAR, DECIMAL, DECIMAL, INTEGER); -DROP TABLE IF EXISTS rate_limit_buckets CASCADE; -``` - -### Rollback V8 -```sql -DROP FUNCTION IF EXISTS refresh_audit_statistics(); -DROP MATERIALIZED VIEW IF EXISTS mv_audit_statistics; -DROP VIEW IF EXISTS v_slow_operations; -DROP VIEW IF EXISTS v_recent_audit_logs; -DROP FUNCTION IF EXISTS apply_audit_retention_policy(); -DROP TRIGGER IF EXISTS trg_populate_audit_request_id ON audit_logs; -DROP FUNCTION IF EXISTS populate_audit_request_id(); -DROP INDEX IF EXISTS idx_audit_enhanced_metadata_gin; -DROP INDEX IF EXISTS idx_audit_request_timing; -DROP INDEX IF EXISTS idx_audit_duration_slow; -DROP INDEX IF EXISTS idx_audit_request_id; -ALTER TABLE audit_logs DROP COLUMN IF EXISTS enhanced_metadata; -ALTER TABLE audit_logs DROP COLUMN IF EXISTS duration_ms; -ALTER TABLE audit_logs DROP COLUMN IF EXISTS request_id; -ALTER TABLE audit_logs DROP COLUMN IF EXISTS user_agent_v2; -``` - -### Rollback V7 -```sql --- Drop all performance indexes created in V7 --- (See individual DROP INDEX commands in V7 comments) -``` - ---- - -## Architecture Compliance - -These migrations follow the project's architectural principles: - -### Hexagonal Architecture -- **Domain Layer**: Entity models (AuditLog, RateLimitBucket) -- **Port Layer**: Repository interfaces -- **Adapter Layer**: JPA repository implementations -- **Infrastructure**: Database migrations - -### SOLID Principles -- **Single Responsibility**: Each entity has one clear purpose -- **Open/Closed**: Extensible via metadata fields -- **Liskov Substitution**: Repositories implement JpaRepository -- **Interface Segregation**: Focused repository methods -- **Dependency Inversion**: Domain defines contracts - -### Design Patterns -- **Repository Pattern**: Data access abstraction -- **Strategy Pattern**: Token bucket algorithm -- **Observer Pattern**: Audit logging trigger -- **Factory Pattern**: Entity builders - ---- - -## Performance Benchmarks - -Expected performance improvements: - -| Operation | Before | After | Improvement | -|-----------|--------|-------|-------------| -| Email lookup | 150ms | 30ms | 80% faster | -| Tenant audit query | 500ms | 100ms | 80% faster | -| Session validation | 100ms | 25ms | 75% faster | -| Token refresh | 80ms | 30ms | 62% faster | -| Rate limit check | N/A | 5ms | New feature | - ---- - -## Security Considerations - -1. **Audit Immutability**: Audit logs cannot be modified, only created -2. **Data Retention**: Automatic cleanup prevents unbounded growth -3. **Rate Limiting**: Protects against brute force and DDoS attacks -4. **Distributed Tracing**: Enables incident investigation -5. **Performance Monitoring**: Detects anomalies and attacks - ---- - -## References - -- Flyway Documentation: https://flywaydb.org/documentation/ -- PostgreSQL Indexing: https://www.postgresql.org/docs/current/indexes.html -- Token Bucket Algorithm: https://en.wikipedia.org/wiki/Token_bucket -- Audit Logging Best Practices: https://www.owasp.org/index.php/Logging_Cheat_Sheet - ---- - -**Created**: 2026-03-15 (retroactively documented; V7–V9 shipped early 2026) -**Author**: Claude Code (AI Assistant) -**Project**: FIVUCSAS Identity Core API -**Version**: 1.0.0 diff --git a/docs/migrations/TESTING_CHECKLIST.md b/docs/migrations/TESTING_CHECKLIST.md deleted file mode 100644 index d1a12fb1..00000000 --- a/docs/migrations/TESTING_CHECKLIST.md +++ /dev/null @@ -1,484 +0,0 @@ -# Migration V7, V8, V9 - Testing Checklist - -## Pre-Migration Verification - -- [ ] Backup production database -- [ ] Review migration files for syntax errors -- [ ] Test migrations on development database -- [ ] Verify application.properties/application.yml Flyway configuration - -## Migration Execution - -### Step 1: Run Migrations - -This project uses **Maven** (not Gradle). There is no `flyway-maven-plugin` -in `pom.xml` — Flyway runs automatically on application startup via Spring Boot -auto-configuration. - -```bash -# Start the application — Flyway applies pending migrations automatically -./mvnw spring-boot:run -Dspring-boot.run.profiles=dev -``` - -### Step 2: Verify Migration Status - -```sql --- Check Flyway schema history -SELECT * FROM flyway_schema_history ORDER BY installed_rank DESC; - --- Verify V7 indexes were created -SELECT indexname, tablename FROM pg_indexes -WHERE indexname LIKE 'idx_%' - AND schemaname = 'public' -ORDER BY tablename, indexname; - --- Verify V8 columns were added -SELECT column_name, data_type -FROM information_schema.columns -WHERE table_name = 'audit_logs' - AND column_name IN ('request_id', 'duration_ms', 'user_agent_v2', 'enhanced_metadata'); - --- Verify V9 table was created -SELECT table_name FROM information_schema.tables -WHERE table_name = 'rate_limit_buckets'; -``` - -## Functional Testing - -### V7: Performance Indexes - -#### Test 1: Email Lookup Performance -```sql --- Explain plan should show Index Scan on idx_users_email_unique -EXPLAIN ANALYZE -SELECT * FROM users WHERE email = 'admin@fivucsas.local' AND deleted_at IS NULL; -``` - -- [ ] Query uses index scan (not sequential scan) -- [ ] Query time < 10ms - -#### Test 2: Tenant User Status Query -```sql --- Explain plan should show Index Scan on idx_users_tenant_status -EXPLAIN ANALYZE -SELECT * FROM users -WHERE tenant_id = '00000000-0000-0000-0000-000000000000' - AND is_active = true - AND deleted_at IS NULL; -``` - -- [ ] Query uses composite index -- [ ] Query time < 20ms - -#### Test 3: Audit Log Tenant Query -```sql --- Explain plan should show Index Scan on idx_audit_tenant_created -EXPLAIN ANALYZE -SELECT * FROM audit_logs -WHERE tenant_id = '00000000-0000-0000-0000-000000000000' -ORDER BY created_at DESC -LIMIT 100; -``` - -- [ ] Query uses composite index -- [ ] Query time < 50ms - -### V8: Audit Log Enhancements - -#### Test 1: Insert Audit Log with New Fields -```sql -INSERT INTO audit_logs ( - tenant_id, user_id, action, resource_type, success, - request_id, duration_ms, user_agent_v2, enhanced_metadata -) VALUES ( - '00000000-0000-0000-0000-000000000000', - (SELECT id FROM users WHERE email = 'admin@fivucsas.local'), - 'TEST_ACTION', - 'TEST_RESOURCE', - true, - uuid_generate_v4(), - 150, - 'Mozilla/5.0 (Test Browser)', - '{"test": "metadata", "environment": "testing"}'::jsonb -); -``` - -- [ ] Insert succeeds -- [ ] All new fields are populated -- [ ] Trigger `trg_populate_audit_request_id` executes - -#### Test 2: Distributed Tracing Query -```sql --- Generate test request ID -DO $$ -DECLARE - test_request_id UUID := uuid_generate_v4(); -BEGIN - -- Insert multiple audit logs with same request_id - INSERT INTO audit_logs (tenant_id, action, resource_type, success, request_id, duration_ms) - VALUES - ('00000000-0000-0000-0000-000000000000', 'API_CALL_1', 'TEST', true, test_request_id, 50), - ('00000000-0000-0000-0000-000000000000', 'API_CALL_2', 'TEST', true, test_request_id, 100), - ('00000000-0000-0000-0000-000000000000', 'API_CALL_3', 'TEST', true, test_request_id, 75); - - -- Query by request_id - RAISE NOTICE 'Test request ID: %', test_request_id; -END $$; - --- Verify distributed tracing -SELECT action, resource_type, duration_ms, created_at -FROM audit_logs -WHERE request_id IN ( - SELECT DISTINCT request_id FROM audit_logs WHERE action LIKE 'API_CALL_%' -) -ORDER BY created_at; -``` - -- [ ] All related audit logs are retrieved -- [ ] Index `idx_audit_request_id` is used -- [ ] Query time < 10ms - -#### Test 3: Slow Operations Detection -```sql --- Insert slow operation -INSERT INTO audit_logs ( - tenant_id, action, resource_type, endpoint, success, duration_ms -) VALUES ( - '00000000-0000-0000-0000-000000000000', - 'SLOW_OPERATION', - 'TEST', - '/api/slow/endpoint', - true, - 5000 -); - --- Query slow operations -SELECT * FROM v_slow_operations LIMIT 10; -``` - -- [ ] View returns slow operations -- [ ] Operations with duration > 1000ms are included -- [ ] Query time < 20ms - -#### Test 4: Audit Statistics Materialized View -```sql --- Refresh materialized view -SELECT refresh_audit_statistics(); - --- Query statistics -SELECT * FROM mv_audit_statistics -WHERE audit_date = CURRENT_DATE -ORDER BY total_operations DESC -LIMIT 10; -``` - -- [ ] Materialized view refreshes successfully -- [ ] Statistics are aggregated correctly -- [ ] Query time < 5ms - -#### Test 5: Retention Policy -```sql --- Test retention policy (dry run) -SELECT apply_audit_retention_policy(); - --- Check archived logs -SELECT COUNT(*) FROM audit_logs -WHERE created_at < CURRENT_TIMESTAMP - INTERVAL '90 days' - AND (old_values IS NULL AND new_values IS NULL); -``` - -- [ ] Function executes without errors -- [ ] Old logs are archived (metadata removed) -- [ ] Logs older than 1 year are deleted - -### V9: Rate Limiting - -#### Test 1: Token Consumption -```sql --- Create test bucket -INSERT INTO rate_limit_buckets ( - limit_key, limit_type, bucket_tokens, max_tokens, refill_rate -) VALUES ( - 'test:bucket:1', 'TEST', 10, 10, 1 -); - --- Consume tokens (should succeed) -SELECT * FROM consume_rate_limit_tokens('test:bucket:1', 5); - --- Verify result --- Expected: allowed=true, remaining_tokens=5, retry_after_seconds=0 - --- Consume more tokens (should succeed) -SELECT * FROM consume_rate_limit_tokens('test:bucket:1', 5); - --- Try to consume when empty (should fail) -SELECT * FROM consume_rate_limit_tokens('test:bucket:1', 1); - --- Expected: allowed=false, remaining_tokens=0, retry_after_seconds>0 -``` - -- [ ] First consumption succeeds (5 tokens) -- [ ] Second consumption succeeds (5 tokens) -- [ ] Third consumption fails (no tokens) -- [ ] Retry-after is calculated correctly - -#### Test 2: Token Refill -```sql --- Wait for refill (1 token per second) -SELECT pg_sleep(3); - --- Try consuming again (should have ~3 tokens) -SELECT * FROM consume_rate_limit_tokens('test:bucket:1', 2); - --- Expected: allowed=true, remaining_tokens~=1 -``` - -- [ ] Tokens are refilled over time -- [ ] Consumption succeeds after refill - -#### Test 3: Rate Limit Status -```sql --- Check status without consuming -SELECT * FROM get_rate_limit_status('test:bucket:1'); -``` - -- [ ] Status shows current available tokens -- [ ] Status shows request and blocked counts -- [ ] No tokens are consumed - -#### Test 4: Bucket Cleanup -```sql --- Create expired bucket -INSERT INTO rate_limit_buckets ( - limit_key, limit_type, bucket_tokens, max_tokens, refill_rate, expires_at -) VALUES ( - 'test:expired:1', 'TEST', 10, 10, 1, CURRENT_TIMESTAMP - INTERVAL '1 hour' -); - --- Run cleanup -SELECT * FROM cleanup_rate_limit_buckets(); - --- Verify expired bucket is deleted -SELECT COUNT(*) FROM rate_limit_buckets WHERE limit_key = 'test:expired:1'; - --- Expected: count=0 -``` - -- [ ] Cleanup function executes -- [ ] Expired buckets are deleted -- [ ] Stale buckets are deleted - -#### Test 5: Rate Limit Monitoring -```sql --- View monitoring dashboard -SELECT * FROM v_rate_limit_monitoring ORDER BY last_seen_at DESC LIMIT 10; -``` - -- [ ] View shows all active buckets -- [ ] Capacity percentage is calculated -- [ ] Block rate percentage is calculated -- [ ] Status indicator is correct (ACTIVE/IDLE/STALE) - -## Java Entity Testing - -### Test 1: AuditLog Entity -```java -@Test -void testAuditLogEntity() { - AuditLog auditLog = AuditLog.builder() - .tenantId(UUID.randomUUID()) - .userId(UUID.randomUUID()) - .action("LOGIN") - .resourceType("USER") - .success(true) - .requestId(UUID.randomUUID()) - .durationMs(150) - .build(); - - // Test security event detection - assertTrue(auditLog.isSecurityEvent()); - assertFalse(auditLog.isFailed()); - assertFalse(auditLog.isSlowOperation()); - - // Test with slow operation - auditLog.setDurationMs(1500); - assertTrue(auditLog.isSlowOperation()); -} -``` - -- [ ] Entity builds correctly -- [ ] Helper methods work as expected -- [ ] JSONB fields are mapped correctly - -### Test 2: RateLimitBucket Entity -```java -@Test -void testRateLimitBucketEntity() { - RateLimitBucket bucket = RateLimitBucket.builder() - .limitKey("test:key:1") - .limitType("TEST") - .bucketTokens(new BigDecimal("50")) - .maxTokens(new BigDecimal("100")) - .refillRate(new BigDecimal("10")) - .requestCount(100L) - .blockedCount(10L) - .build(); - - // Test capacity calculation - assertEquals(50.0, bucket.getCapacityPercent()); - - // Test block rate calculation - assertEquals(10.0, bucket.getBlockRatePercent()); - - // Test status checks - assertFalse(bucket.isEmpty()); - assertFalse(bucket.isFull()); -} -``` - -- [ ] Entity builds correctly -- [ ] Calculations are accurate -- [ ] Status checks work as expected - -## Repository Testing - -### Test 1: AuditLogRepository -```java -@Test -void testAuditLogRepository() { - UUID tenantId = UUID.randomUUID(); - UUID requestId = UUID.randomUUID(); - - // Create test audit logs - AuditLog log1 = auditLogRepository.save(createTestLog(tenantId, requestId, 100)); - AuditLog log2 = auditLogRepository.save(createTestLog(tenantId, requestId, 200)); - - // Test distributed tracing query - List logs = auditLogRepository.findByRequestIdOrderByCreatedAtAsc(requestId); - assertEquals(2, logs.size()); - - // Test slow operations query - Page slowOps = auditLogRepository.findSlowOperations(1000, PageRequest.of(0, 10)); - // Verify results -} -``` - -- [ ] Basic CRUD operations work -- [ ] Custom queries return correct results -- [ ] Pagination works correctly - -### Test 2: RateLimitBucketRepository -```java -@Test -void testRateLimitBucketRepository() { - String limitKey = "test:key:" + UUID.randomUUID(); - - // Create test bucket - RateLimitBucket bucket = rateLimitBucketRepository.save( - RateLimitBucket.builder() - .limitKey(limitKey) - .limitType("TEST") - .bucketTokens(new BigDecimal("100")) - .maxTokens(new BigDecimal("100")) - .refillRate(new BigDecimal("10")) - .build() - ); - - // Test find by key - Optional found = rateLimitBucketRepository.findByLimitKey(limitKey); - assertTrue(found.isPresent()); - - // Test reset - rateLimitBucketRepository.resetBucket(limitKey); - // Verify bucket is reset -} -``` - -- [ ] Basic CRUD operations work -- [ ] Custom queries return correct results -- [ ] Update operations work correctly - -## Performance Benchmarks - -### Baseline Measurements - -Record baseline performance before migration: - -```sql --- Email lookup -\timing on -SELECT * FROM users WHERE email = 'admin@fivucsas.local'; -\timing off - --- Tenant audit query -\timing on -SELECT * FROM audit_logs WHERE tenant_id = '...' ORDER BY created_at DESC LIMIT 100; -\timing off - --- Session validation -\timing on -SELECT * FROM active_sessions WHERE user_id = '...' AND is_active = true; -\timing off -``` - -### After Migration Measurements - -Re-run the same queries and compare: - -- [ ] Email lookup improved by >50% -- [ ] Audit query improved by >70% -- [ ] Session validation improved by >60% - -## Rollback Testing - -### Test Rollback Procedures - -```sql --- Test V9 rollback -BEGIN; --- Execute rollback commands from MIGRATION_V7_V8_V9_SUMMARY.md --- Verify table is dropped -SELECT COUNT(*) FROM information_schema.tables WHERE table_name = 'rate_limit_buckets'; -ROLLBACK; - --- Repeat for V8 and V7 if needed -``` - -- [ ] Rollback commands are documented -- [ ] Rollback succeeds without errors -- [ ] Database returns to previous state - -## Production Deployment Checklist - -- [ ] All tests passed in development environment -- [ ] All tests passed in staging environment -- [ ] Performance benchmarks meet expectations -- [ ] Database backup completed -- [ ] Rollback procedure tested and documented -- [ ] Monitoring alerts configured -- [ ] Team notified of deployment window -- [ ] Post-deployment verification plan ready - -## Post-Deployment Verification - -- [ ] Check Flyway schema history -- [ ] Verify all indexes are created -- [ ] Run sample queries to test performance -- [ ] Monitor application logs for errors -- [ ] Check database CPU and memory usage -- [ ] Verify audit logs are being created correctly -- [ ] Test rate limiting functionality -- [ ] Monitor for any performance degradation - -## Known Issues / Notes - -- Record any issues encountered during testing -- Document any configuration changes needed -- Note any compatibility concerns - ---- - -**Last Updated**: 2026-05-28 -**Tested By**: _____________ -**Environment**: _____________ -**Status**: [ ] PASS / [ ] FAIL / [ ] PARTIAL diff --git a/docs/research/server-providers-comparison.md b/docs/research/server-providers-comparison.md deleted file mode 100644 index a5490f76..00000000 --- a/docs/research/server-providers-comparison.md +++ /dev/null @@ -1,587 +0,0 @@ -# Server Providers Research - Identity Core API - -## Architecture Overview - -The Identity Core API has two distinct compute workloads: - -1. **Identity Core API (Java/Spring Boot)** - Lightweight REST API, JWT auth, multi-tenancy, audit logging - - Memory: 256MB-512MB JVM heap - - No GPU required - - Suitable for standard VPS - -2. **Biometric Processor (FastAPI)** - Face detection, embedding extraction (VGG-Face/FaceNet512), liveness detection, quality assessment - - Requires GPU for ML inference - - Models: VGG-Face (2622-dim) or FaceNet512 (512-dim) - - Communicates via HTTP on port 8001 - -``` -┌────────────────────────────┐ ┌──────────────────────────────┐ -│ Identity Core API (Java) │──HTTP──│ Biometric Processor (FastAPI)│ -│ VPS │ :8001 │ GPU Instance / Serverless │ -└────────────────────────────┘ └──────────────────────────────┘ - │ │ - ▼ │ - PostgreSQL + pgvector ◄───────────────────────┘ - (Vector embeddings) -``` - ---- - -## Part 1: VPS Providers Comparison (EU and Turkey Focus) - -### 1.1 Hostinger VPS (KVM-based, All Tiers) - -All plans use AMD EPYC processors, NVMe SSD storage, and KVM virtualization with full root access. -Docker is fully supported (KVM-based). Free weekly backups, 1 Gbps network, snapshots included. - -| Plan | vCPU | RAM | NVMe Storage | Bandwidth | Promo Price/mo | Renewal Price/mo | -|------|------|-----|-------------|-----------|----------------|------------------| -| KVM 1 | 1 | 4 GB | 50 GB | 4 TB | $4.99 | ~$7.99 | -| KVM 2 | 2 | 8 GB | 100 GB | 8 TB | $6.99 | ~$10.99 | -| KVM 4 | 4 | 16 GB | 200 GB | 16 TB | $9.99 | ~$15.99 | -| KVM 8 | 8 | 32 GB | 400 GB | 32 TB | $19.99 | ~$29.99 | - -**Promo prices require 24-month prepayment.** Renewal prices are significantly higher. - -**EU Data Center Locations (VPS):** -- Lithuania (Vilnius) -- Netherlands (Amsterdam) -- United Kingdom (London) -- Germany (Frankfurt) -- newer addition -- France -- available for cloud/shared hosting only, NOT VPS yet - -**Other Locations:** USA (multiple), Brazil, India, Singapore, Indonesia - -**Docker Support:** YES -- full KVM virtualization, install Docker freely. -**Managed K8s:** No. Self-managed only. - -Sources: [Hostinger VPS Pricing](https://www.hostinger.com/pricing/vps-hosting) | [Hostinger VPS Locations](https://hostingadvices.co.uk/hostinger-vps-locations/) | [VPSBenchmarks](https://www.vpsbenchmarks.com/compare/hostinger) - ---- - -### 1.2 Hetzner Cloud (Germany/Finland) - -Best price-performance in Europe. All-inclusive pricing (traffic, IPv4/IPv6, DDoS, firewall included). -Docker fully supported. Hourly billing with monthly caps. - -#### CX Series (Cost-Optimized, Shared vCPU) -- BEST VALUE, Germany/Finland only - -| Plan | vCPU | RAM | NVMe | Traffic | Price/mo | -|------|------|-----|------|---------|----------| -| CX23 | 2 | 4 GB | 40 GB | 20 TB | **EUR 3.49** | -| CX33 | 4 | 8 GB | 80 GB | 20 TB | **EUR 5.49** | -| CX43 | 8 | 16 GB | 160 GB | 20 TB | **EUR 9.49** | -| CX53 | 16 | 32 GB | 320 GB | 20 TB | **EUR 17.49** | - -#### CAX Series (ARM Ampere, Shared) -- Germany/Finland only - -| Plan | vCPU | RAM | NVMe | Traffic | Price/mo | -|------|------|-----|------|---------|----------| -| CAX11 | 2 | 4 GB | 40 GB | 20 TB | EUR 3.79 | -| CAX21 | 4 | 8 GB | 80 GB | 20 TB | EUR 6.49 | -| CAX31 | 8 | 16 GB | 160 GB | 20 TB | EUR 12.49 | - -#### CPX Series (AMD EPYC, Shared) -- All regions (incl. USA, Singapore) - -| Plan | vCPU | RAM | NVMe | Traffic (EU) | Price/mo | -|------|------|-----|------|-------------|----------| -| CPX11 | 2 | 2 GB | 40 GB | 1 TB | EUR 4.99 | -| CPX21 | 3 | 4 GB | 80 GB | 2 TB | EUR 9.49 | -| CPX31 | 4 | 8 GB | 160 GB | 3 TB | EUR 16.49 | -| CPX41 | 8 | 16 GB | 240 GB | 4 TB | EUR 30.49 | - -#### CCX Series (Dedicated vCPU) -- All regions - -| Plan | vCPU | RAM | NVMe | Traffic (EU) | Price/mo | -|------|------|-----|------|-------------|----------| -| CCX13 | 2 | 8 GB | 80 GB | 1 TB | EUR 12.49 | -| CCX23 | 4 | 16 GB | 160 GB | 2 TB | EUR 24.49 | -| CCX33 | 8 | 32 GB | 240 GB | 3 TB | EUR 48.49 | - -**Data Center Locations:** -- Germany: Falkenstein (FSN1), Nuremberg (NBG1) -- Finland: Helsinki (HEL1) -- USA: Ashburn (VA), Hillsboro (OR) -- Singapore - -**DOES NOT have a Turkey data center.** Closest to Turkey: Helsinki (~2200km) or Nuremberg (~1800km). - -**Docker Support:** YES -- full KVM, Docker/Podman/K8s all work. -**Managed K8s:** YES -- Hetzner offers managed Kubernetes clusters. - -Sources: [Hetzner Cloud](https://www.hetzner.com/cloud) | [Hetzner Locations](https://docs.hetzner.com/cloud/general/locations/) - ---- - -### 1.3 Contabo (Germany-based, Budget King) - -Extremely generous RAM/storage for the price. VPS plans start at 8 GB RAM. -EU location assigned automatically (typically Germany or France). - -| Plan | vCPU | RAM | NVMe / SSD | Port Speed | Traffic | Price/mo | -|------|------|-----|-----------|------------|---------|----------| -| Cloud VPS 10 | 4 | 8 GB | 75 GB / 150 GB | 200 Mbit/s | Unlimited | **EUR 4.50** (~$4.95) | -| Cloud VPS 20 | 6 | 12 GB | 100 GB / 200 GB | 300 Mbit/s | Unlimited | **EUR 7.00** (~$7.70) | -| Cloud VPS 30 | 8 | 24 GB | 200 GB / 400 GB | 600 Mbit/s | Unlimited | **EUR 14.00** (~$15.40) | -| Cloud VPS 40 | 12 | 48 GB | 250 GB / 500 GB | 800 Mbit/s | Unlimited | **EUR 25.00** | -| Cloud VPS 50 | 16 | 64 GB | 300 GB / 600 GB | 1 Gbit/s | Unlimited | **EUR 37.00** | - -**EU Data Center Locations:** -- EU (auto-assigned -- Germany/France typically) -- UK -- Non-EU regions (USA, Asia, Australia) available for extra fee - -**DOES NOT have a Turkey data center.** - -**Docker Support:** YES -- KVM-based, full root access. -**Managed K8s:** No. - -**Note:** Contabo has very generous specs for the price but lower port speeds (200-600 Mbit/s on entry plans) compared to Hetzner (1 Gbit/s on all plans). Performance benchmarks also show lower I/O than Hetzner. - -Sources: [Contabo Pricing](https://contabo.com/en/pricing/) | [Contabo EU Locations](https://contabo.com/en/locations/europe/) | [VPSBenchmarks](https://www.vpsbenchmarks.com/compare/contabo) - ---- - -### 1.4 OVHcloud (France-based) - -Large EU provider with extensive European presence. Good network infrastructure. - -| Plan | vCores | RAM | Storage | Bandwidth | Price/mo | -|------|--------|-----|---------|-----------|----------| -| VPS-1 | 1 | 2 GB | 20 GB SSD | 100 Mbps | ~$4.20 | -| VPS-2 | 4 | 8 GB | 75 GB SSD | 400 Mbps | ~$6.75 | -| VPS-3 | 6 | 12 GB | 100 GB NVMe | 1 Gbps | ~$12.75 | -| VPS-4 | 8 | 24 GB | 200 GB NVMe | 1.5 Gbps | ~$22.08 | -| VPS-5 | 12 | 48 GB | 300 GB NVMe | 2 Gbps | ~$34.34 | - -**EU Data Center Locations:** -- France: Paris, Roubaix, Gravelines, Strasbourg -- Germany: Frankfurt -- Poland: Warsaw -- UK: London -- Under construction: Italy, Netherlands, Spain - -**15+ Local Zones** available for VPS across EU cities. -**DOES NOT have a Turkey data center.** - -**Docker Support:** YES -- KVM-based VPS. -**Managed K8s:** YES -- OVHcloud offers managed Kubernetes service. - -Sources: [OVHcloud VPS](https://us.ovhcloud.com/vps/) | [OVHcloud Locations](https://us.ovhcloud.com/about/global-infrastructure/locations/) - ---- - -### 1.5 Scaleway (France-based, EU-sovereign) - -European-focused cloud with strong GDPR compliance. All data centers in EU. - -#### Development Instances (Paris only, shared, best for dev/test) - -| Plan | vCPU | RAM | Bandwidth | Price/mo | -|------|------|-----|-----------|----------| -| STARDUST1-S | 1 | 1 GB | 100 Mbps | ~EUR 0.11 (excl. IPv4) | -| DEV1-S | 2 | 2 GB | 200 Mbps | ~EUR 6.42 | -| DEV1-M | 3 | 4 GB | 300 Mbps | ~EUR 14.45 | -| DEV1-L | 4 | 8 GB | 400 Mbps | ~EUR 30.66 | - -#### PLAY2 Instances (all regions) - -| Plan | vCPU | RAM | Bandwidth | Price/mo | -|------|------|-----|-----------|----------| -| PLAY2-PICO | 1 | 2 GB | 100 Mbps | ~EUR 10.22 | -| PLAY2-NANO | 2 | 4 GB | 200 Mbps | ~EUR 19.71 | -| PLAY2-MICRO | 4 | 8 GB | 400 Mbps | ~EUR 39.42 | - -#### Dedicated vCPU - -| Plan | vCPU | RAM | Bandwidth | Price/mo | -|------|------|-----|-----------|----------| -| POP2-2C-8G | 2 | 8 GB | 400 Mbps | ~EUR 53.65 | - -**Data Center Locations:** -- Paris (multiple AZs) -- Amsterdam -- Warsaw - -**DOES NOT have a Turkey data center.** Warsaw is the closest to Turkey (~1600km). - -**Docker Support:** YES -- full Linux instances, Docker works natively. -**Managed K8s:** YES -- Scaleway Kapsule (managed Kubernetes). - -**Assessment:** Scaleway is significantly more expensive than Hetzner/Contabo for equivalent specs. EUR 30.66/mo for 4 vCPU / 8 GB RAM vs Hetzner CX33 at EUR 5.49/mo. Not competitive for budget-focused deployments. - -Sources: [Scaleway Pricing](https://www.scaleway.com/en/pricing/virtual-instances/) | [Scaleway Locations](https://www.scaleway.com/en/docs/) - ---- - -## Part 2: Turkish VPS Providers (Istanbul/Ankara Data Centers) - -### 2.1 Turhost (Founded 2004, Turkish) - -One of Turkey's longest-running hosting providers. Data centers at Turk Telekom Istanbul Gayrettepe (Tier 3 certified) and Ankara. - -**VPS Features:** -- NVMe SAN storage, Intel Xeon E5 processors -- 100 Mbit dedicated connection per VPS -- Daily backups stored for 30 days ("Time Machine") -- cPanel/Plesk/DirectAdmin support (Turkish language) -- DDoS protection included - -**Pricing:** Listed in Turkish Lira (TRY), varies by configuration. Prices exclude 20% VAT (KDV). Specific tier pricing requires visiting their site directly as it changes frequently with TRY exchange rates. - -**Docker Support:** YES -- KVM-based VPS with root access. Docker can be installed manually. -**Managed K8s:** No. - -**Strengths:** Local Turkish support, Turkish-language assistance, data sovereignty under KVKK law. -**Weaknesses:** Prices in TRY fluctuate with exchange rate; limited international connectivity compared to Hetzner/OVH. - -Sources: [Turhost VPS](https://www.turhost.com/sunucu/vps-server/) | [Hostings.info Turkey](https://hostings.info/hosting/ratings/turkey) - ---- - -### 2.2 Radore (Founded 2004, Istanbul) - -Premium Turkish data center provider. Located at MetroCity - Levent, Istanbul. -Over 3,500 customers. 99.99% uptime guarantee. "Uptime Experts" branding. - -**Cloud Server (RCD - Radore Cloud Datacenter):** -- Proxmox-based VPS solutions -- Portal-based management -- Scalable resources (CPU/RAM/storage) -- Tier 3 data center, 3,020 m2 space - -**Pricing (approximate, from older source):** -- R-OnApp: 1 CPU, 1 GB RAM, 20 GB storage, unlimited bandwidth -- ~$8.78/mo -- R-Cloud: 1 CPU, 1 GB RAM, 20 GB storage, unlimited bandwidth -- ~$15.70/mo - -**Note:** Radore uses quote-based pricing for larger configurations. Contact satis@radore.com for current rates. - -**Docker Support:** YES -- Proxmox-based virtualization supports Docker. -**Managed K8s:** Not publicly listed. Offers managed services upon request. - -**Strengths:** Enterprise-grade Turkish data center, good for compliance-sensitive workloads. -**Weaknesses:** Quote-based pricing is opaque; significantly more expensive than international providers. - -Sources: [Radore](https://radore.com/) | [Radore RCD](https://radore.com/en/rcd) - ---- - -### 2.3 Natro (Founded 1999, Part of team.blue) - -Long-standing Turkish hosting provider, now part of the team.blue group (European hosting conglomerate). - -**VPS Plans (XCloud):** - -| Plan | vCPU | RAM | Storage | -|------|------|-----|---------| -| XCloud Mini | 1 | 1 GB | 20 GB SSD | -| ... (8 tiers) | ... | ... | ... | -| XCloud Ultra+ | 16 | 64 GB | 1 TB SSD | - -**Features:** -- 100% SSD, full redundancy -- DDoS protection, free backup -- 1-click app installs (CMS, E-Commerce) -- OS options: AlmaLinux, Ubuntu, Debian, Windows Server 2019 -- Monthly or annual billing -- 14-day refund policy - -**Pricing:** Listed in TRY on their website. Significant new-customer discounts that expire on renewal. - -**Docker Support:** YES -- KVM/VDS with root access. Docker installable. -**Managed K8s:** No. - -Sources: [Natro VPS](https://www.natro.com/sunucu-kiralama/vps-cloud-server) | [Natro Review](https://www.websiteplanet.com/web-hosting/natro/) - ---- - -### 2.4 Medianova (Founded 2005, Istanbul) - -**NOT a VPS provider.** Medianova is a CDN and cloud security company. - -- Largest CDN in Turkey, Middle East, and Africa -- 50+ PoPs in 21 countries, 11 PoPs in 3 Turkish cities -- 100% SSD Anycast network -- Pay-as-you-go: ~$0.04-$0.20/GB data transfer -- Enterprise customers: Turkcell, Vodafone, Hepsiburada - -**Relevance:** Could be used as a CDN layer in front of the Identity Core API, but does NOT offer VPS/compute instances. - -Sources: [Medianova](https://www.medianova.com/) | [CDN Planet](https://www.cdnplanet.com/cdns/medianova/) - ---- - -### 2.5 DigitalOcean Istanbul -- NOT AVAILABLE - -DigitalOcean does **NOT** have a data center in Istanbul or anywhere in Turkey as of February 2026. -A Turkey data center has been a popular community request (493+ votes) with "gathering feedback" status, but no announcement has been made. - -DigitalOcean's closest region to Turkey is **Frankfurt, Germany** or **London, UK**. - -Sources: [DigitalOcean Turkey Request](https://ideas.digitalocean.com/infrastructure/p/turkey-data-center) | [DO Locations](https://docs.digitalocean.com/platform/regional-availability/) - ---- - -### 2.6 Hetzner Turkey -- NOT AVAILABLE - -Hetzner does **NOT** have any data centers in Turkey. -Their locations are: Germany (Falkenstein, Nuremberg), Finland (Helsinki), USA (Ashburn, Hillsboro), Singapore. - -Closest to Turkey: Nuremberg (~1800km), Helsinki (~2200km). - ---- - -### 2.7 HostArmada (International, HAS Turkey/Istanbul location) - -International provider with a data center in Istanbul, Turkey. Unmanaged VPS with full root access. - -**VPS Plans:** - -| Plan | vCPU | RAM | NVMe Storage | Bandwidth | Promo Price/mo | -|------|------|-----|-------------|-----------|----------------| -| Spark | 1 | 1 GB | 40 GB | 2 TB | $3.69 | -| Fusion | 4 | 8 GB | 160 GB | 6 TB | $10.74 | - -**Features:** -- 17 Tbit/s DDoS protection -- Dedicated IPv4 with reverse DNS -- Free automated backups + manual snapshots -- 23 data centers worldwide (including Istanbul) -- 7-day money-back guarantee - -**Docker Support:** YES -- full root access, KVM-based. -**Managed K8s:** No. - -Sources: [HostArmada VPS](https://hostarmada.com/vps-hosting/) | [HostArmada Pricing](https://hostarmada.com/pricing/) - ---- - -### 2.8 Other Notable Turkey VPS Providers - -| Provider | Notes | -|----------|-------| -| **HOSTKEY** | International provider with VPS servers in Istanbul | -| **UltaHost** | Istanbul VPS available, fast servers | -| **LightNode** | Turkey VPS with Istanbul location | -| **PQ Hosting** | VPS in Turkey (Istanbul/Izmir) | -| **Serverspace** | International cloud provider, Turkey presence | -| **Hosting.com.tr** | Turkish domain registrar + hosting | -| **Isimtescil.net** | Founded 1998, Turkish registrar + hosting | -| **Atakdomain.com** | Founded 2003, Turkish hosting provider | - ---- - -## Part 3: EU Providers with Good Turkey Proximity - -### Proximity to Istanbul (approximate network latency) - -| Location | Distance to Istanbul | Expected Latency | -|----------|---------------------|------------------| -| **Sofia, Bulgaria** | ~500 km | ~10-15 ms | -| **Bucharest, Romania** | ~600 km | ~10-15 ms | -| **Athens, Greece** | ~550 km | ~10-15 ms | -| **Warsaw, Poland** | ~1400 km | ~25-35 ms | -| **Frankfurt, Germany** | ~1800 km | ~35-50 ms | -| **Nuremberg, Germany** | ~1800 km | ~35-50 ms | -| **Amsterdam, Netherlands** | ~2200 km | ~45-60 ms | -| **Paris, France** | ~2300 km | ~45-65 ms | -| **Helsinki, Finland** | ~2200 km | ~40-55 ms | - -**Key Insight:** No major budget VPS provider (Hetzner, Contabo, OVH, Scaleway) has data centers in Romania, Bulgaria, or Greece. The closest EU locations to Turkey are: -- **OVHcloud Warsaw** (Poland) -- ~25-35ms to Istanbul -- **Scaleway Warsaw** (Poland) -- ~25-35ms to Istanbul -- **Hetzner Nuremberg/Falkenstein** (Germany) -- ~35-50ms to Istanbul -- **Hostinger Lithuania** (Vilnius) -- ~30-40ms to Istanbul - -For Turkey-located servers, **HostArmada Istanbul** or local Turkish providers (Turhost, Natro, Radore) are the only options. - ---- - -## Part 4: Docker Support Analysis - -### Do ALL VPS providers support Docker? - -**Short answer: YES, if they use KVM or full virtualization.** All providers listed in this document support Docker. - -### The Docker Compatibility Rule - -| Virtualization Type | Docker Support | Notes | -|--------------------|---------------|-------| -| **KVM** | FULL support | All providers above use KVM | -| **Xen HVM** | FULL support | Rare in 2026 | -| **OpenVZ 7+** | Partial | Needs host-level config, unreliable | -| **OpenVZ 6** | NO support | Cannot run Docker at all | -| **LXC/LXD** | Limited | Nested containers, not recommended | - -### Provider-by-Provider Docker Status - -| Provider | Virtualization | Docker? | Notes | -|----------|---------------|---------|-------| -| Hostinger | KVM | YES | Full support, Docker templates available | -| Hetzner | KVM | YES | Full support, Docker CE preinstallable | -| Contabo | KVM | YES | Full support | -| OVHcloud | KVM | YES | Full support | -| Scaleway | KVM | YES | Full support, managed K8s (Kapsule) | -| Turhost | KVM | YES | Manual install required | -| Radore | Proxmox/KVM | YES | Full support | -| Natro | KVM/VDS | YES | Manual install required | -| HostArmada | KVM | YES | Full support | - -**WARNING:** Some ultra-cheap VPS providers (not listed here) still use OpenVZ 6/7, which does NOT reliably support Docker. Always verify the virtualization technology is KVM before purchasing a VPS for Docker workloads. - -### Managed Kubernetes Options - -| Provider | Service | Notes | -|----------|---------|-------| -| Hetzner | Managed K8s | Affordable, EU-only | -| OVHcloud | Managed K8s | EU data centers, GDPR-compliant | -| Scaleway | Kapsule | Paris/Amsterdam/Warsaw | -| DigitalOcean | DOKS | No Turkey location | -| None of the Turkish providers | -- | No managed K8s offerings found | - ---- - -## Part 5: Price Comparison Matrix (2-4 vCPU, 4-8 GB RAM) - -### Target: VPS suitable for Java API + PostgreSQL (4-8 GB RAM range) - -| Provider | Plan | vCPU | RAM | Storage | Price/mo | Location Options | -|----------|------|------|-----|---------|----------|-----------------| -| **Hetzner** | CX23 | 2 | 4 GB | 40 GB NVMe | **EUR 3.49** | DE, FI | -| **Hetzner** | CX33 | 4 | 8 GB | 80 GB NVMe | **EUR 5.49** | DE, FI | -| **Contabo** | VPS 10 | 4 | 8 GB | 75 GB NVMe | **EUR 4.50** | EU (auto) | -| **OVHcloud** | VPS-1 | 1 | 2 GB | 20 GB SSD | ~$4.20 | EU (13+ cities) | -| **OVHcloud** | VPS-2 | 4 | 8 GB | 75 GB SSD | ~$6.75 | EU (13+ cities) | -| **Hostinger** | KVM 1 | 1 | 4 GB | 50 GB NVMe | $4.99* | NL, LT, UK, DE | -| **Hostinger** | KVM 2 | 2 | 8 GB | 100 GB NVMe | $6.99* | NL, LT, UK, DE | -| **HostArmada** | Fusion | 4 | 8 GB | 160 GB NVMe | $10.74* | **Istanbul**, EU, US | -| **Scaleway** | DEV1-M | 3 | 4 GB | -- | ~EUR 14.45 | Paris | -| **Scaleway** | DEV1-L | 4 | 8 GB | -- | ~EUR 30.66 | Paris | - -*Promo pricing with annual/multi-year commitment - -### Best Value Rankings (for EU deployment) - -1. **Hetzner CX33** -- EUR 5.49/mo for 4 vCPU, 8 GB RAM, 80 GB NVMe, 20 TB traffic (Germany/Finland) -2. **Contabo VPS 10** -- EUR 4.50/mo for 4 vCPU, 8 GB RAM, 75 GB NVMe, unlimited traffic (EU) -3. **OVHcloud VPS-2** -- ~$6.75/mo for 4 vCores, 8 GB RAM, 75 GB SSD (EU wide) -4. **Hostinger KVM 2** -- $6.99/mo for 2 vCPU, 8 GB RAM, 100 GB NVMe (EU) -5. **Hetzner CX23** -- EUR 3.49/mo for 2 vCPU, 4 GB RAM, 40 GB NVMe (if 4 GB is enough) - -### Best Value for Turkey Proximity - -1. **HostArmada Fusion (Istanbul)** -- $10.74/mo, 4 vCPU, 8 GB RAM, in Istanbul itself (~0-5ms) -2. **Hetzner CX33 (Nuremberg)** -- EUR 5.49/mo, 4 vCPU, 8 GB RAM (~35-50ms to Istanbul) -3. **OVHcloud VPS-2 (Warsaw)** -- ~$6.75/mo, 4 vCores, 8 GB RAM (~25-35ms to Istanbul) -4. **Turhost/Natro (Istanbul)** -- Pricing in TRY, check current rates (~0-5ms, local) -5. **Hostinger KVM 2 (Lithuania)** -- $6.99/mo, 2 vCPU, 8 GB RAM (~30-40ms to Istanbul) - ---- - -## Part 6: GPU Providers (for Biometric Processor) - -### Dedicated GPU Instances (always-on) - -| Provider | GPU | VRAM | vCPUs | RAM | Hourly | Monthly (730h) | -|----------|-----|------|-------|-----|--------|-----------------| -| **Vast.ai** | T4 | 16 GB | varies | varies | $0.10-0.20 | ~$73-146 | -| **RunPod** | RTX 3090 | 24 GB | -- | -- | ~$0.20 | ~$146 | -| **AWS g4dn.xlarge (Spot)** | T4 | 16 GB | 4 | 16 GB | ~$0.16 | ~$117 | -| **Hetzner GEX44** | RTX 4000 Ada | 20 GB | 14 | -- | -- | ~$194 | -| **RunPod** | T4 | 16 GB | -- | -- | $0.40 | ~$292 | -| **AWS g4dn.xlarge (On-Demand)** | T4 | 16 GB | 4 | 16 GB | $0.526 | ~$384 | -| **Google Cloud** | T4 (n1) | 16 GB | 4 | 15 GB | $0.55-0.75 | ~$400-550 | -| **Azure** | NC T4 v3 | 16 GB | 4 | 28 GB | ~$0.53 | ~$387 | -| **Google Cloud** | L4 (g2) | 24 GB | 4-8 | 16-32 GB | $0.67-0.85 | ~$490-620 | -| **AWS g5.xlarge** | A10G | 24 GB | 4 | 16 GB | $1.006 | ~$734 | -| **Lambda Cloud** | A6000 | 48 GB | -- | -- | $0.80 | ~$584 | -| **Hetzner GEX130** | RTX 6000 Ada | 48 GB | 24 | 128 GB | -- | ~$885 | - -### Serverless GPU (pay-per-inference, scales to zero) - -Best for bursty/low-volume workloads. You pay nothing when idle. - -| Provider | GPU | Cost/sec | 10K req/mo (500ms each) | 100K req/mo | -|----------|-----|----------|------------------------|-------------| -| **Modal** | T4 | ~$0.000164 | ~$0.82 | ~$8 | -| **RunPod Serverless** | T4 | ~$0.00022 | ~$1.11 | ~$11 | -| **Replicate** | T4 | ~$0.00055 | ~$2.75 | ~$28 | -| **AWS SageMaker Serverless** | T4 | ~$0.00088 | ~$4.40 | ~$44 | - -**Break-even**: Serverless becomes more expensive than dedicated GPU at ~200K-500K requests/month. - ---- - -## Part 7: Recommended Configurations by Scale - -| Scale | API Server | Biometric Processor | Est. Total | -|-------|-----------|---------------------|------------| -| **Dev/MVP** | Hetzner CX23 (EUR 3.49) | RunPod/Modal Serverless | $5-15/mo | -| **Small prod** (<50K users) | Hetzner CX33 (EUR 5.49) | RunPod Serverless | $10-50/mo | -| **Small prod (Turkey)** | HostArmada Istanbul ($10.74) | RunPod Serverless | $12-55/mo | -| **Medium prod** (50-500K users) | Hetzner CX33 (EUR 5.49) | Hetzner GEX44 ($194) | ~$200/mo | -| **Large prod** (500K+ users) | AWS/GCP VPS ($24-48) | AWS g4dn Spot + fallback | $150-400/mo | - ---- - -## Part 8: Key Decisions - -### For the VPS (Java API) -- **Hetzner CX33** is the clear winner for EU: EUR 5.49/mo for 4 vCPU, 8 GB RAM, 80 GB NVMe -- **Contabo VPS 10** is a close second at EUR 4.50/mo with more RAM but lower I/O performance -- **HostArmada Istanbul** if Turkey data residency is required (~$10.74/mo) -- 4 GB RAM is sufficient for the API alone; 8 GB gives headroom for PostgreSQL + Redis on same node -- Avoid Scaleway for VPS -- 3-6x more expensive than Hetzner for comparable specs - -### For Turkey-Specific Deployments -- **Data residency under KVKK:** Use HostArmada Istanbul, Turhost, or Natro -- **Best latency to Turkey from EU:** OVHcloud Warsaw or Scaleway Warsaw (~25-35ms) -- **Best price near Turkey:** Hetzner Nuremberg at EUR 5.49/mo (~35-50ms latency) -- **Local Turkish providers** (Turhost, Natro, Radore) are more expensive and have less transparent pricing, but offer Turkish-language support and KVKK compliance - -### For the Biometric Processor -- **T4 (16GB VRAM)** is the sweet spot for FaceNet512/VGG-Face inference -- Start with **serverless GPU** (Modal or RunPod) to minimize costs during development -- Move to **dedicated GPU** (Hetzner GEX44 or AWS Spot) when throughput demands it - -### Docker Considerations -- ALL providers in this comparison support Docker (all use KVM) -- Avoid any provider using OpenVZ (some ultra-budget providers still do) -- For managed container orchestration, Hetzner K8s or OVHcloud K8s are the most affordable EU options - ---- - -## Sources - -### VPS Providers -- [Hostinger VPS Pricing](https://www.hostinger.com/pricing/vps-hosting) | [Hostinger Locations](https://hostingadvices.co.uk/hostinger-vps-locations/) -- [Hetzner Cloud](https://www.hetzner.com/cloud) | [Hetzner Locations](https://docs.hetzner.com/cloud/general/locations/) -- [Contabo Pricing](https://contabo.com/en/pricing/) | [Contabo EU](https://contabo.com/en/locations/europe/) -- [OVHcloud VPS](https://us.ovhcloud.com/vps/) | [OVHcloud Locations](https://us.ovhcloud.com/about/global-infrastructure/locations/) -- [Scaleway Pricing](https://www.scaleway.com/en/pricing/virtual-instances/) - -### Turkish Providers -- [Turhost VPS](https://www.turhost.com/sunucu/vps-server/) -- [Radore Data Center](https://radore.com/) | [Radore RCD](https://radore.com/en/rcd) -- [Natro VPS](https://www.natro.com/sunucu-kiralama/vps-cloud-server) -- [Medianova CDN](https://www.medianova.com/) -- [HostArmada VPS](https://hostarmada.com/vps-hosting/) | [HostArmada Pricing](https://hostarmada.com/pricing/) - -### Turkey VPS Market -- [Best Turkey VPS 2026 - HostAdvice](https://hostadvice.com/vps/turkey/) -- [Turkey VPS Ratings - Hostings.info](https://hostings.info/hosting/ratings/best-vpsvds-hosting-providers-in-turkey) -- [DigitalOcean Turkey Request](https://ideas.digitalocean.com/infrastructure/p/turkey-data-center) - -### GPU Providers -- [Hetzner GPU](https://www.hetzner.com/dedicated-rootserver/matrix-gpu/) -- [RunPod](https://www.runpod.io/pricing) | [Vast.ai](https://vast.ai/pricing) | [Lambda Cloud](https://lambda.ai/pricing) -- [Modal](https://modal.com/pricing) | [Replicate](https://replicate.com/pricing) -- [AWS EC2 G4/G5](https://aws.amazon.com/ec2/instance-types/g4/) | [AWS SageMaker](https://aws.amazon.com/sagemaker/pricing/) -- [Google Cloud GPUs](https://cloud.google.com/compute/gpus-pricing) | [Azure NC Series](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nc-family) - -### Docker/Virtualization -- [KVM vs OpenVZ for Docker](https://petrosky.io/kvm-vps-vs-openvz-2025/) -- [Docker VPS Hosting - HostAdvice](https://hostadvice.com/docker-hosting/docker-vps-hosting/) -- [VPSBenchmarks](https://www.vpsbenchmarks.com/) - -*Research date: February 2026*