Add support for custom user domains for webhooks/responders

[azasypkin]

Summary

Currently, webhooks support two URL types: path-based (/api/webhooks/{user_handle}/{path}) and subdomain-based ({prefix}-{user_handle}.webhooks.secutils.dev/{path}). To support completely arbitrary customer-owned domains (e.g., api.customer.com/webhook), the system needs changes at multiple layers: domain verification/ownership, database storage, API validation, request routing, TLS certificate provisioning, and UI configuration.

Steps

1. Add domain ownership verification system

Create new database table user_data_webhooks_custom_domains in migrations/ to store user-owned custom domains with verification and TLS status:

CREATE TABLE user_data_webhooks_custom_domains (
    domain TEXT PRIMARY KEY,
    user_id BLOB NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    verification_token TEXT NOT NULL,
    verified_at BIGINT,           -- NULL until verified
    tls_provisioned_at BIGINT,    -- NULL until cert ready
    created_at BIGINT NOT NULL
);

Verification flow:

• User adds domain api.customer.com
• System generates verification token (e.g., secutils-verify=abc123xyz)
• User creates DNS TXT record: _secutils-verify.api.customer.com TXT "secutils-verify=abc123xyz"
• API endpoint checks DNS and updates verified_at timestamp

Create API endpoints in src/utils/webhooks/api_ext.rs:

• POST /api/utils/webhooks/custom_domains — Add domain, generate token
• GET /api/utils/webhooks/custom_domains — List user's domains
• POST /api/utils/webhooks/custom_domains/{domain}/verify — Trigger verification check
• DELETE /api/utils/webhooks/custom_domains/{domain} — Remove domain and cleanup cert

2. Refactor ResponderLocation to use enum for host type

Currently subdomain_prefix: Option<String> implicitly determines routing type. With custom domains, there are now three mutually exclusive host types. Refactor src/utils/webhooks/responders/responder_location.rs to make this explicit:

/// Determines how the webhook responder is accessed.
#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum ResponderHostType {
    /// Path-only: /api/webhooks/{user_handle}/{path}
    /// No custom host, simplest option.
    PathOnly,
    /// Subdomain: {prefix}-{user_handle}.webhooks.secutils.dev/{path}
    /// Uses Secutils-managed subdomain with wildcard TLS.
    Subdomain {
        prefix: String,
    },
    /// Custom domain: api.customer.com/{path}
    /// User-owned domain, requires verification and per-domain TLS.
    CustomDomain {
        domain: String,
    },
}

#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct ResponderLocation {
    pub path_type: ResponderPathType,
    pub path: String,
    pub host_type: ResponderHostType,
}

JSON serialization examples:

// Path-only
{ "pathType": "=", "path": "/hook", "hostType": { "type": "pathOnly" } }

// Subdomain
{ "pathType": "^", "path": "/api", "hostType": { "type": "subdomain", "prefix": "myapp" } }

// Custom domain
{ "pathType": "=", "path": "/webhook", "hostType": { "type": "customDomain", "domain": "api.customer.com" } }

Migration: Existing data uses subdomain_prefix column. Migration converts:

• subdomain_prefix IS NULL → ResponderHostType::PathOnly
• subdomain_prefix IS NOT NULL → ResponderHostType::Subdomain { prefix }

3. Implement automatic TLS provisioning via cert-manager (HTTP-01)

When a custom domain is verified, automatically provision a TLS certificate using cert-manager with HTTP-01 challenge (since we don't control customer DNS for DNS-01):

Process:

1. Customer points api.customer.com → Secutils ingress IP (CNAME or A record)
2. After DNS verification succeeds, API creates a cert-manager Certificate resource via Kubernetes API:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: custom-domain-api-customer-com
  namespace: secutils
  labels:
    secutils.dev/custom-domain: "true"
spec:
  secretName: tls-custom-api-customer-com
  issuerRef:
    name: letsencrypt-prod-issuer
    kind: ClusterIssuer
  dnsNames:
    - api.customer.com

3. cert-manager handles HTTP-01 challenge (creates temp ingress for /.well-known/acme-challenge/)
4. Certificate stored in secret, tls_provisioned_at updated
5. Traefik picks up the new TLS secret automatically

Rust implementation: Use kube crate to create/delete Certificate resources:

async fn provision_tls_certificate(kube_client: &Client, domain: &str) -> Result<()> {
    let certs: Api<Certificate> = Api::namespaced(kube_client.clone(), "secutils");
    let cert = Certificate { /* ... */ };
    certs.create(&PostParams::default(), &cert).await?;
    Ok(())
}

Cleanup: When domain is deleted, remove the Certificate resource (cert-manager cleans up the secret).

Rate limits to consider:

• Let's Encrypt: 50 certificates per registered domain per week
• 300 new orders per 3 hours per account
• cert-manager handles automatic renewal (30 days before expiry)

4. Update subscription config for custom domains

Add fields to src/config/subscriptions_config/subscription_webhooks_config.rs:

pub struct SubscriptionWebhooksConfig {
    // ...existing fields...
    /// Whether custom domains are allowed for this subscription tier.
    pub custom_domains_enabled: bool,
    /// Maximum number of custom domains user can register.
    pub max_custom_domains: usize,
}

Extend ClientSubscriptionWebhooksConfig in src/users/user_subscription/client_subscription_features.rs to expose to UI.

Tier limits:

• Free: 0 custom domains
• Basic: 1 custom domain
• Professional: 5 custom domains
• Ultimate: Unlimited

5. Modify API validation and request routing

Validation in src/utils/webhooks/api_ext.rs:

• When ResponderHostType::CustomDomain { domain } is used, verify:
  • Domain exists in user_data_webhooks_custom_domains for this user
  • Domain is verified (verified_at IS NOT NULL)
  • TLS is provisioned (tls_provisioned_at IS NOT NULL)
  • User hasn't exceeded max_custom_domains limit

Request routing in src/server/handlers/webhooks_responders.rs:

Update parse_webhook_host() to handle custom domains:

async fn parse_webhook_host(host: &str, db: &Database) -> Result<WebhookHost> {
    // 1. Check if it's a known Secutils subdomain pattern
    if let Some(captures) = SUBDOMAIN_REGEX.captures(host) {
        return Ok(WebhookHost::Subdomain { prefix, user_handle });
    }
    
    // 2. Check if it's a verified custom domain
    if let Some(custom_domain) = db.get_verified_custom_domain(host).await? {
        return Ok(WebhookHost::CustomDomain { 
            domain: host.to_string(),
            user_id: custom_domain.user_id,
        });
    }
    
    // 3. Unknown host
    Err(Error::UnknownWebhookHost)
}

6. Configure Traefik for custom domain routing

Current static HostRegexp won't work for arbitrary domains. Use catch-all route with application-level routing:

# Existing subdomain routes (higher priority)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: secutils-webhooks-subdomains-https
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostRegexp(`[a-z0-9-]+\.webhooks\.secutils\.dev`)
      priority: 100
      services:
        - name: secutils-api-svc
          port: 7070
  tls:
    secretName: secutils-webhooks-wildcard-tls

---
# Catch-all for custom domains (lower priority)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: secutils-webhooks-custom-domains-https
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: PathPrefix(`/`)
      priority: 1  # Low priority, only matches if nothing else does
      services:
        - name: secutils-api-svc
          port: 7070
  tls:
    passthrough: false
    # Traefik uses SNI to select correct certificate from secrets

TLS Secret Loading: Configure Traefik to watch secrets with label secutils.dev/custom-domain=true:

# In Traefik deployment args or config
--providers.kubernetessecret.labelselector=secutils.dev/custom-domain=true

7. Update WebUI for custom domain configuration

New "Custom Domains" management section:

• List verified domains with status (pending verification, verified, TLS ready)
• Add domain form with verification instructions
• "Verify Now" button to trigger DNS check
• Delete domain button

Update responder_edit_flyout.tsx:

• Radio button group for host type selection:
  • "Path only" (default)
  • "Subdomain" (shows prefix input)
  • "Custom domain" (shows dropdown of verified domains)
• Custom domain option disabled if none verified or subscription doesn't allow

Update responders.tsx:

• Display full URL based on host type
• Show warning icon if custom domain TLS not yet ready

Implementation Order

1. Database migration + domain CRUD API (no TLS yet)
2. Refactor ResponderLocation to use ResponderHostType enum + migration
3. Update API validation to check domain ownership
4. Add TLS provisioning via cert-manager
5. Update Traefik configuration
6. WebUI: Custom domains management page
7. WebUI: Responder editor with host type selection
8. Subscription tier limits

Further Considerations

1. DNS verification approach: DNS TXT record is standard (similar to Google/AWS domain verification).

2. What if TLS provisioning fails? Add status field to track provisioning state. UI shows error with retry option. Common failures: DNS not pointed correctly, rate limited. Consider background job to retry failed provisions.

3. Domain transfer between users? If user A deletes domain and user B adds it, need to handle gracefully. Require re-verification. Consider cooldown period.

4. Wildcard custom domains? For v1, only exact domains. Wildcard (*.customer.com) would require DNS-01 challenge which needs customer DNS API access, maybe in the future.