Follow-up to #170, and the substance of CodeRabbit's fly.toml:50 Major that merged unaddressed.
Gap
#170's /ready gates Fly deploys with grace_period = "75s" (fly.toml:47). But a single connect+migrate attempt is bounded by db_connect_timeout_secs (default 60, config.rs:214), sized deliberately "generous enough for a peer instance to finish migrating" while it holds the migration advisory lock; on timeout it backs off db_retry_initial_secs (default 5) before attempt 2. /ready returns 503 the entire time.
Evidence (executed, arithmetic)
With default knobs: db_connect_timeout_secs(60) + db_retry_initial_secs(5) + db_connect_timeout_secs(60) = 125s, which exceeds the 75s grace_period. So two attempts plus one backoff already blow past the grace window while the node is healthily waiting.
Consequence (needs Fly-semantics confirmation)
During a rolling deploy where a peer instance holds the migration lock, a new instance can stay 503 past grace_period and Fly may roll back a node that is waiting exactly as designed. The knobs are internally inconsistent: the connect+retry budget can exceed the deploy grace window. (The config arithmetic is verified; whether Fly actually rolls back at grace_period is platform behavior I did not execute.)
Direction
Derive grace_period from db_connect_timeout_secs + the retry budget rather than a flat 75s below the single-attempt bound; or lower db_connect_timeout_secs; or have /ready distinguish "starting" so the deploy waits. Confirm Fly's exact grace_period-vs-rollback semantics before choosing.
Follow-up to #170, and the substance of CodeRabbit's
fly.toml:50Major that merged unaddressed.Gap
#170's
/readygates Fly deploys withgrace_period = "75s"(fly.toml:47). But a single connect+migrate attempt is bounded bydb_connect_timeout_secs(default 60,config.rs:214), sized deliberately "generous enough for a peer instance to finish migrating" while it holds the migration advisory lock; on timeout it backs offdb_retry_initial_secs(default 5) before attempt 2./readyreturns 503 the entire time.Evidence (executed, arithmetic)
With default knobs:
db_connect_timeout_secs(60) + db_retry_initial_secs(5) + db_connect_timeout_secs(60) = 125s, which exceeds the 75sgrace_period. So two attempts plus one backoff already blow past the grace window while the node is healthily waiting.Consequence (needs Fly-semantics confirmation)
During a rolling deploy where a peer instance holds the migration lock, a new instance can stay 503 past
grace_periodand Fly may roll back a node that is waiting exactly as designed. The knobs are internally inconsistent: the connect+retry budget can exceed the deploy grace window. (The config arithmetic is verified; whether Fly actually rolls back atgrace_periodis platform behavior I did not execute.)Direction
Derive
grace_periodfromdb_connect_timeout_secs+ the retry budget rather than a flat 75s below the single-attempt bound; or lowerdb_connect_timeout_secs; or have/readydistinguish "starting" so the deploy waits. Confirm Fly's exactgrace_period-vs-rollback semantics before choosing.