fix/indexed-search: add livenessProbe to zoekt-webserver to prevent hung pods

[devdinu]

zoekt-webserver was unhealthy for long time

NAME               READY   STATUS    RESTARTS   AGE    
indexed-search-0   1/2     Running   0          6h18m  
indexed-search-1   1/2     Running   0          6h19m  
indexed-search-2   1/2     Running   0          6h21m  
indexed-search-3   1/2     Running   0          6h23m  
indexed-search-4   1/2     Running   0          6h25m  
indexed-search-5   1/2     Running   0          6h27m  
indexed-search-6   1/2     Running   0          6h29m  
indexed-search-7   1/2     Running   0          6h31m  

  Warning  Unhealthy  4m20s (x3300 over 4h39m)  kubelet  Readiness probe failed: Get "http://192.168.11.20:6070/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

ref https://linear.app/sourcegraph/issue/PLAT-509/incident-indexed-search-pods-were-unhealthy-for-long-time PLAT-509

Checklist

• Follow the manual testing process
• Update changelog
• Update Kubernetes update doc

Test plan

• Manual Verification - local with kind cluster

• Tested against cloud-dev-qa with mi2 generate kustomize and kustomize apply

[devdinu]

• fix/indexed-search: add livenessProbe to zoekt-webserver to prevent hung pods #850 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[michaellzc]

can you quickly validate this on a cloud instance?

[devdinu]

can you quickly validate this on a cloud instance?

Tested diff and applied against clouddev-qa, only change's indexed-search.

https://sourcegraph.slack.com/archives/C05DWT4ANHH/p1774644755678949

[keegancsmith]

so if the pod is hung, it will take 10 minutes before we kill it? That seems alright.

FYI I kinda remember us intentionally not doing this back in the day. But it likely had to do with hard to control stuff with on-prem + kubernetes being more immature. Which is why we extending the watchdog stuff. But yeah, the recent incident I think caused the go runtime to kinda crash so we need this check to not be internal.

Here is the history I managed to find in the infrastructure repo!

Yes, a liveness probe was set for zoekt-webserver, but it was short-lived. The K8s definitions lived under kubernetes/ in this repo (as Go code using a kubegen library).

• Added on Feb 2, 2018 by Nick Snyder in commit 82c2aa37 — an HTTP liveness probe hitting /healthz on port http (6070) with 5s timeout and 5s initial delay.
• Removed 4 days later on Feb 6, 2018 by Beyang Liu in commit d3749e67 with the message: "indexed-search: remove liveness check (it can erroneously kill zoekt-webserver)".

So it existed for about 4 days before being pulled because it was causing false-positive failures that killed the pod.

https://ampcode.com/threads/T-019d3db6-b71c-739f-855d-aca839ba9170