Skip to content

Add support for excluding specific containers in log analysis across multiple healthcheck runbooks #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stewartshea merged 3 commits into from
Mar 25, 2026
Merged

Add support for excluding specific containers in log analysis across multiple healthcheck runbooks #630

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6c891b7
Add support for excluding specific containers in log analysis across …
stewartshea Mar 25, 2026
cc8fa3d
Refactor logging calls to use `logger.warning` instead of `logger.war…
stewartshea Mar 25, 2026
04965f7
Update K8sLog to refine number filtering in log messages
stewartshea Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion codebundles/k8s-daemonset-healthcheck/runbook.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,15 @@ Analyze Application Log Patterns for DaemonSet `${DAEMONSET_NAME}` in Namespace
... context=${CONTEXT}
... kubeconfig=${kubeconfig}
... log_age=${LOG_AGE}
... excluded_containers=${EXCLUDED_CONTAINERS}

${scan_results}= RW.K8sLog.Scan Logs For Issues
... log_dir=${log_dir}
... workload_type=daemonset
... workload_name=${DAEMONSET_NAME}
... namespace=${NAMESPACE}
... categories=@{LOG_PATTERN_CATEGORIES}
... excluded_containers=${EXCLUDED_CONTAINERS}

${log_health_score}= RW.K8sLog.Calculate Log Health Score scan_results=${scan_results}

Expand Down Expand Up @@ -98,12 +100,14 @@ Detect Log Anomalies for DaemonSet `${DAEMONSET_NAME}` in Namespace `${NAMESPACE
... context=${CONTEXT}
... kubeconfig=${kubeconfig}
... log_age=${LOG_AGE}
... excluded_containers=${EXCLUDED_CONTAINERS}

${anomaly_results}= RW.K8sLog.Analyze Log Anomalies
... log_dir=${log_dir}
... workload_type=daemonset
... workload_name=${DAEMONSET_NAME}
... namespace=${NAMESPACE}
... excluded_containers=${EXCLUDED_CONTAINERS}

# Process anomaly issues
${anomaly_issues}= Evaluate $anomaly_results.get('issues', [])
Expand Down Expand Up @@ -945,9 +949,21 @@ Suite Initialization
... pattern=\d+
... example=1
... default=1
${EXCLUDED_CONTAINER_NAMES}= RW.Core.Import User Variable EXCLUDED_CONTAINER_NAMES
... type=string
... description=Comma-separated list of container names to exclude from log analysis (e.g., linkerd-proxy, istio-proxy, vault-agent).
... pattern=.*
... example=linkerd-proxy,istio-proxy,vault-agent
... default=linkerd-proxy,istio-proxy,vault-agent

# Convert comma-separated string to list
# Convert comma-separated strings to lists
@{LOG_PATTERN_CATEGORIES}= Split String ${LOG_PATTERN_CATEGORIES_STR} ,
@{EXCLUDED_CONTAINERS_RAW}= Run Keyword If "${EXCLUDED_CONTAINER_NAMES}" != "" Split String ${EXCLUDED_CONTAINER_NAMES} , ELSE Create List
@{EXCLUDED_CONTAINERS}= Create List
FOR ${container} IN @{EXCLUDED_CONTAINERS_RAW}
${trimmed_container}= Strip String ${container}
Append To List ${EXCLUDED_CONTAINERS} ${trimmed_container}
END

Set Suite Variable ${kubeconfig}
Set Suite Variable ${KUBERNETES_DISTRIBUTION_BINARY}
Expand All @@ -961,6 +977,8 @@ Suite Initialization
Set Suite Variable ${ANOMALY_THRESHOLD}
Set Suite Variable ${CONTAINER_RESTART_AGE}
Set Suite Variable ${CONTAINER_RESTART_THRESHOLD}
Set Suite Variable ${EXCLUDED_CONTAINER_NAMES}
Set Suite Variable @{EXCLUDED_CONTAINERS}
${env}= Evaluate {"KUBECONFIG":"${kubeconfig.key}","KUBERNETES_DISTRIBUTION_BINARY":"${KUBERNETES_DISTRIBUTION_BINARY}","CONTEXT":"${CONTEXT}","NAMESPACE":"${NAMESPACE}","DAEMONSET_NAME":"${DAEMONSET_NAME}","CONTAINER_RESTART_AGE":"${CONTAINER_RESTART_AGE}","CONTAINER_RESTART_THRESHOLD":"${CONTAINER_RESTART_THRESHOLD}"}
Set Suite Variable ${env}

Expand Down
47 changes: 43 additions & 4 deletions codebundles/k8s-deployment-healthcheck/runbook.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,12 @@ Suite Initialization
... pattern=.*
... example=linkerd-proxy,istio-proxy,vault-agent
... default=linkerd-proxy,istio-proxy,vault-agent
${CONTAINER_NAME}= RW.Core.Import User Variable CONTAINER_NAME
... type=string
... description=Optional: the specific container name to fetch logs from. If not set, the primary application container is auto-detected by excluding known sidecars.
... pattern=.*
... example=controller
... default=

${CONTAINER_RESTART_AGE}= RW.Core.Import User Variable CONTAINER_RESTART_AGE
... type=string
Expand Down Expand Up @@ -152,6 +158,7 @@ Suite Initialization
Set Suite Variable ${LOG_SCAN_TIMEOUT}
Set Suite Variable ${EXCLUDED_CONTAINER_NAMES}
Set Suite Variable @{EXCLUDED_CONTAINERS}
Set Suite Variable ${CONTAINER_NAME}

Set Suite Variable ${CONTAINER_RESTART_AGE}
Set Suite Variable ${CONTAINER_RESTART_THRESHOLD}
Expand Down Expand Up @@ -401,9 +408,41 @@ Fetch Deployment Logs for `${DEPLOYMENT_NAME}` in Namespace `${NAMESPACE}`
... data:logs-bulk
# Skip pod-related checks if deployment is scaled to 0
IF not ${SKIP_POD_CHECKS}
# Fetch raw logs
# Determine which container to fetch logs from
IF "${CONTAINER_NAME}" != ""
${target_container}= Set Variable ${CONTAINER_NAME}
ELSE
# Auto-detect primary container by listing containers and excluding known sidecars
${container_json}= RW.CLI.Run Cli
... cmd=${KUBERNETES_DISTRIBUTION_BINARY} get deployment/${DEPLOYMENT_NAME} --context ${CONTEXT} -n ${NAMESPACE} -o jsonpath='{.spec.template.spec.containers[*].name}'
... env=${env}
... secret_file__kubeconfig=${kubeconfig}
... include_in_history=false
@{all_containers}= Split String ${container_json.stdout}
${container_count}= Get Length ${all_containers}
${target_container}= Set Variable ${EMPTY}
IF ${container_count} > 0
FOR ${cname} IN @{all_containers}
${is_excluded}= Evaluate "${cname}" in ${EXCLUDED_CONTAINERS}
IF not ${is_excluded}
${target_container}= Set Variable ${cname}
BREAK
END
END
IF "${target_container}" == ""
${target_container}= Set Variable ${all_containers}[0]
END
END
END

# Build the kubectl logs command with or without -c flag
IF "${target_container}" != ""
${logs_cmd}= Set Variable ${KUBERNETES_DISTRIBUTION_BINARY} logs deployment/${DEPLOYMENT_NAME} -c ${target_container} --context ${CONTEXT} -n ${NAMESPACE} --tail=${LOG_LINES} --since=${LOG_AGE}
ELSE
${logs_cmd}= Set Variable ${KUBERNETES_DISTRIBUTION_BINARY} logs deployment/${DEPLOYMENT_NAME} --context ${CONTEXT} -n ${NAMESPACE} --tail=${LOG_LINES} --since=${LOG_AGE}
END
${deployment_logs}= RW.CLI.Run Cli
... cmd=${KUBERNETES_DISTRIBUTION_BINARY} logs deployment/${DEPLOYMENT_NAME} --context ${CONTEXT} -n ${NAMESPACE} --tail=${LOG_LINES} --since=${LOG_AGE}
... cmd=${logs_cmd}
... env=${env}
... secret_file__kubeconfig=${kubeconfig}
... show_in_rwl_cheatsheet=true
Expand All @@ -412,13 +451,13 @@ Fetch Deployment Logs for `${DEPLOYMENT_NAME}` in Namespace `${NAMESPACE}`
IF ${deployment_logs.returncode} == 0
# Filter logs to remove repetitive health check messages and focus on meaningful content
${filtered_logs}= RW.CLI.Run Cli
... cmd=echo "${deployment_logs.stdout}" | grep -v -E "(Checking.*Health|Health.*Check|healthcheck|/health|GET /|POST /health|probe|liveness|readiness)" | grep -E "(error|ERROR|warn|WARN|exception|Exception|fail|FAIL|fatal|FATAL|panic|stack|trace|timeout|connection.*refused|unable.*connect|authentication.*failed|denied|forbidden|unauthorized|500|502|503|504)" | tail -50 || echo "No significant errors or warnings found in recent logs"
... cmd=echo "${deployment_logs.stdout}" | grep -v -E "(Checking.*Health|Health.*Check|healthcheck|/health|GET /health|POST /health|probe|liveness|readiness)" | grep -E "(error|ERROR|warn|WARN|exception|Exception|fail|FAIL|fatal|FATAL|panic|stack|trace|timeout|connection.*refused|unable.*connect|authentication.*failed|denied|forbidden|unauthorized|500|502|503|504)" | tail -50 || echo "No significant errors or warnings found in recent logs"
... env=${env}
... include_in_history=false

# Also get a sample of non-health-check logs for context
${context_logs}= RW.CLI.Run Cli
... cmd=echo "${deployment_logs.stdout}" | grep -v -E "(Checking.*Health|Health.*Check|healthcheck|/health|GET /|POST /health|probe|liveness|readiness)" | head -20 | tail -10
... cmd=echo "${deployment_logs.stdout}" | grep -v -E "(Checking.*Health|Health.*Check|healthcheck|/health|GET /health|POST /health|probe|liveness|readiness)" | head -20 | tail -10
... env=${env}
... include_in_history=false

Expand Down
7 changes: 6 additions & 1 deletion codebundles/k8s-stacktrace-health/runbook.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,12 @@ Analyze Workload Stacktraces for ${WORKLOAD_TYPE} `${WORKLOAD_NAME}` in Namespac
# Skip pod-related checks if workload is scaled to 0
IF not ${SKIP_STACKTRACE_CHECKS}
# Convert comma-separated string to list for excluded containers
@{EXCLUDED_CONTAINERS}= Run Keyword If "${EXCLUDED_CONTAINER_NAMES}" != "" Split String ${EXCLUDED_CONTAINER_NAMES} , ELSE Create List
@{EXCLUDED_CONTAINERS_RAW}= Run Keyword If "${EXCLUDED_CONTAINER_NAMES}" != "" Split String ${EXCLUDED_CONTAINER_NAMES} , ELSE Create List
@{EXCLUDED_CONTAINERS}= Create List
FOR ${container} IN @{EXCLUDED_CONTAINERS_RAW}
${trimmed_container}= Strip String ${container}
Append To List ${EXCLUDED_CONTAINERS} ${trimmed_container}
END

# Fetch logs using RW.K8sLog library (same pattern as deployment healthcheck)
${log_dir}= RW.K8sLog.Fetch Workload Logs
Expand Down
7 changes: 6 additions & 1 deletion codebundles/k8s-stacktrace-health/sli.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,12 @@ Suite Initialization


# Convert comma-separated string to list
@{EXCLUDED_CONTAINERS}= Run Keyword If "${EXCLUDED_CONTAINER_NAMES}" != "" Split String ${EXCLUDED_CONTAINER_NAMES} , ELSE Create List
@{EXCLUDED_CONTAINERS_RAW}= Run Keyword If "${EXCLUDED_CONTAINER_NAMES}" != "" Split String ${EXCLUDED_CONTAINER_NAMES} , ELSE Create List
@{EXCLUDED_CONTAINERS}= Create List
FOR ${container} IN @{EXCLUDED_CONTAINERS_RAW}
${trimmed_container}= Strip String ${container}
Append To List ${EXCLUDED_CONTAINERS} ${trimmed_container}
END
Set Suite Variable @{EXCLUDED_CONTAINERS}

# Initialize score variables
Expand Down
20 changes: 19 additions & 1 deletion codebundles/k8s-statefulset-healthcheck/runbook.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,15 @@ Analyze Application Log Patterns for StatefulSet `${STATEFULSET_NAME}` in Namesp
... context=${CONTEXT}
... kubeconfig=${kubeconfig}
... log_age=${LOG_AGE}
... excluded_containers=${EXCLUDED_CONTAINERS}

${scan_results}= RW.K8sLog.Scan Logs For Issues
... log_dir=${log_dir}
... workload_type=statefulset
... workload_name=${STATEFULSET_NAME}
... namespace=${NAMESPACE}
... categories=@{LOG_PATTERN_CATEGORIES}
... excluded_containers=${EXCLUDED_CONTAINERS}

${log_health_score}= RW.K8sLog.Calculate Log Health Score scan_results=${scan_results}

Expand Down Expand Up @@ -99,12 +101,14 @@ Detect Log Anomalies for StatefulSet `${STATEFULSET_NAME}` in Namespace `${NAMES
... context=${CONTEXT}
... kubeconfig=${kubeconfig}
... log_age=${LOG_AGE}
... excluded_containers=${EXCLUDED_CONTAINERS}

${anomaly_results}= RW.K8sLog.Analyze Log Anomalies
... log_dir=${log_dir}
... workload_type=statefulset
... workload_name=${STATEFULSET_NAME}
... namespace=${NAMESPACE}
... excluded_containers=${EXCLUDED_CONTAINERS}

# Process anomaly issues
${anomaly_issues}= Evaluate $anomaly_results.get('issues', [])
Expand Down Expand Up @@ -970,9 +974,21 @@ Suite Initialization
... pattern=\d+
... example=1
... default=1
${EXCLUDED_CONTAINER_NAMES}= RW.Core.Import User Variable EXCLUDED_CONTAINER_NAMES
... type=string
... description=Comma-separated list of container names to exclude from log analysis (e.g., linkerd-proxy, istio-proxy, vault-agent).
... pattern=.*
... example=linkerd-proxy,istio-proxy,vault-agent
... default=linkerd-proxy,istio-proxy,vault-agent

# Convert comma-separated string to list
# Convert comma-separated strings to lists
@{LOG_PATTERN_CATEGORIES}= Split String ${LOG_PATTERN_CATEGORIES_STR} ,
@{EXCLUDED_CONTAINERS_RAW}= Run Keyword If "${EXCLUDED_CONTAINER_NAMES}" != "" Split String ${EXCLUDED_CONTAINER_NAMES} , ELSE Create List
@{EXCLUDED_CONTAINERS}= Create List
FOR ${container} IN @{EXCLUDED_CONTAINERS_RAW}
${trimmed_container}= Strip String ${container}
Append To List ${EXCLUDED_CONTAINERS} ${trimmed_container}
END

Set Suite Variable ${kubeconfig}
Set Suite Variable ${KUBERNETES_DISTRIBUTION_BINARY}
Expand All @@ -986,6 +1002,8 @@ Suite Initialization
Set Suite Variable ${ANOMALY_THRESHOLD}
Set Suite Variable ${CONTAINER_RESTART_AGE}
Set Suite Variable ${CONTAINER_RESTART_THRESHOLD}
Set Suite Variable ${EXCLUDED_CONTAINER_NAMES}
Set Suite Variable @{EXCLUDED_CONTAINERS}
${env}= Evaluate {"KUBECONFIG":"${kubeconfig.key}","KUBERNETES_DISTRIBUTION_BINARY":"${KUBERNETES_DISTRIBUTION_BINARY}","CONTEXT":"${CONTEXT}","NAMESPACE":"${NAMESPACE}","STATEFULSET_NAME":"${STATEFULSET_NAME}","CONTAINER_RESTART_AGE":"${CONTAINER_RESTART_AGE}","CONTAINER_RESTART_THRESHOLD":"${CONTAINER_RESTART_THRESHOLD}"}
Set Suite Variable ${env}

Expand Down
Loading
Loading