KDUMP in OpenShift CoreOS Baremetal Nodes

This repository is focused on setting up kdump on OpenShift nodes. We created it to address situations where worker nodes freeze without warning, preventing us from troubleshooting the underlying cause. With kdump we can capture detailed crash dumps, providing the insights needed to diagnose and prevent these issues from recurring.

• Review Summary

📖 Table of Content

• KDUMP Introduction and Key Concepts

KDUMP Configuration, Installation and Troubleshooting

• KDUMP Manual Configuration

• KDUMP Machineconfig Configuration

• KDUMP Examples

• KDUMP Troubleshooting

• Configure Serial Console for KDUMP Issues Troubleshooting

Crash Tool Configuration and Usage

• Using Crash Tool RPM to Analyze a vmcore

• Using Crash Tool Custom Container to Analyze a vmcore

• Crash Tool Guide

🔗 Documentation and Articles

Official Docs

• The importance of configuring kernel dumps <-- Recommended

• Configuring kdump on the command line (RHEL8 CoreOS 4.12 and lower)

• Configuring kdump on the command line (RHEL9 CoreOS 4.13 and upper)

• Supported kdump configurations and targets (RHEL8 CoreOS 4.12 and lower)

• Supported kdump configurations and targets (RHEL9 CoreOS 4.13 and upper)

• Butane file Configuration Specifications

Unofficial Blogs

• A vmcore for your system may be smaller than you think! <-- Recommended

• Untestand the Machine Config Pool (OpenShift Container Platform 4.x)

• White Paper: Crash Utility by David Anderson <-- Recommended

🔗 Issues and Solutions

Configuration

• Common kdump Configuration Mistakes <-- Recommended

• How do I configure kdump for use with the RHEL 6, 7, 8 High Availability Add-On?

• Setting up kdump in Red Hat Openshift Container Platform and Red Hat CoreOS

• What is early kdump support and how do I configure it?

Troubleshoot

• How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux

• How to extract/unpack/uncompress the contents of the initramfs boot image on RHEL 7,8,9 ?

SSH Target

• Missing Logs in /var/crash Post Kdump Setup in RHOCP4

• How to setup kdump to dump a vmcore on ssh location in Red Hat Openshift Container Platform nodes

Hardware and Network

• kdump fails to generate vmcore with SysRq on servers installed with LEGACY BIOS and vga controller...

• Kdump failes with "kdump: get_host_ip exited with non-zero status!"

• kdump failure when network requires multiple nics to reach dump target

🔗 Manual Pages

NOTE: The links provided below are for reference only, Always refer to the manual pages specific to your CoreOS version to ensure compatibility!

• kdump(5) - Configuration of kdump

• kdump.conf(5) - configuration file for kdump kernel

• kdumpctl(8) - control interface for kdump

• kexec(8) - Linux man page

• dracut - low-level tool for generating an initramfs/initrd image

• dracut.cmdline(7) - dracut kernel command line options

• dracut-module-setup.sh - Github Repository

• makedumpfile(8) - make a small dumpfile of kdump

---