Watchmem

Watchmem is a C++20 library used to watch Linux and Windows systems for memory shortage. You can use it in your app to get notified when memory ressources run low on your system (or when enough memory becomes available again).

A major goal of Watchmem is to provide a unified interface for doing this across Windows and Linux, which have fundamentally different memory management systems. To this end, the Watchmem library provides several "watchers" (currently two on Linux and two on Windows), which each use a different technique to watch for memory shortage. Each watcher is only enabled (in fact, only compiled) for the OS it supports.

In your application, you just always provide configuration for all provided watchers (and we try to provide sane default values). This way, your application does not need any #ifdef __linux__ … logic. Just configure everything and let watchmem handle the OS selection.

Current Status

This is beta software. It is not yet in use by any major application, thus not very battle-tested. The library does come with some scripts used for testing the implemented techniques (see 'testing' below). This library is a result of me diving into the various ways of detecting memory shortages. As such, please use this library with caution. If you use this library in your application and either find some problems or learn that the library works reliably, please let me know!

General Usage

This is the simplest possible usage of Watchmem:

#include <chrono>
#include <iostream>
#include <thread>

#include "watchmem.h"

int main(int argc, char **argv) {
  watchmem::Settings settings{};
  settings.callback = []() { std::cout << "Memory shortage detected!\n"; };

  watchmem::Watcher watcher(settings);
  watcher.watch(); // This call will immediately return, watching is done in
                   // separate threads

  std::this_thread::sleep_for(std::chrono::seconds{10});

  return 0;
}

It starts all available watchers for your OS with their default settings. When a watcher detects memory shortage, the lambda callback is called. After 10 seconds, the watcher object is destroyed, which also causes all watchers to stop. See examples/simple.cpp for this example.

This is always the basic workflow:

• Create a Settings object, make all the settings you want, but at least configure the Settings::callback.
• Create a Watcher object from your settings.
• Call Watcher::watch(). The call returns, your watchers are active.

If a watcher cannot be set up for some reason, the watch() call will throw a WatchFailed exception. So the watch() call above should probably look like this:

…
  watchmem::Watcher watcher(settings);
  try {
    watcher.watch();
  } catch (const watchmem::WatchFailed &ex) {
    std::cerr << "Could not set up watch: " << ex.what() << "\n";
  }
…

Threads

Watching happens in threads separate from the thread that you called .watch() in. The watch() call returns immediately and your watchers are active until you destroy the Watcher object.

Important: When the callback is called from one of the threads spawned by the Watchmem library. So whatever you do in the callback, it must be thread-safe against all other threads in your application. Note that the callback might even be called concurrently by multiple watchers!

Rate-Limiting the Callback

The various watchers have different behavior regarding what happens if the low-memory state persists. Some watchers only call the callback once the low-memory state starts and will only call the callback again once the low-memory state was left and re-entered. Other watchers will continually (with a certain frequency) call the callback as long as the low-memory state persists. See the documentation of the individual watchers for details.

Because of this, you can rate-limit the frequency with which each watcher may call your callback by setting the Settings::cooldown setting. This is a time that the watcher will be 'inactive' after the callback was called once.

Note that this is a per-watcher value. So if you set this to e.g. one second, and you have two watchers active, your callback may be called twice within one second (but not more often).

Examples

The following examples are provided:

• examples/simple.cpp: The most basic version of using the watchers.
• examples/all_settings.cpp: An example showing all possible settings.

Watchers

Linux: Pressure Stall Interface (PSI) Watcher

The Linux PSI watcher watches for "pressure" events in the Linux cgroup2 system. It uses the notification system provided by the kernel on the /sys/fs/cgroup/<cgroup>/memory.pressure file. (See for example here for a quick intro into the cgroup2 PSI system.)

This allows this watcher to be (more or less) immediately notified about low memory conditions without having to poll.

Linux PSI Watcher Configuration

disable : Set to true to disable the watcher even if it is available

kind : Whether to watch for "some" pressure or for "full" pressure. See cgroup2 PSI documentation for details

threshuS : Amount of time within the PSI window (windowuS) that the limit must be reached for the watcher to trigger. The watcher will fire if some/all applications are stalled on memory operations for this many microseconds during any windowuS microsecond sliding interval. Default: 50000 (50ms)

windowuS : Window length for the PSI watcher. The watcher will fire if some/all applications are stalled on memory operations for threshuS microseconds during any microsecond sliding interval of this length. Default: 2000000 (2 seconds)

path : If set, use this path for PSI monitoring instead of auto-detection. Useful for monitoring a specific cgroup's memory.pressure file

Linux: High Event Watcher

The Linux 'High Event' watcher also uses the cgroup2 interface, but watches the event counters at /sys/fs/cgroup/<cgroup>/memory.events. It can triggers as soon as one of the event counters (by default the 'high' event counter) in that file increases. The 'high' counter counts how many times the 'high' limit of the respective cgroup was exceeded. Thus, by setting the corresponding 'high' limit, you can control when this watcher should trigger. (See this documentation for further info about the 'high' limit.)

Linux High Event Watcher Configuration

disable : Set to true to disable the watcher even if it is available

path : If set, use this path for event monitoring instead of auto-detection. Useful for monitoring a specific cgroup's memory.events file.

Windows: Memory Resource Notification Watcher

This watcher watches for the 'memory resource notification' sent by Windows in low memory events (or high memory events, see configuration). See this Microsoft documentation for details.

Note that there is an (undocumented by Microsoft) registry key which can control when the 'low memory resource notification' should be triggered. It is mentioned on this MSSQL wiki article. Since this registry key is not officially documented by Microsoft, it may change in future versions.

You must add a DWORD value named LowMemoryThreshold under the key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SessionManager\MemoryManagement. The value you set is the number of megabytes at which the low memory notification should trigger.

Windows Memory Resource Notification Watcher Configuration

disable : Set to true to disable the watcher even if it is available

kind : Whether to watch for low memory (LOW), or "enough memory available" (HIGH)

Windows: Polling Memory Watcher

This watcher just simply polls the Windows OS for the amount of current "free" memory. It allows to trigger on values of MEMORYSTATUSEX::ullAvailPhys, which is "the amount of physical memory that can be immediately reused without having to write its contents to disk first. It is the sum of the size of the standby, free, and zero lists." according to the documentation.

Additionally, the watcher can watch for values of MEMORYSTATUSEX::dwMemoryLoad, which is "a number between 0 and 100 that specifies the approximate percentage of physical memory that is in use".

This watcher does manual polling of the respective values, thus you can configure a polling interval.

Windows Polling Memory Watcher Configuration

disable : Set to true to disable the watcher even if it is available

kind : Whether to trigger when available memory drops below the specified limit (LOW), or grows above the specified limit (HIGH). Note: if you set this to HIGH, you must change the thresholdPercentage value.

thresholdBytes : Number of bytes of available physical memory at which the watcher should trigger. In LOW mode, the watcher triggers when available memory drops below this amount. In HIGH mode, it triggers when available memory grows above this amount. Default: 500 MB

thresholdPercentage : Percentage of used physical memory at which the watcher should trigger. In LOW mode, the watcher triggers when the percentage of used memory is above this value. In HIGH mode, the trigger fires when the percentage of used physical memory is below this value. Default: 101 (effectively disabled in LOW mode)

pollFrequency : Frequency in which to poll for memory values. Default: 250 milliseconds

keepFiring : If set to true, the watcher will fire during every poll cycle as long as the condition is true. If set to false, the trigger will only re-fire once the condition was false for at least one polling cycle. Default: false

Building

You should need nothing but a C++20 compatible compiler. To build the library, use CMake:

cmake -B build -S .
cmake --build build

Alternatively, after running the initial CMake configuration, you can use make:

cmake -B build -S .
cd build && make

This will build everything, including examples and test executables.

Testing

Afer you built everything (see above, you must at least build the watch_cli and exhaust_mem executables), you can run tests. Since this library basically is a wrapper around OS features, "unit testing" is hardly possible. Instead, we have two sets of scripts, one for Windows and one for Linux, which create test scenarios.

Both sets of scripts spawn containers and run the tests in the containers. The basic idea is: An 'outer' script spins up a container, setting a restrictive memory limit for that container. The other script runs both exhaust_mem and watch_cli inside the container. The watch_cli executable is just a thin executable configuring and starting the watchers, and reporting to stdout when memory shortage is detected. The exhaust_mem executable consumes memory with the aim of triggering the watchers inside watch_cli. The inner script then checks that watch_cli detects memory shortage when it should.

Note on Linux tests: You need a systemd-based system, since the test runner script uses systemd-run to spawn the container.

Note on Windows tests: For the tests to run, you need Docker Desktop in Windows container mode. By default, Docker Desktop runs in "Linux container mode". Also the run_docker_test.ps1 script will try to build the executables using MSVC, i.e., VS developer tools must be available.

Untested Notice: The Windows 'memory resource notification' watcher is currently not automatically tested. This is because it is not possible to modify the relevant limit (via special registry key, see above) inside a Docker container.

AI Disclosure

The main code of the library (everything inside include/ and src/) was not created by AI in any way.

Generative AI tools were used to create the test scripts and executables (code in test/), as well as performing code "review". All code generated by AI tools was reviewed by a human.