Skip to content

Monitor and plot RSS memory and CPU usage during qlever index#277

Open
tanmay-9 wants to merge 26 commits into
qlever-dev:mainfrom
tanmay-9:compute-index-mem-usage
Open

Monitor and plot RSS memory and CPU usage during qlever index#277
tanmay-9 wants to merge 26 commits into
qlever-dev:mainfrom
tanmay-9:compute-index-mem-usage

Conversation

@tanmay-9
Copy link
Copy Markdown
Collaborator

@tanmay-9 tanmay-9 commented Apr 6, 2026

So far, the qlever index command gave no insight into how much memory an index build actually needs or which index phase is responsible for the peak.

With this change, every index build records RSS memory and CPU usage over time, writes <name>.usage-log.tsv, and renders <name>.usage-log.png once the index build finishes. The plot shades each index build phase (parsing, vocabulary merge, conversion, each permutation, and the text index) as a separate band and annotates the memory peak, so resource usage can be attributed to a specific phase. For comparison across runs and settings, the plot is captioned with the git hash of the index binary, the STXXL_MEMORY setting, and the batch size. This works whether the index is built natively or in a container (docker / podman).

The sampling rate can be set with --resource-monitor-interval and the plot density on long builds with --resource monitor-plot-max-points (the sampling itself is unaffected, only how many points are drawn). There is also a --replot-resource-usage option that re-renders the plot from an existing <name>.usage-log.tsv without re-running the index build, which is useful for tweaking plot settings.

@tanmay-9 tanmay-9 changed the title Compute the physical memory usage used by the qlever index command Compute the physical memory used by the qlever index command Apr 7, 2026
@tanmay-9 tanmay-9 changed the title Compute the physical memory used by the qlever index command Track memory usage during qlever index Apr 8, 2026
tanmay-9 and others added 19 commits April 9, 2026 10:06
…u cores used and add downsampling (max_points=500) for plot
@tanmay-9 tanmay-9 changed the title Track memory usage during qlever index Monitor and plot RSS memory and CPU usage during qlever index May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant