sys_mmap: eager zero-fill on MAP_ANONYMOUS makes sparse mappings ~5x slower than Linux

[Max042004]

Summary

sys_mmap zeroes the entire region synchronously for every MAP_ANONYMOUS mapping (src/syscall/mem.c:2439):

/* Zero the mapped region */
memset((uint8_t *) g->host_base + result_off, 0, length);

This is functionally correct, but pays the full zero-fill cost up front. Linux backs new anonymous pages with the shared zero page and zeroes lazily on first touch, so workloads that map a region and touch only a fraction of it (e.g. CPython pymalloc 256 KiB arenas) pay for pages they never use.

Measurements (main @ e6b3455, Apple M-series)

Static aarch64 binary looping mmap(256 KiB, MAP_ANON|MAP_PRIVATE) → touch → munmap, 5000 iterations:

Workload	elfuse	OrbStack (Linux)
1-byte touch	7,041 ns/cycle	1,392 ns/cycle
touch every 4 KiB page (64 pages)	6,934 ns/cycle	8,904 ns/cycle

On elfuse the 1-byte-touch cost equals the full-touch cost — the whole region is paid for at mmap time. On Linux the cost scales with pages actually touched.

Downstream effect: CPython for _ in range(5M): pass runs ~11.5 ns/iter on elfuse vs ~7.3 on OrbStack with the same python3.12 binary; the gap mostly disappears with PYTHONMALLOC=malloc (bypassing pymalloc's mmap-backed arenas), confirming it is mmap-bound rather than interpreter-bound.

Notes toward a fix

• The lazy machinery already exists: guest_materialize_lazy() (src/core/guest.c:3265) plus the translation-fault path in src/syscall/proc.c already perform fault-time PTE creation, per-2MiB-block zeroing, and TLBI — but only for regions carrying the noreserve flag. Extending this to all anonymous mappings is the natural fix. Open questions: stale-PTE invalidation cost on address reuse (currently guest_invalidate_ptes per mmap on the NORESERVE path), and concurrent faults on the same range from multiple vCPUs.
• Host madvise(MADV_DONTNEED) is not a shortcut: on macOS it behaves like MADV_FREE, and replacing the memset with it regressed the benchmark to 14–30 µs/cycle in a local experiment.
• The memset itself is only ~25% of the gap (measured by unsoundly disabling it: ~5.6 µs/cycle remains). The rest is the per-syscall region-tracking / guest_extend_page_tables / guest_update_perms / TLBI path — lazy materialization also skips most of that work for pages never touched.

Reproduction

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <time.h>

int main(int argc, char **argv) {
    int N = (argc > 1) ? atoi(argv[1]) : 5000;
    size_t SZ = 256 * 1024;
    struct timespec t0, t1;
    clock_gettime(CLOCK_MONOTONIC, &t0);
    for (int i = 0; i < N; i++) {
        char *p = mmap(NULL, SZ, PROT_READ | PROT_WRITE,
                       MAP_ANON | MAP_PRIVATE, -1, 0);
        p[0] = 1; /* or: for (size_t j = 0; j < SZ; j += 4096) p[j] = 1; */
        munmap(p, SZ);
    }
    clock_gettime(CLOCK_MONOTONIC, &t1);
    double ns = (t1.tv_sec - t0.tv_sec) * 1e9 + (t1.tv_nsec - t0.tv_nsec);
    printf("%.1f ns/cycle\n", ns / N);
    return 0;
}

aarch64-linux-musl-gcc -O2 -static -o bench_mmap bench_mmap.c
build/elfuse bench_mmap 5000