Skip to content

feat: update chapter 03 to use new LAIF container#101

Open
marlon-tobaben wants to merge 29 commits into
mainfrom
update_chapter_3
Open

feat: update chapter 03 to use new LAIF container#101
marlon-tobaben wants to merge 29 commits into
mainfrom
update_chapter_3

Conversation

@marlon-tobaben
Copy link
Copy Markdown
Member

@marlon-tobaben marlon-tobaben commented May 12, 2026

This PR closes #79 , only open issue is that HDF5 is terribly slow with one worker, this must be a bug. This was already in older version of the repository. We should maybe check this. New ideas based on PR: #103 and #104.

General changes

  • Remove disclaimer that 03 hasn't been ported from README.MDs
  • Update scripts to use relative paths and not specific project folders
  • Removed env vars: export MPICH_MPIIO_STATS=1 and export MPICH_MEMORY_REPORT=1

Simple benchmarks (using tiny-imagenet-200)

  • Update to new LUMI AIF container
  • Use squashFS venv for LMDB (and add script to create squashFS venv)
  • HDF5 and squashFS use only the container
  • Add instructions to README.MD for all steps
  • Make post-processing script more robust (used to expect time spent on line 4 of log, but this can easily break)

Training benchmarks with vision transformer (using tiny-imagenet-200)

  • removed profiling as it was not documented at all
  • added a version of the script in 01 that allows to compare runtime between the three file formats
  • added post-processing script for this and instructions to README

Benchmarks using larger version of ImageNet (ImageNet-1k)

@marlon-tobaben marlon-tobaben self-assigned this May 12, 2026
@marlon-tobaben marlon-tobaben changed the title feat: update chapter 03 to new LAIF container feat: update chapter 03 to use new LAIF container May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port session 3 (data formats) to LUMI AIF containers

1 participant