Report number of batches in a spectrum dataset

[bittremieux]

Useful for timing estimates in the progress bar.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.50%. Comparing base (486221c) to head (e188d80).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #60   +/-   ##
=======================================
  Coverage   97.49%   97.50%           
=======================================
  Files          24       24           
  Lines         957      960    +3     
=======================================
+ Hits          933      936    +3     
  Misses         24       24           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bittremieux]

Hmm, using this a bit more, it actually gives the following warning:

WARNING: UserWarning: Your IterableDataset has __len__ defined. In combination with multi-process data loading (when num_workers > 1), __len__ could be inaccurate if each worker is not configured independently to avoid having duplicate data.

So maybe not an ideal tweak in the end. Is the SpectrumDataset compatible with getting accurate timing estimates from the PyTorch Lightning progress bar (for which the number of batches is needed)?

[bittremieux]

Addition: because SpectrumDataset is an IterableDataset, it also doesn't support shuffling. We might want to do this during training.