Follow-up (Installation October 2024)

[paraynaud]

Hello, here is a small update on what I've been able to reproduce and the issues I faced so far.
Disclaimer: I opened an issue because I didn't find the "Discussion" part of the repository.

Accomplishments

I am able to execute all four scripts: training.py, inference.py, evaluation.py, aggregate.py.
To do so, I needed to make some adjustments in the scripts, see #11, and I had to truncate a bit the database during the training.

Issues

• The installations of the dependencies was a bit tricky:
  • Installing Openslide on Windows is cumbersome.
    At the end, I got no other choice than adding

  OPENSLIDE_PATH = "c:\\path-to-openslide-bins\\openslide-bin-4.0.0.6-windows-x64\\bin\\"
  import os
  with os.add_dll_directory(OPENSLIDE_PATH):
      import openslide

  to my scripts (after downloading the binaries of Openslide).
  • pip install -r requirements.txt didn't work for me.
    I installed the libraries one by one, but I got errors with the versions informed in the current requirements.txt.
    Some comments that could be useful for future installations:
    • The version of pandas limits the version of python to 3.8
    • torchvision limits the torch version used (the versions used are not up-to-date)
    • Through my experiments, I ended up using torch==1.8.1 and torchvision==0.9.1 (GPU versions).
      The installation of Torch previous versions are available here https://pytorch.org/get-started/previous-versions/, and I used:
      • pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
      • this command line indicated the use of CUDA 11.1
      • If the installation is done properly, then the GPU should be activated automatically, check with torch.cuda.is_available()==True
    • I installed wandb==0.18.3:
      • I make sure to connect to wandb with wandb.login(key="your-40-characters-keys") at the beginning of the training, because I got issues with the terminal connection
    • Make sure about the numpy version used, pip will likely try to install newer versions.

Questions

Can I ask the specifications of the hardware you used to obtain your results ?
In my case: Windows 11, 16GB of RAM with an Nvidia RTX4090 Laptop ; I had to truncate the database I used for making training run properly, i.e. train_files[0:--]; valid_files[0:-].
Otherwise, I got the error:

  ...
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\torch\utils\data\dataloader.py", line 517, in __next__
    data = self._next_data()
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\torch\utils\data\dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\dev\Desktop\Vega\code\MIDOGpp\slide\data_loader.py", line 56, in __getitem__
    x = self.x.get(idxs, xmin, ymin)
  File "C:\Users\dev\Desktop\Vega\code\MIDOGpp\slide\data_loader.py", line 111, in get
    res = self.open(fn, x, y)
  File "C:\Users\dev\Desktop\Vega\code\MIDOGpp\slide\data_loader.py", line 119, in open
    return Image(pil2tensor(fn.get_patch(x, y) / 255., np.float32))
  File "C:\Users\dev\Desktop\Vega\code\MIDOGpp\slide\data_loader.py", line 25, in get_patch
    return np.array(self.slide.read_region(location=(int(x * self.down_factor),int(y * self.down_factor)),
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\openslide\__init__.py", line 374, in read_region
    crop = self._image.crop(image_topleft +
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\PIL\Image.py", line 1301, in crop
    self.load()
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\PIL\TiffImagePlugin.py", line 1238, in load
    return super().load()
  File "C:\Users\dev\anaconda3\envs\MIDOG2\lib\site-packages\PIL\ImageFile.py", line 229, in load
    self.im = Image.core.map_buffer(
  ValueError: buffer is not large enough

Did you encounter this one ?

In any case, fell free to let me know if I made some kind of mistakes, or any advice !

[maubreville]

As far as I know we used a machine with 64GB of RAM and a consumer graphics card with around 8 GB of VRAM, but I am not sure. @FraukeWilm Can you comment on this?

I am also not sure why truncation of the dataset would be necessary.

BTW: I opened the discussions tab.

wrt the versions:

• To the best of my knowledge all versions should be specified in the requirements.txt file.

[paraynaud]

Hello,

I had the time to look into this further, and I found out that the 321-th slide and its annotations are causing issue when they are exploited during the training.
I deleted the info about this slide in the datasets_xvalidation.csv file, and now I'm able to run the training with all the slides !
I'm not sure about the reason behind the error, I will dig it further when I have time.

PS: I tested Tifffile as a replacement of Openslide, and I got a similar error only for the 321-th slide as well