use for own dataset

[tienduchoang]

hello everyone, can you show me any guide to run augmentation on my own dataset.

[arcelien]

You should update the dataloader, the Dataset class in pba/data_utils.py to load your dataset. It should work for any image size.

You can see load_test() in pba/data_utils.py or scripts/test_search.sh for an example.

[tienduchoang]

Hello @arcelien Thank for your respond.
Because your input data is resized in fix size. like below:

train_data = train_data.reshape(-1, 3, 32, 32)

How to work with a folder that contain different size of images.

[sayakpaul]

I was looking for this and I am pretty sure many people will be, An example notebook might be extremely helpful in that case. Maybe an end-to-end demo covering augmenting your own dataset using PBA and train a model using them. It will really be immensely helpful.

I ran the pba.ipynb demo notebook and it went pretty smoothly.

[arcelien]

Note that the line train_data = train_data.reshape(-1, 3, 32, 32) is in the load_cifar() function which loads CIFAR data. You should define a new function that loads your dataset instead with the correct size.

See load_test() for an example of loading 224 x 224 data. The image size is passed around in a self.image_size variable in the dataloader not constrained to any fixed size.

I will look into such a notebook, but may not have the bandwidth for that very soon.

[tienduchoang]

@arcelien @sayakpaul thank for your help.

[tienduchoang]

Hi @arcelien, i have word image dataset, i want to use your code to augment this dataset for my OCR system.
Each images have width in range(26 - 150), height in range(26 - 40).
Total number of classes is 1, because dataset just contain word images.

I don't know how to get start with this dataset. Do you have any guide to help me augment this dataset?

My sample dataset follow this link:

https://drive.google.com/drive/folders/1pUiWY6PZwvNWCSX46vvD3g9QCfKH7yTk?usp=sharin

Thank you!

[zengruizhao]

Hi @arcelien , can we only load all data in data_loader? But it's not friendly to large data sets. And I want to know whether we can build a data generator. Looking forward your reply. Thanks.

[arcelien]

The way the current dataloader is implemented only supports loading all data at once, but it should not be too difficult to adapt this to generate data on demand or use a new dataloader which does so.

To adapt this, I'd imagine we could use something like the torchvision.datasets.SVHN data generator to load data in the next_batch() function and apply all the augmentation/transformations then instead of in the initialization.

[441711335]

Hi @arcelien , how can we get " MEAN" and "STD" in files of augmentation_transforms.py while using PBT in my own database.

[arcelien]

If you're using the default dataloader you can pass the flag --recompute_dset_stats while training. Otherwise, you can just compute the per-channel mean and standard deviation of your dataset (i.e. call mean and std on your data tensors).

[441711335]

@arcelien thank you for your help!

[jaspreet-sambee]

The way the current dataloader is implemented only supports loading all data at once, but it should not be too difficult to adapt this to generate data on demand or use a new dataloader which does so.

To adapt this, I'd imagine we could use something like the torchvision.datasets.SVHN data generator to load data in the next_batch() function and apply all the augmentation/transformations then instead of in the initialization.

Hi @arcelien , Hvae a doubt on this, probably did not understand it clearly.

1. Even if we load the data in the 'next_batch()' function, aren't we still loading the whole dataset into memory? Or did you mean to create a kind of data iterator which loads only a batch inside the next_batch function?
2. I am currently using a dataset with imagenet like images of size 224x224x3 with a resnet50 model. But i see it takes a lot of GPU memory for each of the workers and hence other workers have to wait for the first few workers to finish. I am not clear how so much GPU memory (roughly 13GB) is being used per worker. Am i missing something in the data_loader side?

Thanks!

[arcelien]

1. I'm not certain, but I don't think pytorch's dataloader loads the whole dataset into memory.

2. Perhaps you can look at the batch size? Or you can set the fractional gpu size per worker to something smaller.

[jaspreet-sambee]

2. Perhaps you can look at the batch size? Or you can set the fractional gpu size per worker to something smaller.

My batch size is only 16. And if i set the GPU size anything lesser than 0.6 (~13GB), it runs out of GPU memory.
I see that all the workers call the load_data from RAY and hence each of them loads all the data. And in the next batch function, each worker is sampling a batch. And i hope each worker is not putting all the data on the GPU or is my understanding wrong here?

[arcelien]

It could be that with ImageNet, you just need to use much smaller batches or distribute over multiple GPUs or run fewer trials concurrently. For reference, with a (32x32x3) input like CIFAR/SVHN, the size per trial is ~2GB with a network like ResNet 50.