Skip to content

Allow partial download of imagesets#52

Open
AbhiDhir wants to merge 3 commits intomasterfrom
partial-download
Open

Allow partial download of imagesets#52
AbhiDhir wants to merge 3 commits intomasterfrom
partial-download

Conversation

@AbhiDhir
Copy link
Copy Markdown
Contributor

Overview

  • Allows the option for partial download of imagesets rather than requiring full download \
  • download_full_imagesets is a new optional parameter in config
    • if set to true, behaves how it used to (all imagesets are fully downloaded into cache)
    • if set to false or not present, setSizeFilter must be set to values indicating how many images from each imageset are required
      • This is currently up to the plugin to check for and can be seen in the plugins repo

Notes

  • The current implementation for asynchronous download may be done incorrectly or not as efficiently as possible,
    • I will work more on this by doing time testing
  • This pull request goes along with the request in the ravenml-dataset-plugins repo

@AbhiDhir AbhiDhir requested a review from jacobDeutsch10 June 22, 2021 21:27
@AbhiDhir AbhiDhir self-assigned this Jun 22, 2021
@jacobDeutsch10
Copy link
Copy Markdown
Contributor

Just tested by making a small dataset and it worked for the most part. The only issue i saw was that it if an image-id doesn't have an associated file for a given prefix/suffix pair it will download an empty file for that image-id and prefix/suffix pair.

in the tfrecord plugin the associated_files are:
associated_files = [ ('meta_', '.json'), ('image_', '.png'), ('image_', '.jpg'), ('image_', '.jpeg'), ('bboxLabels_', '.csv') ]
but since the imageset i used only contains files with for ('meta_', '.json') and ('image_', '.png'). we get empty files for each of the remaining pairs.

I think this could cause some issues if this list was ordered differently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants