`emb-embed-explore` Image Folder Import Error

[NetZissou]

when I select a folder with three images to embed, this pops up (full traceback in slack) and it does not recognize the embeddings (clustering errors out stating that embedding must be performed first)

Error messages shared through slack.

[NetZissou]

Investigation shows that the crash is caused due to source & OS conflicts.

The ImageFolderDataset contains lambda functions which can't be serialized by Python during runtime. However, the multiprocessing method spawn requires serialization which caused the crash. More details please see issue.

Additionally, the Streamlit application behavior should be fixed as it notify users embeddings complete whereas the images were failed to be preprocessed in the first place.

We might consider create a dedicated utility function for image preprocessing for emb-explorer and remove hpc-inference from the dependency list.

[NetZissou]

^^^ also a blocker for #27 as hpc-inference is a git+https:// dependency, and PyPI rejects direct-URL deps.

A ImageFolderDataset module that works for single machine, optimize for image preprocessing on multi-cores CPUs makes sense for the development cycle of this pkg. We shed transitive weight we never use for emb-explorer: matplotlib, nvidia-ml-py, psutil, pyyaml, distributed/rank code.