Unless anyone has any objections I think we should submit this package to conda-forge so that the conda related maintenance (recipe) can be done there. We could then remove conda specific stuff from the CI here. This should clean up the build system.
It'll require #35 to be completed.
