Added AzureBlobFileSystem support for StructuredDatasets

[MorpheusXAUT]

TL;DR

This PR adds support for storing StructuredDatasets using AzureBlobFileSystem (abfs).

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

As discussed before, we've added support for abfs (using adlfs/stow under the hood) for StructuredDatasets by adding it to the registered protocols for transformers.
I've also noticed one file (plugins/flytekit-spark/flytekitplugins/spark/sd_transformers.py) which still used string constants and didn't support GCS either. As we're currently not using Spark anywhere, I wasn't able to verify this change though, so I can revert those lines if you'd prefer.

Tracking Issue

flyteorg/flyte#2709

Follow-up issue

NA

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov]

Codecov Report

Merging #1109 (f5bdbb3) into master (c6e7237) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1109   +/-   ##
=======================================
  Coverage   86.95%   86.95%           
=======================================
  Files         276      276           
  Lines       25492    25493    +1     
  Branches     2865     2865           
=======================================
+ Hits        22167    22168    +1     
  Misses       2847     2847           
  Partials      478      478           

Impacted Files	Coverage Δ
flytekit/types/structured/basic_dfs.py	100.00% <100.00%> (ø)
flytekit/types/structured/structured_dataset.py	93.06% <100.00%> (+0.02%)	⬆️
flytekit/deck/deck.py	82.97% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c6e7237...f5bdbb3. Read the comment docs.

[wild-endeavor]

hey @MorpheusXAUT do you also have the data persistence plugin for abfs? is that what you meant by you previously added support for it under the hood?

[MorpheusXAUT]

hey @MorpheusXAUT do you also have the data persistence plugin for abfs? is that what you meant by you previously added support for it under the hood?

@wild-endeavor oh, good point 🤔 I believe that still hasn't been cleaned up & contributed back to this repo, sorry about that. 😕
I'll try to get that sorted first thing tomorrow. Should we add that to the same PR or create a new one?

[wild-endeavor]

this pr is fine. thanks!

[MorpheusXAUT]

@wild-endeavor Looking at it again, I was actually mistaken last night.

We don't use an extra (custom-written) datapersistence plugin for abfs, but instead make use of fsspec and its adlfs (albeit a fork at the moment, since we're waiting for a PR to be merged).

I've added adlfs to flytekit-data-fsspec plugin's extras_require map, the version constraint matches the version we're using/have tested it with.
I believe that's everything required on flytekit's/the data persistance plugin's side as the actual import for adlfs was configured in our projects.

EDIT: not quite sure why that one check failed, looks unrelated to me?

[wild-endeavor]

rerunning the failed onnx job. not entirely sure what's happening there.

[eapolinario]

Is it time to turn this into an enum, @wild-endeavor ?

[welcome]

Congrats on merging your first pull request! 🎉