refactoring tests to log io behavior

[betolink]

This PR is to integrate code that logs fsspec network operations and tweak the I/O libraries (hypy, fsspec).
The fsspec-logs notebook should be a stand-alone test when ready (90% there)

[betolink]

I think this PR is ready to be reviewed, still work in progress. h5coro needs upstream changes to work with s3 links that don't require a login. We probably need to create a pip installable package called h5benchmark and then all the bootstrapping code and relative paths should go away.

The main notebooks are the ones with portable in their name, I think once we have subsetting ready we could move on to complete the benchmarking numbers.

[asteiker]

@betolink I received an error in the 3rd codeblock:

results = xarray_original.run(io_params)

ZeroDivisionError: division by zero

[betolink]

I ran it on a fresh clone and didn't get that error, perhaps we do some screen sharing to see what's going on.

[betolink]

oh I see, there is a bug, fixing it now....

[asteiker]

@betolink I received an error in the last plotting block:

for name, group in df.groupby(['tool', 'dataset', 'format']):
    tool, dataset, formated = name
    x = f'{tool}, {dataset}, {formated}'
    y = group['time'].mean()
    ax.bar(f'{tool}, {dataset}, {formated}', group['time'].mean(), label=f'{tool}, {dataset}, {formated}', align='center')
    ax.text(x, y + 0.05, f'{group["time"].mean():.2f}', ha='center', va='bottom', color='black', fontsize=8)

KeyError: 'tool'

[betolink]

this notebook was just prototyping some stuff, the ones that we want to plot the results are the ones named "portable-"

[asteiker]

@betolink We should add a warning to this notebook to explain that it's not functioning due to the anonymous access issue, and/or comment all the code blocks that produce an error.

[betolink]

Good idea, or in the meantime we can use the same granules just from the CryoCloud bucket.

[asteiker]

@betolink

• One small suggestion is to add a short title + description to the top of this notebook to explain its usage.
• I also got a divide by zero error here in the 3rd code block:

results = h5py_original.run(io_params)

[asteiker]

Like the h5py test notebook:

• One small suggestion is to add a short title + description to the top of this notebook to explain its usage.
• I also got a divide by zero error here in the 3rd code block:
  results = h5py_original.run(io_params)

[asteiker]

I wasn't able to run many of the notebooks end to end (tested in CryoCloud). Once those errors are resolved plus suggested updates to include brief title and descriptions for the new notebooks, then it looks good on my end.

Also for our collaborators, do we need any additional documentation for them to utilize the notebooks (i.e. any breaking changes for the other CO formats that others would need to be aware of)?

[JessicaS11]

Are these file names/paths still being used, or could we remove this file since the links were updated in links.json so it's effectively in the git history?

[JessicaS11]

Is this a second storage location (different hub) for the same set of files? Or are the files different?