Skip to content

Doesn't seem to work with h5py #82

@CMCDragonkai

Description

@CMCDragonkai

I have a file I'm opening up with fs-s3fs.

This file works since I've downloaded and written to the disk.

But if I try to open up the stream in h5py, it just fails:

with fs.open_fs('s3://...').open('...', mode="rb") as weights_file:
    with h5py.File(weights_file, mode="r") as weights_hdf5:
        print(weights_hdf5)

The error is:

Traceback (most recent call last):
  File "/nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cmcdragonkai/Projects/formbay/image-classifier/image_classifier_server/services/infer.py", line 53, in __call__
    with h5py.File(weights_file, mode="r") as weights_hdf5:
  File "/nix/store/x87nsr4hhvns002c67yg43nbaknd71pn-python3.7-h5py-2.10.0/lib/python3.7/site-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/nix/store/x87nsr4hhvns002c67yg43nbaknd71pn-python3.7-h5py-2.10.0/lib/python3.7/site-packages/h5py/_hl/files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 88, in h5py.h5f.open
  File "h5py/defs.pyx", line 666, in h5py.defs.H5Fopen
  File "h5py/h5fd.pyx", line 158, in h5py.h5fd.H5FD_fileobj_read
  File "/nix/store/jws4ma2692dadgbv52l2aa3s8l9mkp8s-python3.7-fs-2.4.11/lib/python3.7/site-packages/fs/iotools.py", line 116, in readinto
    return self._f.readinto(b)
  File "/nix/store/ldqzyr3f6l8rvxq5qprfak820kh7dm7x-python3.7-fs-s3fs-1.1.1/lib/python3.7/site-packages/fs_s3fs/_s3fs.py", line 154, in readinto
    return self._f.readinto()
TypeError: readinto() takes exactly one argument (0 given)

There's something wrong with the readinto() method that h5py appears to be using.

But if I just do weights_file.read() all is fine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions