Libraries mismatch : Python / requirements versions ?

[raphaeljolivet]

Hi,

I am trying to run it locally, but I seem to get some version mismatch of numpy vs skimage.
I get this, when testing classification :

File "DeepSolar/.venv3/lib/python3.5/site-packages/skimage/util/arraycrop.py", line 8, in
from numpy.lib.arraypad import _validate_lengths
ImportError: cannot import name '_validate_lengths'

requirements.txt has floating versions (and also was missing pandas)
Could you please provide the versions of each dependency ?
Also, which version of python are you using ? 2.7 or 3.5 ? I guess it's 3.5 ?

Thanks in advance for your help.

[raphaeljolivet]

Ok, so finally, I am able to have it work.
In summary :

• I need to use python2.7
• I had to downgrade to older version of numpy (1.15) because Skimage is incompatible with last version
• I had to download inception from the source, because installing it from pip does not include inception_model. I just copied it in the project folder, added a couple of init.py and added . to the PYTHONPATH

For information here is a frozen requirement.txt :

numpy==1.15.4
tensorflow==1.12.0
scikit-image==0.14.1
matplotlib==2.2.3
pandas==0.23.4

[raphaeljolivet]

Please update requirement.txt and README accordingly.

[CoolSciGuy]

Hi @raphaeljolivet, were you able to download training and eval sets?

[raphaeljolivet]

Yes I did.

…

-------- Message d'origine -------- On 8 févr. 2019 15:37, CoolSciGuy a écrit :

Hi ***@***.***(https://github.com/raphaeljolivet), were you able to download training and eval sets? — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#7 (comment)), or [mute the thread](https://github.com/notifications/unsubscribe-auth/ABJVwGQ4cpzy_JqKEpdaDcmdnMyFFLLeks5vLYu-gaJpZM4aEEVd).

[CoolSciGuy]

Yes I did.
…
-------- Message d'origine -------- On 8 févr. 2019 15:37, CoolSciGuy a écrit :
Hi @.***(https://github.com/raphaeljolivet), were you able to download training and eval sets? — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#7 (comment)), or mute the thread.

Thanks for responding, did you follow instructions as suggested by developer? I am having no luck.

[sassej]

I am getting this error:
$ python train_classification.py --fine_tune=True
Training set built. Size: 366467
Traceback (most recent call last):
File "train_classification.py", line 262, in
train()
File "train_classification.py", line 117, in train
scope=None)
File "/Users/sassej/DeepSolar/inception_model.py", line 84, in inference
with slim.arg_scope([slim.ops.conv2d, slim.ops.fc], weight_decay=0.00004):
AttributeError: 'module' object has no attribute 'arg_scope'

[raphaeljolivet]

Hi,

Look at my fork here:
https://github.com/raphaeljolivet/DeepSolar

I have been able to run the classification script, by integrating directly the inception library and with the dependencies listed in requirements.txt

Please note that I have simplified the usage of test_classification here, it does not require the generate_data_list.py anymore. Please refer to the header of test_classificiation.py for usage.

[namgivu]

hi @raphaeljolivet

I tried your fork and cannot get install -r requirements.txt working
I'm using python 2.7.18 virtualenv thru pyenv

namgivu@localhost:~/code/b/deepsonar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python -m pip install -r requirements.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting numpy==1.15.4
  Using cached numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.whl (13.8 MB)
Collecting tensorflow==1.12.0
  Using cached tensorflow-1.12.0-cp27-cp27mu-manylinux1_x86_64.whl (83.1 MB)
Collecting scikit-image==0.14.1
  Using cached scikit_image-0.14.1-cp27-cp27mu-manylinux1_x86_64.whl (25.4 MB)
Collecting matplotlib==2.2.3
  Using cached matplotlib-2.2.3-cp27-cp27mu-manylinux1_x86_64.whl (12.6 MB)
Collecting pandas==0.23.4
  Using cached pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl (8.9 MB)
Collecting fiona
  Downloading Fiona-1.8.20.tar.gz (1.3 MB)
     |████████████████████████████████| 1.3 MB 1.6 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-okL5YE/fiona/setup.py'"'"'; __file__='"'"'/tmp/pip-install-okL5YE/fiona/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-Codbjh
         cwd: /tmp/pip-install-okL5YE/fiona/
    Complete output (2 lines):
    Failed to get options via gdal-config: [Errno 2] No such file or directory
    A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

---

removing two lines in the requirements.txt below to get it thru

fiona
opencv-python

numpy==1.15.4
tensorflow==1.12.0
scikit-image==0.14.1
matplotlib==2.2.3
pandas==0.23.4
# fiona
# opencv-python
shapely
pyproj
progressbar2==3.39.2

[namgivu]

I have succeeded running the train classification with your fork version - great thanks for the fix for inception

./.venv/bin/python train_classification.py --fine_tune=False

Though I got the error

ValueError: Dimension 3 in both shapes must be equal, but are 96 and 32. Shapes are [32,35,35,96] and [32,35,35,32].
From merging shape 2 with other shapes. for 'inception_v3/mixed_35x35x256a/concat/concat_dim' (op: 'Pack') with input shapes: [32,35,35,64], [32,35,35,64], [32,35,35,96], [32,35,35,32].

Would be higly appreciated as you would help to have a look.

---

In full as below

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ll
total 88
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 12:45 ./
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 ../
-rw-rw-r-- 1 namgivu namgivu  8196 Jun  4 09:56 .DS_Store
-rwxrwxr-x 1 namgivu namgivu  2725 Jun  4 09:56 generate_data_list.py*
drwxrwxr-x 8 namgivu namgivu  4096 Jun  4 09:56 .git/
drwxrwxr-x 3 namgivu namgivu  4096 Jun  4 10:36 .idea/
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 inception/
-rwxrwxr-x 1 namgivu namgivu     0 Jun  4 09:56 __init__.py*
-rw-rw-r-- 1 namgivu namgivu     7 Jun  4 09:56 .python-version
-rw-rw-r-- 1 namgivu namgivu  2893 Jun  4 09:56 README.md
-rw-rw-r-- 1 namgivu namgivu   147 Jun  4 10:06 requirements.txt
lrwxrwxrwx 1 namgivu namgivu    42 Jun  4 12:45 SPI_eval -> /media/namgivu/Elements/deepsonar/SPI_eval
-rwxrwxr-x 1 namgivu namgivu  3004 Jun  4 09:56 test_classification.py*
-rwxrwxr-x 1 namgivu namgivu 10541 Jun  4 09:56 test_segmentation.py*
-rwxrwxr-x 1 namgivu namgivu 10551 Jun  4 09:56 train_classification.py*
-rwxrwxr-x 1 namgivu namgivu  8087 Jun  4 09:56 train_segmentation.py*
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 10:07 .venv/

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ll
total 88
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 12:45 ./
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 ../
-rw-rw-r-- 1 namgivu namgivu  8196 Jun  4 09:56 .DS_Store
-rwxrwxr-x 1 namgivu namgivu  2725 Jun  4 09:56 generate_data_list.py*
drwxrwxr-x 8 namgivu namgivu  4096 Jun  4 09:56 .git/
drwxrwxr-x 3 namgivu namgivu  4096 Jun  4 10:36 .idea/
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 inception/
-rwxrwxr-x 1 namgivu namgivu     0 Jun  4 09:56 __init__.py*
-rw-rw-r-- 1 namgivu namgivu     7 Jun  4 09:56 .python-version
-rw-rw-r-- 1 namgivu namgivu  2893 Jun  4 09:56 README.md
-rw-rw-r-- 1 namgivu namgivu   147 Jun  4 10:06 requirements.txt
lrwxrwxrwx 1 namgivu namgivu    42 Jun  4 12:45 SPI_eval -> /media/namgivu/Elements/deepsonar/SPI_eval/
-rwxrwxr-x 1 namgivu namgivu  3004 Jun  4 09:56 test_classification.py*
-rwxrwxr-x 1 namgivu namgivu 10541 Jun  4 09:56 test_segmentation.py*
-rwxrwxr-x 1 namgivu namgivu 10551 Jun  4 09:56 train_classification.py*
-rwxrwxr-x 1 namgivu namgivu  8087 Jun  4 09:56 train_segmentation.py*
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 10:07 .venv/

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python train_classification.py --fine_tune=False
Traceback (most recent call last):
  File "train_classification.py", line 264, in <module>
    train()
  File "train_classification.py", line 83, in train
    raise EnvironmentError('Data list not existed. Please run generate_data_list.py first.')
EnvironmentError: Data list not existed. Please run generate_data_list.py first.

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python generate_data_list.py
Train set list done. # positive samples: 0 # negative samples: 0
Validation set list done. # positive samples: 0 # negative samples: 0
Test set list done. # positive samples: 1221 # negative samples: 92279

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python train_classification.py --fine_tune=False
Training set built. Size: 0
Traceback (most recent call last):
  File "train_classification.py", line 264, in <module>
    train()
  File "train_classification.py", line 119, in train
    scope=None)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/inception/inception_model.py", line 95, in inference
    scope=scope)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/inception/slim/inception_model.py", line 126, in inception_v3
    net = tf.concat(3, [branch1x1, branch5x5, branch3x3dbl, branch_pool])
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1121, in concat
    dtype=dtypes.int32).get_shape().assert_is_compatible_with(
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1050, in convert_to_tensor
    as_ref=False)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 971, in _autopacking_conversion_function
    return _autopacking_helper(v, dtype, name or "packed")
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 923, in _autopacking_helper
    return gen_array_ops.pack(elems_as_tensors, name=scope)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4875, in pack
    "Pack", values=values, axis=axis, name=name)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1792, in __init__
    control_input_ops)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1631, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimension 3 in both shapes must be equal, but are 96 and 32. Shapes are [32,35,35,96] and [32,35,35,32].
        From merging shape 2 with other shapes. for 'inception_v3/mixed_35x35x256a/concat/concat_dim' (op: 'Pack') with input shapes: [32,35,35,64], [32,35,35,64], [32,35,35,96], [32,35,35,32].

[namgivu]

Hi @raphaeljolivet,
May you have free time, please have a look at the error here[(https://github.com//issues/7#issuecomment-854381647). Thanks a lot!