AttributeError: 'str' object has no attribute 'decode' - v5.0.0

Hello,

After just installed the latest version of SCT, this happens:

extop@:/aran/testing$ sct_deepseg_sc -i 3D_T1.nii -c t1 -centerline cnn

--
Spinal Cord Toolbox (5.0.0)

Config deepseg_sc:
  Centerline algorithm: cnn
  Brain in image: True
  Kernel dimension: 2d
  Contrast: t1
  Threshold: 0.15
Creating temporary folder (/tmp/sct-20201111092723.726825-8mjw_gc5)
Reorient the image to RPI, if necessary...
Finding the spinal cord centerline...
Traceback (most recent call last):
  File "/home/extop/sct_5.0.0/scripts/sct_deepseg_sc.py", line 208, in <module>
    main()
  File "/home/extop/sct_5.0.0/scripts/sct_deepseg_sc.py", line 192, in main
    threshold_seg=threshold, remove_temp_files=remove_temp_files, verbose=verbose)
  File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_sc/core.py", line 483, in deep_segmentation_spinalcord
    centerline_fname=file_ctr)
  File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_sc/core.py", line 93, in find_centerline
    ctr_model.load_weights(ctr_model_fname)
  File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 2656, in load_weights
    f, self.layers, reshape=reshape)
  File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 3328, in load_weights_from_hdf5_group
    original_keras_version = f.attrs['keras_version'].decode('utf8')
AttributeError: 'str' object has no attribute 'decode'

Nevertheless, I am able to run sct_deepseg for example.
Thanks!

Aran

@Aran, thank you for reporting this bug. Could you please copy/paste the output of:

sct_check_dependencies
sct_testing

Hello,

Here the sct_check_dependencies that went weird

extop@:/mnt/aran/testing/001_1$ sct_check_dependencies 

--
Spinal Cord Toolbox (5.0.0)

SCT info:
- version: 5.0.0
- path: /home/extop/sct_5.0.0
OS: linux (Linux-5.3.0-62-generic-x86_64-with-debian-buster-sid)
CPU cores: Available: 12, Used by ITK functions: 12
RAM: Total: 128511MB, Used: 20537MB, Available: 106404MB
Check Python executable.............................[OK]
  Using bundled python 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) 
[GCC 7.3.0] at /home/extop/sct_5.0.0/python/envs/venv_sct/bin/python
Check if pytest-cov is installed....................[OK] (__version__ = '2.10.1')
Check if requests is installed......................[OK] (2.24.0)
Check if requirements-parser is installed...........[OK] (0.2.0)
Check if scipy is installed.........................[OK] (1.5.4)
Check if scikit-image is installed..................[OK] (0.17.2)
Check if scikit-learn is installed..................[OK] (0.23.2)
Check if tensorflow (1.5.0) is installed............[OK] (1.5.0)
Check if torch (1.5.0+cpu) is installed.............[WARNING] (1.6.0 != 1.5.0+cpu mandated version))
Check if torchvision (0.6.0+cpu) is installed.......[WARNING] (0.7.0 != 0.6.0+cpu mandated version))
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.51.0)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3 is installed.......................[OK] (1.25.11)
Check if pytest_console_scripts is installed........[OK]
Check if pytest-xdist is installed..................[OK] (2.1.0)
Check if figure can be opened with PyQt.............[OK]

And then, the sct_testing

extop@:/mnt/aran/testing/001_1$ sct_testing 

--
Spinal Cord Toolbox (5.0.0)

Trying URL: https://github.com/sct-data/sct_testing_data/releases/download/r20201030/sct_testing_data-r20201030.zip
Downloading: sct_testing_data-r20201030.zip
Status: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.17M/7.17M [00:12<00:00, 595kB/s]
Creating temporary folder (/tmp/sct-20201113075848.995069-2wp8i281)
Unzip data to: /tmp/sct-20201113075848.995069-2wp8i281
Removing temporary folders...
Done!

Creating temporary folder (/tmp/sct-20201113075849.064687-gpcy0fls)
Will run through the following tests:
- sequentially: sct_deepseg_gm sct_deepseg_lesion sct_deepseg_sc
- in parallel with 12 jobs: sct_analyze_lesion sct_analyze_texture sct_apply_transfo sct_convert sct_compute_ernst_angle sct_compute_hausdorff_distance sct_compute_mtr sct_compute_mscc sct_compute_snr sct_create_mask sct_crop_image sct_dice_coefficient sct_detect_pmj sct_dmri_compute_dti sct_dmri_concat_b0_and_dwi sct_dmri_concat_bvals sct_dmri_concat_bvecs sct_dmri_compute_bvalue sct_dmri_moco sct_dmri_separate_b0_and_dwi sct_dmri_transpose_bvecs sct_extract_metric sct_flatten_sagittal sct_fmri_compute_tsnr sct_fmri_moco sct_get_centerline sct_image sct_label_utils sct_label_vertebrae sct_maths sct_merge_images sct_process_segmentation sct_propseg sct_qc sct_register_multimodal sct_register_to_template sct_resample sct_smooth_spinalcord sct_straighten_spinalcord sct_warp_template
Checking sct_deepseg_gm.............................[FAIL]
   
   ====================================================================================================
   sct_deepseg_gm -i t2s/t2s_uncropped.nii.gz -o output.nii.gz -qc testing-qc
   ====================================================================================================
   
   
   --
   Spinal Cord Toolbox (5.0.0)
   
   Traceback (most recent call last):
   File "/home/extop/sct_5.0.0/scripts/sct_deepseg_gm.py", line 138, in <module>
   run_main()
   File "/home/extop/sct_5.0.0/scripts/sct_deepseg_gm.py", line 121, in run_main
   use_tta)
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_gm/deepseg_gm.py", line 313, in segment_file
   use_tta)
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_gm/deepseg_gm.py", line 233, in segment_volume
   deepgmseg_model.load_weights(model_abs_path)
   File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 2656, in load_weights
   f, self.layers, reshape=reshape)
   File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 3328, in load_weights_from_hdf5_group
   original_keras_version = f.attrs['keras_version'].decode('utf8')
   AttributeError: 'str' object has no attribute 'decode'
Checking sct_deepseg_lesion.........................[FAIL]
   
   ====================================================================================================
   sct_deepseg_lesion -i t2/t2.nii.gz -c t2
   ====================================================================================================
   
   
   --
   Spinal Cord Toolbox (5.0.0)
   
   
   Method:
   Centerline algorithm: svm
   Assumes brain section included in the image: True
   
   Creating temporary folder (/tmp/sct-20201113075855.108779-xyc7gjs3)
   Reorient the image to RPI, if necessary...
   
   Finding the spinal cord centerline...
   Creating temporary folder (/tmp/sct-20201113075855.446297-yc033ooj)
   Remove temporary files...
   rm -rf /tmp/sct-20201113075855.446297-yc033ooj
   
   Cropping the image around the spinal cord...
   Normalizing the intensity...
   load data...
   
   Segmenting the MS lesions using deep learning on 3D patches...
   Traceback (most recent call last):
   File "/home/extop/sct_5.0.0/scripts/sct_deepseg_lesion.py", line 176, in <module>
   main()
   File "/home/extop/sct_5.0.0/scripts/sct_deepseg_lesion.py", line 152, in main
   brain_bool=brain_bool, remove_temp_files=remove_temp_files, verbose=verbose)
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_lesion/core.py", line 201, in deep_segmentation_MSlesion
   im=im_res3d.copy())
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_lesion/core.py", line 88, in segment_3d
   seg_model = load_trained_model(model_fname)
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_sc/cnn_models_3d.py", line 91, in load_trained_model
   return load_model(model_file, custom_objects=custom_objects)
   File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/models.py", line 242, in load_model
   model_config = json.loads(model_config.decode('utf-8'))
   AttributeError: 'str' object has no attribute 'decode'
Checking sct_deepseg_sc.............................[FAIL]
   
   ====================================================================================================
   sct_deepseg_sc -i t2/t2.nii.gz -c t2 -qc testing-qc
   ====================================================================================================
   
   
   --
   Spinal Cord Toolbox (5.0.0)
   
   Config deepseg_sc:
   Centerline algorithm: svm
   Brain in image: True
   Kernel dimension: 2d
   Contrast: t2
   Threshold: 0.7
   Creating temporary folder (/tmp/sct-20201113075858.725650-0mpax34n)
   Reorient the image to RPI, if necessary...
   Finding the spinal cord centerline...
   Creating temporary folder (/tmp/sct-20201113075858.980012-fde3c2iu)
   Remove temporary files...
   rm -rf /tmp/sct-20201113075858.980012-fde3c2iu
   Cropping the image around the spinal cord...
   Normalizing the intensity...
   Segmenting the spinal cord using deep learning on 2D patches...
   Traceback (most recent call last):
   File "/home/extop/sct_5.0.0/scripts/sct_deepseg_sc.py", line 208, in <module>
   main()
   File "/home/extop/sct_5.0.0/scripts/sct_deepseg_sc.py", line 192, in main
   threshold_seg=threshold, remove_temp_files=remove_temp_files, verbose=verbose)
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_sc/core.py", line 509, in deep_segmentation_spinalcord
   im_in=im_norm_in)
   File "/home/extop/sct_5.0.0/spinalcordtoolbox/deepseg_sc/core.py", line 350, in segment_2d
   seg_model.load_weights(model_fname)
   File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 2656, in load_weights
   f, self.layers, reshape=reshape)
   File "/home/extop/sct_5.0.0/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 3328, in load_weights_from_hdf5_group
   original_keras_version = f.attrs['keras_version'].decode('utf8')
   AttributeError: 'str' object has no attribute 'decode'
Checking sct_analyze_lesion.........................[OK]
Checking sct_analyze_texture........................[OK]
Checking sct_apply_transfo..........................[OK]
Checking sct_convert................................[OK]
Checking sct_compute_ernst_angle....................[OK]
Checking sct_compute_hausdorff_distance.............[OK]
Checking sct_compute_mtr............................[OK]
Checking sct_compute_mscc...........................[OK]
Checking sct_compute_snr............................[OK]
Checking sct_create_mask............................[OK]
Checking sct_crop_image.............................[OK]
Checking sct_dice_coefficient.......................[OK]
Checking sct_detect_pmj.............................[OK]
Checking sct_dmri_compute_dti.......................[OK]
Checking sct_dmri_concat_b0_and_dwi.................[OK]
Checking sct_dmri_concat_bvals......................[OK]
Checking sct_dmri_concat_bvecs......................[OK]
Checking sct_dmri_compute_bvalue....................[OK]
Checking sct_dmri_moco..............................[OK]
Checking sct_dmri_separate_b0_and_dwi...............[OK]
Checking sct_dmri_transpose_bvecs...................[OK]
Checking sct_extract_metric.........................[OK]
Checking sct_flatten_sagittal.......................[OK]
Checking sct_fmri_compute_tsnr......................[OK]
Checking sct_fmri_moco..............................[OK]
Checking sct_get_centerline.........................[OK]
Checking sct_image..................................[OK]
Checking sct_label_utils............................[OK]
Checking sct_label_vertebrae........................[OK]
Checking sct_maths..................................[OK]
Checking sct_merge_images...........................[OK]
Checking sct_process_segmentation...................[OK]
Checking sct_propseg................................[OK]
Checking sct_qc.....................................[OK]
Checking sct_register_multimodal....................[OK]
Checking sct_register_to_template...................[OK]
Checking sct_resample...............................[OK]
Checking sct_smooth_spinalcord......................[OK]
Checking sct_straighten_spinalcord..................[OK]
Checking sct_warp_template..........................[OK]
status: [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Failures: sct_deepseg_gm sct_deepseg_lesion sct_deepseg_sc
Finished! Elapsed time: 129s

This is a known issue in 5.0.0, which has now been fixed in the master branch and will be available in v5.0.1. You could already try this out if you install SCT from the latest (master) branch.

(related to Depracated Matplotlib v.4.3)

Hi @Aran. SCT v5.0.1 has been released.

Could you please try reinstalling this newest version and let us know how that goes?

Thank you kindly,
Joshua

Hello both!

Apologies for the late response.

The problem still persists

extop@vh:/mnt/DADES/aran/testin/Canal/007_1$ sct_deepseg_sc -i CNS_007_1_3D_T1_MS-P.nii -c t1 -centerline cnn

--
Spinal Cord Toolbox (5.0.1)

sct_deepseg_sc -i CNS_007_1_3D_T1_MS-P.nii -c t1 -centerline cnn
--

Config deepseg_sc:
  Centerline algorithm: cnn
  Brain in image: True
  Kernel dimension: 2d
  Contrast: t1
  Threshold: 0.15
Creating temporary folder (/tmp/sct-20201123083139.498690-a1l9qjqv)
Reorient the image to RPI, if necessary...
Finding the spinal cord centerline...
Traceback (most recent call last):
  File "/home/extop/sct_5.0.1/scripts/sct_deepseg_sc.py", line 208, in <module>
    main()
  File "/home/extop/sct_5.0.1/scripts/sct_deepseg_sc.py", line 192, in main
    threshold_seg=threshold, remove_temp_files=remove_temp_files, verbose=verbose)
  File "/home/extop/sct_5.0.1/spinalcordtoolbox/deepseg_sc/core.py", line 483, in deep_segmentation_spinalcord
    centerline_fname=file_ctr)
  File "/home/extop/sct_5.0.1/spinalcordtoolbox/deepseg_sc/core.py", line 93, in find_centerline
    ctr_model.load_weights(ctr_model_fname)
  File "/home/extop/sct_5.0.1/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 2656, in load_weights
    f, self.layers, reshape=reshape)
  File "/home/extop/sct_5.0.1/python/envs/venv_sct/lib/python3.6/site-packages/keras/engine/topology.py", line 3328, in load_weights_from_hdf5_group
    original_keras_version = f.attrs['keras_version'].decode('utf8')
AttributeError: 'str' object has no attribute 'decode'

Outputs of sct_testing and sct_check_dependencies are identic as shown in my previous post

thans!

Hi @Aran. I’m sorry to hear that the update has not fixed your problem. Thank you for your patience as we investigate. :slightly_smiling_face:


I notice that the output of sct_check_dependencies from this comment is missing some entries. To demonstrate, this is (more or less) what you should see when you run sct_check_dependencies:

joshua@XPS-15-9560:~$ sct_check_dependencies

--
Spinal Cord Toolbox (git-jn/2857-template-registration-tutorial-c66848a2497f21bd665c9915336b8acebbf57f1c)

sct_check_dependencies 
--

SCT info:
- version: git-jn/2857-template-registration-tutorial-c66848a2497f21bd665c9915336b8acebbf57f1c
- path: /home/joshua/repos/spinalcordtoolbox
OS: linux (Linux-5.4.0-54-generic-x86_64-with-debian-bullseye-sid)
CPU cores: Available: 8, Used by ITK functions: 8
RAM: Total: 7812MB, Used: 2858MB, Available: 4347MB
Check Python executable.............................[OK]
  Using bundled python 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) 
[GCC 7.3.0] at /home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if colored is installed.......................[OK] (1.4.2)
Check if dipy is installed..........................[OK] (1.3.0)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (2.10.0)
Check if ivadomed (2.4.0) is installed..............[OK] (2.4.0)
Check if Keras (2.1.5) is installed.................[OK] (2.1.5)
Check if matplotlib is installed....................[OK] (3.3.3)
Check if nibabel is installed.......................[OK] (3.2.0)
Check if numpy is installed.........................[OK] (1.19.4)
Check if onnxruntime (1.4.0) is installed...........[OK] (1.4.0)
Check if pandas is installed........................[OK] (1.1.4)
Check if psutil is installed........................[OK] (5.7.3)
Check if pyqt5 (5.11.3) is installed................[OK] (5.11.3)
Check if pytest is installed........................[OK] (6.1.2)
Check if pytest-cov is installed....................[OK] (__version__ = '2.10.1')
Check if raven is installed.........................[OK]
Check if requests is installed......................[OK] (2.25.0)
Check if requirements-parser is installed...........[OK] (0.2.0)
Check if scipy is installed.........................[OK] (1.5.4)
Check if scikit-image is installed..................[OK] (0.17.2)
Check if scikit-learn is installed..................[OK] (0.23.2)
Check if tensorflow (1.5.0) is installed............[OK] (1.5.0)
Check if torch (1.5.0+cpu) is installed.............[OK] (1.5.0+cpu)
Check if torchvision (0.6.0+cpu) is installed.......[OK] (0.6.0+cpu)
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.53.0)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3 is installed.......................[OK] (1.26.2)
Check if pytest_console_scripts is installed........[OK]
Check if pytest-xdist is installed..................[OK] (2.1.0)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if DISPLAY variable is set....................[OK]
Check if figure can be opened with PyQt.............[OK]

Specifically, the line I am looking for is "Check if h5py is installed", because h5py is the root cause of the issue.


As an alternative, we can check the dependencies another way. Could you please run the following commands and paste the full output?

cd $SCT_DIR
source python/etc/profile.d/conda.sh
conda activate venv_sct
pip freeze

This should result in an output similar to what you see below:

joshua@XPS-15-9560:~$ cd $SCT_DIR
joshua@XPS-15-9560:~/repos/spinalcordtoolbox$ source python/etc/profile.d/conda.sh
joshua@XPS-15-9560:~/repos/spinalcordtoolbox$ conda activate venv_sct
(venv_sct) joshua@XPS-15-9560:~/repos/spinalcordtoolbox$ pip freeze
absl-py==0.11.0
alabaster==0.7.12
apipkg==1.5
attrs==20.3.0
Babel==2.9.0
bids-neuropoly==0.2
bids-validator==1.5.7
bleach==1.5.0
cachetools==4.1.1
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==7.1.2
colored==1.4.2
commonmark==0.9.1
coverage==5.3
cryptography==3.2.1
cycler==0.10.0
decorator==4.4.2
dipy==1.3.0
docopt==0.6.2
docutils==0.16
execnet==1.7.1
future==0.18.2
futures==3.1.1
google-auth==1.23.0
google-auth-oauthlib==0.4.2
grpcio==1.33.2
h5py==2.10.0
html5lib==0.9999999
idna==2.10
imageio==2.9.0
imagesize==1.2.0
importlib-metadata==2.0.0
iniconfig==1.1.1
ivadomed==2.4.0
Jinja2==2.11.2
joblib==0.17.0
Keras==2.1.5
kiwisolver==1.3.1
Markdown==3.3.3
MarkupSafe==1.1.1
matplotlib==3.3.3
mock==4.0.2
networkx==2.5
nibabel==3.2.0
num2words==0.5.10
numpy==1.19.4
oauthlib==3.1.0
onnxruntime==1.4.0
packaging==20.4
pandas==1.1.4
patsy==0.5.1
Pillow==8.0.1
pluggy==0.13.1
protobuf==3.14.0
psutil==5.7.3
py==1.9.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybids==0.12.4
pycparser==2.20
Pygments==2.7.2
pyOpenSSL==19.1.0
pyparsing==2.4.7
PyQt5==5.11.3
PyQt5-sip==4.19.19
pytest==6.1.2
pytest-console-scripts==1.1.0
pytest-cov==2.10.1
pytest-forked==1.3.0
pytest-xdist==2.1.0
python-dateutil==2.8.1
pytz==2020.4
PyWavelets==1.1.1
PyYAML==5.3.1
raven==6.10.0
recommonmark==0.6.0
requests==2.25.0
requests-oauthlib==1.3.0
requirements-parser==0.2.0
rsa==4.6
scikit-image==0.17.2
scikit-learn==0.23.2
scipy==1.5.4
seaborn==0.11.0
six==1.15.0
snowballstemmer==2.0.0
Sphinx==3.3.1
sphinx-rtd-theme==0.5.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-programoutput==0.16
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
-e git+https://github.com/neuropoly/spinalcordtoolbox.git@c66848a2497f21bd665c9915336b8acebbf57f1c#egg=spinalcordtoolbox
SQLAlchemy==1.3.20
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==1.5.0
tensorflow-tensorboard==1.5.1
threadpoolctl==2.1.0
tifffile==2020.9.3
toml==0.10.2
torch==1.5.0+cpu
torchvision==0.6.0+cpu
tqdm==4.53.0
transforms3d==0.3.1
urllib3==1.26.2
Werkzeug==1.0.1
xlwt==1.3.0
zipp==3.4.0

Hello @joshuacwnewton,

My apologies about the sct_check_dependencies. Here is the full output, i believe everything is correct

sct_check_dependencies 

--
Spinal Cord Toolbox (5.0.1)

sct_check_dependencies 
--

SCT info:
- version: 5.0.1
- path: /home/extop/sct_5.0.1
OS: linux (Linux-5.3.0-62-generic-x86_64-with-debian-buster-sid)
CPU cores: Available: 12, Used by ITK functions: 12
RAM: Total: 128511MB, Used: 4862MB, Available: 122310MB
Check Python executable.............................[OK]
  Using bundled python 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) 
[GCC 7.3.0] at /home/extop/sct_5.0.1/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if colored is installed.......................[OK] (1.4.2)
Check if dipy is installed..........................[OK] (1.3.0)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (3.1.0)
Check if ivadomed (2.4.0) is installed..............[OK] (2.4.0)
Check if Keras (2.1.5) is installed.................[OK] (2.1.5)
Check if matplotlib is installed....................[OK] (3.3.2)
Check if nibabel is installed.......................[OK] (3.2.0)
Check if numpy is installed.........................[OK] (1.19.4)
Check if onnxruntime (1.4.0) is installed...........[OK] (1.4.0)
Check if pandas is installed........................[OK] (1.1.4)
Check if psutil is installed........................[OK] (5.7.3)
Check if pyqt5 (5.11.3) is installed................[OK] (5.11.3)
Check if pytest is installed........................[OK] (6.1.2)
Check if pytest-cov is installed....................[OK] (__version__ = '2.10.1')
Check if raven is installed.........................[OK]
Check if requests is installed......................[OK] (2.24.0)
Check if requirements-parser is installed...........[OK] (0.2.0)
Check if scipy is installed.........................[OK] (1.5.4)
Check if scikit-image is installed..................[OK] (0.17.2)
Check if scikit-learn is installed..................[OK] (0.23.2)
Check if tensorflow (1.5.0) is installed............[OK] (1.5.0)
Check if torch (1.5.0+cpu) is installed.............[WARNING] (1.6.0 != 1.5.0+cpu mandated version))
Check if torchvision (0.6.0+cpu) is installed.......[WARNING] (0.7.0 != 0.6.0+cpu mandated version))
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.51.0)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3 is installed.......................[OK] (1.25.11)
Check if pytest_console_scripts is installed........[OK]
Check if pytest-xdist is installed..................[OK] (2.1.0)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if DISPLAY variable is set....................[OK]
Check if figure can be opened with PyQt.............[OK]

Nevertheless, I also post the alternative you commented

extop@vh:~$ cd $SCT_DIR
extop@vh:~/sct_5.0.1$ source python/etc/profile.d/conda.sh 
extop@vh:~/sct_5.0.1$ conda activate venv_sct
(venv_sct) extop@vh:~/sct_5.0.1$ pip freeze
absl-py==0.11.0
apipkg==1.5
attrs==20.3.0
bids-neuropoly==0.2
bids-validator==1.5.7
bleach==1.5.0
cached-property==1.5.2
cachetools==4.1.1
certifi==2020.11.8
cffi==1.14.3
chardet==3.0.4
ci-info==0.2.0
click==7.1.2
colored==1.4.2
coverage==5.3
cryptography==3.2.1
cycler==0.10.0
Cython==0.29.21
dcm2bids==2.1.4
decorator==4.4.2
dipy==1.3.0
docopt==0.6.2
etelemetry==0.2.2
execnet==1.7.1
filelock==3.0.12
future==0.18.2
futures==3.1.1
google-auth==1.23.0
google-auth-oauthlib==0.4.2
grpcio==1.33.2
h5py==3.1.0
html5lib==0.9999999
idna==2.10
imageio==2.9.0
importlib-metadata==2.0.0
iniconfig==1.1.1
install==1.3.4
isodate==0.6.0
ivadomed==2.4.0
Jinja2==2.11.2
joblib==0.17.0
Keras==2.1.5
kiwisolver==1.3.1
lxml==4.6.1
Markdown==3.3.3
MarkupSafe==1.1.1
matplotlib==3.3.2
mock==4.0.2
mriqc==0.15.2+11.g5898396
networkx==2.5
nibabel==3.2.0
nilearn==0.6.2
nipype==1.5.1
nitime==0.8.1
nitransforms==20.0.0rc3
niworkflows==1.3.2
num2words==0.5.10
numpy==1.19.4
oauthlib==3.1.0
onnxruntime==1.4.0
packaging==20.4
pandas==1.1.4
patsy==0.5.1
Pillow==8.0.1
pluggy==0.13.1
protobuf==3.13.0
prov==2.0.0
psutil==5.7.3
py==1.9.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybids==0.12.3
pycparser==2.20
pydot==1.4.1
pydotplus==2.0.2
pyOpenSSL==19.1.0
pyparsing==2.4.7
PyQt5==5.11.3
PyQt5-sip==4.19.19
pytest==6.1.2
pytest-console-scripts==1.0.0
pytest-cov==2.10.1
pytest-forked==1.3.0
pytest-xdist==2.1.0
python-dateutil==2.8.1
pytz==2020.4
PyWavelets==1.1.1
PyYAML==5.3.1
raven==6.10.0
rdflib==5.0.0
requests==2.24.0
requests-oauthlib==1.3.0
requirements-parser==0.2.0
rsa==4.6
scikit-image==0.17.2
scikit-learn==0.23.2
scipy==1.5.4
seaborn==0.11.0
simplejson==3.17.2
six==1.15.0
sklearn==0.0
# Editable install with no version control (spinalcordtoolbox==5.0.1)
-e /home/extop/sct_5.0.1
SQLAlchemy==1.3.20
statsmodels==0.12.1
svgutils==0.3.1
templateflow==0.6.3
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==1.5.0
tensorflow-tensorboard==1.5.1
threadpoolctl==2.1.0
tifffile==2020.9.3
toml==0.10.2
torch==1.6.0
torchvision==0.7.0
tqdm==4.51.0
traits==6.1.1
transforms3d==0.3.1
urllib3==1.25.11
Werkzeug==1.0.1
xlwt==1.3.0
xvfbwrapper==0.2.9
zipp==3.4.0

I noticed that my h5py is 3.1.0 and yours is 2.10. Probably that’s the problem
And alabaster, Babel and others are missing.

Be in touch! And thank you for your work with my problem

Aran

Thank you very much, @Aran. You are spot-on with your observation about the h5py versions – that is indeed the issue here. :slightly_smiling_face:

The question to figure out now is: What went wrong during SCT’s installation to cause this to happen?

SCT’s installation script has a verbose mode (-v) that may provide us with more clues as to what’s going on. Could you please reinstall SCT using the following command, then attach the resulting install_log.txt file?

./install_sct -v > install_log.txt

(The output might look a little strange in verbose mode, but don’t worry, as that’s normal.)

Thank you again for your patience!
Joshua

Hello!

Attached you will find the install_log file.
install_log.txt (31.1 KB)

In line 288 it states clearly that it is installing h5py 2.10.0.

Nevertheless, an immediate sct_check_dependencie gives me the same as before with h5py 3.1…
Between both things i did run the source /home/extop/.bashrc


extop@:~/spinalcordtoolbox-5.0.1$ source /home/extop/.bashrc
extop@:~/spinalcordtoolbox-5.0.1$ sct_check_dependencies 

--
Spinal Cord Toolbox (5.0.1)

sct_check_dependencies 
--

SCT info:
- version: 5.0.1
- path: /home/extop/sct_5.0.1
OS: linux (Linux-5.3.0-62-generic-x86_64-with-debian-buster-sid)
CPU cores: Available: 12, Used by ITK functions: 12
RAM: Total: 128511MB, Used: 11805MB, Available: 115259MB
Check Python executable.............................[OK]
  Using bundled python 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) 
[GCC 7.3.0] at /home/extop/sct_5.0.1/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if colored is installed.......................[OK] (1.4.2)
Check if dipy is installed..........................[OK] (1.3.0)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (3.1.0)
Check if ivadomed (2.4.0) is installed..............[OK] (2.4.0)
Check if Keras (2.1.5) is installed.................[OK] (2.1.5)
Check if matplotlib is installed....................[OK] (3.3.2)
Check if nibabel is installed.......................[OK] (3.2.0)
Check if numpy is installed.........................[OK] (1.19.4)
Check if onnxruntime (1.4.0) is installed...........[OK] (1.4.0)
Check if pandas is installed........................[OK] (1.1.4)
Check if psutil is installed........................[OK] (5.7.3)
Check if pyqt5 (5.11.3) is installed................[OK] (5.11.3)
Check if pytest is installed........................[OK] (6.1.2)
Check if pytest-cov is installed....................[OK] (__version__ = '2.10.1')
Check if raven is installed.........................[OK]
Check if requests is installed......................[OK] (2.24.0)
Check if requirements-parser is installed...........[OK] (0.2.0)
Check if scipy is installed.........................[OK] (1.5.4)
Check if scikit-image is installed..................[OK] (0.17.2)
Check if scikit-learn is installed..................[OK] (0.23.2)
Check if tensorflow (1.5.0) is installed............[OK] (1.5.0)
Check if torch (1.5.0+cpu) is installed.............[WARNING] (1.6.0 != 1.5.0+cpu mandated version))
Check if torchvision (0.6.0+cpu) is installed.......[WARNING] (0.7.0 != 0.6.0+cpu mandated version))
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.51.0)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3 is installed.......................[OK] (1.25.11)
Check if pytest_console_scripts is installed........[OK]
Check if pytest-xdist is installed..................[OK] (2.1.0)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if DISPLAY variable is set....................[OK]
Check if figure can be opened with PyQt.............[OK]

Thank you very much for sharing the log. I agree that the installation looks like it has gone OK. :slightly_smiling_face:

This is very strange indeed! I would like to do a little more investigating to find out where the other version of h5py is coming from. Could you please run the following command in your terminal?

$SCT_DIR/python/envs/venv_sct/bin/python

This should open up an interactive Python interpreter. Then, once inside, could you run the following commands and share the output?

import sys
sys.path
import h5py
h5py.__version__
import importlib
importlib.util.find_spec('h5py')
exit()

The results should look something similar to this:

joshua@XPS-15-9560:~$ $SCT_DIR/python/envs/venv_sct/bin/python
Python 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/lib/python36.zip', '/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6', '/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/lib-dynload', '/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/site-packages', '/home/joshua/repos/spinalcordtoolbox']
>>> import h5py
>>> h5py.__version__
'2.10.0'
>>> import importlib
>>> importlib.util.find_spec('h5py')
ModuleSpec(name='h5py', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fd4850ed358>, origin='/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/site-packages/h5py/__init__.py', submodule_search_locations=['/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/site-packages/h5py'])
>>> exit()
joshua@XPS-15-9560:~$ 

Thank you again for your patience and cooperation while solving this issue. :slight_smile:
-Joshua

Goooooooooood morning!

Thank you very much helping with this issue!

Here is the output you asked

extop@b:~$ $SCT_DIR/python/envs/venv_sct/bin/python
Python 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/extop/sct_5.0.1/python/envs/venv_sct/lib/python36.zip', '/home/extop/sct_5.0.1/python/envs/venv_sct/lib/python3.6', '/home/extop/sct_5.0.1/python/envs/venv_sct/lib/python3.6/lib-dynload', '/home/extop/.local/lib/python3.6/site-packages', '/home/extop/sct_5.0.1/python/envs/venv_sct/lib/python3.6/site-packages', '/home/extop/sct_5.0.1']
>>> import h5py
>>> h5py.__version__
'3.1.0'
>>> import importlib
>>> importlib.util.find_spec('h5py')
ModuleSpec(name='h5py', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f595a1eb5f8>, origin='/home/extop/.local/lib/python3.6/site-packages/h5py/__init__.py', submodule_search_locations=['/home/extop/.local/lib/python3.6/site-packages/h5py'])
>>> exit()
extop@b:~$ 

Cheers,

Aran

Thank you. That does indeed explain things. :slight_smile:

Your system has a directory (called the “user site directory”) located at /home/extop/.local/ where Python packages can be installed to. In your case, different versions of packages may have been installed here, separate from the SCT installation (e.g. by running pip install --user h5py).

The packages in this directory are taking precedence over the packages installed with SCT, and thus causing issues. To verify, could you run the following?

export PYTHONNOUSERSITE=True
sct_check_dependencies

This temporarily sets the PYTHONNOUSERSITE environment variable, which should prevent the packages from /home/extop/.local/ from interfering.

Even If this works, however, this is still only a short-term fix, as you would have to type EXPORT PYTHONNOUSERSITE=True every time you used an SCT script. So, I plan to discuss this further with the SCT development team to determine a more appropriate long-term fix. :slight_smile:

Kindly,
Joshua

Dear Joshua,

This solves it! Amazing…

Attached there is an image from the output of the instructions you said.

And here is the desired h5py! Afterwards, a sct_deepseg_sc worked perfectly, producing the desired segmentation.

Also the output from sct_testing was perfect, no errors!

I will expect the new branch! But now the SCT works!

Thank you very much,

Aran

1 Like

I’m so glad to hear the good news! Thank you again for all of the help in pinpointing this cause of this issue. Once fixed, this will help improve SCT for other people as well. :slightly_smiling_face:

One last thing you can do to make this slightly more convenient is to add the “export PYTHONNOUSERSITE=True” line to the end of a file called .bashrc. This will make it so that the variable is set automatically each time you open a terminal window.

gedit ~/.bashrc

This should open up a text editor. Then you can put the command next to the other entries created by SCT. For example:

# SPINALCORDTOOLBOX (installed on 2020-11-26 09:44:26)
export PATH="/home/joshua/repos/spinalcordtoolbox/bin:$PATH"
export SCT_DIR=/home/joshua/repos/spinalcordtoolbox
export MPLBACKEND=Agg
export PYTHONNOUSERSITE=True

The downside to adding it to .bashrc is that if you use other Python CLI tools, there is a small chance that they could be impacted, because the change will be active for all terminal windows. (Hence why this is a temporary solution at best until we determine a better fix on our end.)

As for the long-term fix, a report has been made in our issue tracker here if you would like to follow progress. :slightly_smiling_face:

Kind regards,
Joshua

Hi @Aran,

One more update: A fix for this issue has now been added to SCT. If you would like, you can install the development version of SCT to access the fix now. Otherwise, the fix will be included in the next version of SCT.

Thank you again for helping to improve SCT. :slight_smile:

Kindly,
Joshua

2 Likes