How to use the updated sct_propseg (v5.3.0) for CentOS/RHEL 6?

Hi,
I was wondering what are the steps to compile additional binary files for specific SCT version?

Here is what I mean, I have version 3.2.3 and I cannot change it. I need some functionalities from version 5.3.0. For example, it turns out that PropSeg in 5.3.0 is MORE successful then PropSeg in 3.2.3.

Since I cannot change SCT version (due to compiler issues) I was thinking of adding additional binary (i.e. sct_propseg_5.3.0). I thought maybe just to create sct_propseg_5.3.0.py and update additional files (like Makefiles and so on) in the process.

How could I go through with something like this? I mean, before executing ./install_sct again, what are the files in spinalcordtoolbox-3.2.3 that I would have to update?

Much thanks! This community has been of HUGE Help to me!

Hi @Stefan_Radonjic,

Thank you for your post. Before suggesting creating a hybrid version of SCT, I would like to know more about "Since I cannot change SCT version (due to compiler issues)". Could you please report the issue you are observing? Ideally a copy/paste of the Terminal output.

Cheers
Julien

HI @jcohenadad !

Well, system my company is using cannot change its GLIBC version, and when I try to install latest version of SCT it saids GLIBC 2.14 is requiered. We cannot do anything about that for the time being, so Hybrid version of SCT might be our only solution.

Also, here is ERROR log when trying to install SCT 3.2.3. Which is strange because we already had it installed, I just wanted to install it (AGAIN) as a sanity check before going hybrid. Good thing that I did because it doesnt seem to work now.

Building wheels for collected packages: nibabel, Keras, theano, unknown, html5lib, unknown
  Running setup.py bdist_wheel for nibabel ... done
  Stored in directory: /home/sradonjic/.cache/pip/wheels/a8/ba/6c/34dc8726562736c6f2ca41d324641035377832dc11d4b3168f
  Running setup.py bdist_wheel for Keras ... done
  Stored in directory: /home/sradonjic/.cache/pip/wheels/d1/70/83/9be5aef9c4c863ea21adacd0be83139b20d3d819401a2b07d3
  Running setup.py bdist_wheel for theano ... done
  Stored in directory: /home/sradonjic/.cache/pip/wheels/89/40/74/3a0b7d937890c66c4373120117ebf4ba99f4889b4a0a6a810c
  Running setup.py bdist_wheel for unknown ... done
  Stored in directory: /home/sradonjic/.cache/pip/wheels/9a/8e/50/89667f452c1817d655a618f551e5227f4ce65f882856bd6bc9
  Running setup.py bdist_wheel for html5lib ... done
  Stored in directory: /home/sradonjic/.cache/pip/wheels/50/ae/f9/d2b189788efcf61d1ee0e36045476735c838898eef1cad6e29
  Running setup.py bdist_wheel for unknown ... error
  Complete output from command /mnt/algo/sct_3.2.3_upgrade/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-BgWt8P/unknown/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpDwMJiSpip-wheel- --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-build-BgWt8P/unknown/setup.py'

  ----------------------------------------
  Failed building wheel for unknown
  Running setup.py clean for unknown
  Complete output from command /mnt/algo/sct_3.2.3_upgrade/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-BgWt8P/unknown/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-build-BgWt8P/unknown/setup.py'

  ----------------------------------------
  Failed cleaning build dir for unknown
Successfully built nibabel Keras theano unknown html5lib
Failed to build unknown
Installing collected packages: futures, nibabel, unknown, sympy, nipy, dipy, tqdm, contextlib2, raven, protobuf, backports.weakref, setuptools, markdown, werkzeug, html5lib, bleach, tensorflow-tensorboard, funcsigs, mock, tensorflow, theano, Keras, h5py
  Found existing installation: setuptools 23.0.0
    Uninstalling setuptools-23.0.0:
Exception:
Traceback (most recent call last):
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/site-packages/pip/req/req_install.py", line 754, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/mnt/algo/sct_3.2.3_upgrade/python/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/mnt/algo/sct_3.2.3_upgrade/python/bin/easy_install-2.7'
You are using pip version 9.0.1, however version 21.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
pip install error with exit status 2. Check logs for more details
            and verify your internet connection before reinstalling
Installation failed

@jcohenadad Here is what I did:

  • Installed SCT_5.3.0 by COMMENTING OUT last couple of lines in install_sct which correspond to validation of dependencies. Problem is that Keras was not successfully installed due to wrong GLIBC version (which as we discussed I cannot change). Here is the error I was getting:

    As mentioned, I fixed this error by commenting out last couple of lines in install_sct because I dont think I need Keras in order to execute sct_propseg or sct_label_vertabrae (only binaries I need)

Here is the issue. When I run sct_propseg on machine where I installed SCT_5.3.0 as described above, I run into this error:

--
Spinal Cord Toolbox (5.3.0)

sct_propseg -i /opt/proc_data/10008886_280521_V-2805_22664208/Output/analysisFolder/stiched_spine.nii.gz -c t2 -CSF -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION/stiched_spin$
--

Creating temporary folder (/tmp/sct-20210604033426.295974-label_vertebrae-8_lthurk)
Creating temporary folder (/tmp/sct-20210604033426.523197-propseg-wko3ao_l)
Creating temporary folder (/tmp/sct-20210604033427.024734-6h35v9i4)
Remove temporary files...
rm -rf /tmp/sct-20210604033427.024734-6h35v9i4
/mnt/algo/sct_5.3.0/bin/isct_propseg -t t2 -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION -verbose -mesh -centerline-binary -CSF -centerline-coord -cross -init-tube -low-resol$
Automatic cord detection failed. Please initialize using -init-centerline or -init-mask (see help)

stderror is

While on the other machine (my local) where I installed SCT_5.3.0 without any issues (correct GLBC version), I had no errors what so ever.

What could be causing this?

Hi @Stefan_Radonjic,

It looks like there are 2 different problems in this thread: Fixing the v3.2.3 installation, and fixing the v5.3.0 installation. Before tackling either of the problems, I just want to clarify this part here:

  • What differences did you find between the v3.2.3 propseg and v5.3.0 propseg?
  • What was the exact “sct_propseg” command used? (I ask because in the pasted error log, the full command was cut off.)

Thanks for your patience. :slightly_smiling_face:
-Joshua

Hi @joshuacwnewton! The correct segmentation is produced with SCT 5.3 while the incorrect one is produced with SCT 3.2.3. I’ve used same data for both and -CSF flag as well.

Thank you for elaborating! In that case, we can focus on just the 5.3.0 installation, then.

If possible, could you please modify your “sct_propseg” command as follows:

# sample command just to demonstrate

# before
sct_propseg -i t2.nii.gz -c t2 -CSF

# after
sct_propseg -i t2.nii.gz -c t2 -CSF -v 2 > propseg_output_log.txt

To explain:

  • -v 2” will give us additional debugging information
  • " > propseg_output_log.txt" will put the terminal output into a text file for you (so you don’t have to copy and paste to the forum).

Then, could you rerun propseg and share with us the “propseg_output_log.txt” file that is created?

Kind regards,
Joshua

HI @joshuacwnewton and thats for quick reply!

Although, I think you misunderstood what I am trying to achieve. I need success rate of SCT’s 5.3.0 PropSeg in 3.2.3 version. Meaning, I would like to replicate changes in PropSeg that are in SC5 5.3.0 in the SCT 3.2.3 (If you understand what I mean).

Here is the file generated from SCT 3.2.3. which is getting us INCORRECT segmentation. I would like to have SCT 3.2.3 to give a CORRET one isntead.propseg-log.txt (5.4 KB)

I am currently trying to edit sct_propseg.py from SCT 3.2.3 to mimic sct_propseg.py from SCT 5.3.0.

Thank you for the clarification. I understand what you mean. :slightly_smiling_face:

I’ve looked into the possibility, but it is not straightforward to do this. Specifically, it turns out that the core algorithm for sct_propseg hasn’t actually changed between v3.2.3 and v5.3.0.

Binaries that are downloaded with v3.2.3:

Binaries that are downloaded with v5.3.0:

This means that the improvements you’ve noticed are due to updates to other parts of SCT. (For example, in 2019 there was a major refactoring of the OptiC centerline detection algorithm. OptiC is used in sct_propseg, so this may be one of the sources of improvement.)

Because of the scope of the changes that have been made over the years, it would be a nontrivial effort to stitch the improvements from v5.3.0 into v3.2.3.


You mention in an earlier comment that you were able to get v5.3.0 into a partially working state:

So, if we instead got “sct_propseg” to work for v5.3.0, would that be sufficient?

Kind regards,
Joshua

@joshuacwnewton oh, I understand!

But, yes! If we could somehow make sct_5.3.0 PropSeg to work that would be great! But, I cannot install it due to GLIBC version problem (which I cannot touch because it will mess with entire system). Any suggestions on how to do this?

I will generate logs from installation and send them in!

Of course! I think we might be able to work around this limitation. :slightly_smiling_face:

There was an earlier assessment you made that I agree with: sct_propseg and sct_label_vertebrae do not rely on Keras. So, in theory, they should be able to be run in isolation, even with the older GLIBC version.

That’s why I made this earlier suggestion. If we look at the debugging information for sct_propseg, we might be able to determine why it was that it failed.

@joshuacwnewton oh, sorry, I did not understood what you meant by your earlier suggestion. GIve me one second to generate you logs.

1 Like

@joshuacwnewton here you go, and really appreciate your help! Hope that this is possible. propseg_output_log.txt (143.2 KB)

Thank you! :slightly_smiling_face:

This is interesting… When I look at this output log, I don’t see the error that was mentioned in your earlier comment, and the output files appear to be created OK:

e[33mFile /home/stefan/sdintegration/pythonPlugin/test/__main__.Test.test_verify/Output/analysisFolder/stiched_spine_seg.nii.gz already exists. Will overwrite it.e[0m
e[33mFile /home/stefan/sdintegration/pythonPlugin/test/__main__.Test.test_verify/Output/analysisFolder/stiched_spine_centerline.nii.gz already exists. Will overwrite it.e[0m

Could you double-check to see if it’s working OK for you?

@joshuacwnewton sorry, a lot happened from that day. Basically, that was my “failed” attempt of patching SCT_3.2.3.

I think we can return to installation of SCT_5.3.0 as you suggested. The output of PropSeg I’ve sent you is from MY LOCAL Machine, where I installed SCT_5.3.0 without issues. When I try to install it on our system, I get the following: sct_error_log.txt (66.4 KB)

Also, I’ve noticed that when downloading data it fails due to some network error. :frowning:

Ahhh, I see!

The good news is: What you see in that error log is actually just a post-installation check. Meaning that, if you’re able get to that point, SCT has already installed itself.

So, on your remote machine, you should still be able to run the v5.3.0 version of sct_propseg (even with the GLIBC error).

@joshuacwnewton I just tried, and I get

--
Spinal Cord Toolbox (5.3.0)

sct_propseg -i /opt/proc_data/10008886_280521_V-2805_22664208/Output/analysisFolder/stiched_spine.nii.gz -c t2 -CSF -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION/stiched_spin$
--

Creating temporary folder (/tmp/sct-20210604033426.295974-label_vertebrae-8_lthurk)
Creating temporary folder (/tmp/sct-20210604033426.523197-propseg-wko3ao_l)
Creating temporary folder (/tmp/sct-20210604033427.024734-6h35v9i4)
Remove temporary files...
rm -rf /tmp/sct-20210604033427.024734-6h35v9i4
/mnt/algo/sct_5.3.0/bin/isct_propseg -t t2 -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION -verbose -mesh -centerline-binary -CSF -centerline-coord -cross -init-tube -low-resol$
Automatic cord detection failed. Please initialize using -init-centerline or -init-mask (see help)

stderror is

Could this be because while I was INSTALLING sct_5.3.0 it COULD NOT download the data?

Here is an example:

sct_download_data -d PAM50 -o /mnt/algo/sct_5.3.0/data/PAM50
--

Removing existing destination folder '/mnt/algo/sct_5.3.0/data/PAM50'
Trying URL: https://github.com/sct-data/PAM50/releases/download/r20201104/PAM50-r20201104.zip
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream
Link download error, trying next mirror (error was: HTTPSConnectionPool(host='github-releases.githubusercontent.com', port=443): Max retries exceeded with url: /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)))

Hi @Stefan_Radonjic ,

Thank you for the update. Could you try that command on your remote machine with the “-v 2” suggestion from earlier? The full output log is helpful for identifying exactly what went wrong.

Kind regards,
Joshua