How to use the updated sct_propseg (v5.3.0) for CentOS/RHEL 6?

Stefan_Radonjic · June 4, 2021, 11:06am

@jcohenadad Here is what I did:

Installed SCT_5.3.0 by COMMENTING OUT last couple of lines in install_sct which correspond to validation of dependencies. Problem is that Keras was not successfully installed due to wrong GLIBC version (which as we discussed I cannot change). Here is the error I was getting:

MicrosoftTeams-image2533×655 45.5 KB

As mentioned, I fixed this error by commenting out last couple of lines in install_sct because I dont think I need Keras in order to execute sct_propseg or sct_label_vertabrae (only binaries I need)

Here is the issue. When I run sct_propseg on machine where I installed SCT_5.3.0 as described above, I run into this error:

--
Spinal Cord Toolbox (5.3.0)

sct_propseg -i /opt/proc_data/10008886_280521_V-2805_22664208/Output/analysisFolder/stiched_spine.nii.gz -c t2 -CSF -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION/stiched_spin$
--

Creating temporary folder (/tmp/sct-20210604033426.295974-label_vertebrae-8_lthurk)
Creating temporary folder (/tmp/sct-20210604033426.523197-propseg-wko3ao_l)
Creating temporary folder (/tmp/sct-20210604033427.024734-6h35v9i4)
Remove temporary files...
rm -rf /tmp/sct-20210604033427.024734-6h35v9i4
/mnt/algo/sct_5.3.0/bin/isct_propseg -t t2 -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION -verbose -mesh -centerline-binary -CSF -centerline-coord -cross -init-tube -low-resol$
Automatic cord detection failed. Please initialize using -init-centerline or -init-mask (see help)

stderror is

While on the other machine (my local) where I installed SCT_5.3.0 without any issues (correct GLBC version), I had no errors what so ever.

What could be causing this?

joshuacwnewton · June 4, 2021, 4:25pm

Hi @Stefan_Radonjic,

It looks like there are 2 different problems in this thread: Fixing the v3.2.3 installation, and fixing the v5.3.0 installation. Before tackling either of the problems, I just want to clarify this part here:

What differences did you find between the v3.2.3 propseg and v5.3.0 propseg?
What was the exact “sct_propseg” command used? (I ask because in the pasted error log, the full command was cut off.)

Thanks for your patience.
-Joshua

Stefan_Radonjic · June 6, 2021, 3:27pm

Hi @joshuacwnewton! The correct segmentation is produced with SCT 5.3 while the incorrect one is produced with SCT 3.2.3. I’ve used same data for both and -CSF flag as well.

joshuacwnewton · June 6, 2021, 4:34pm

Thank you for elaborating! In that case, we can focus on just the 5.3.0 installation, then.

If possible, could you please modify your “sct_propseg” command as follows:

# sample command just to demonstrate

# before
sct_propseg -i t2.nii.gz -c t2 -CSF

# after
sct_propseg -i t2.nii.gz -c t2 -CSF -v 2 > propseg_output_log.txt

To explain:

“ -v 2” will give us additional debugging information
" > propseg_output_log.txt" will put the terminal output into a text file for you (so you don’t have to copy and paste to the forum).

Then, could you rerun propseg and share with us the “propseg_output_log.txt” file that is created?

Kind regards,
Joshua

Stefan_Radonjic · June 6, 2021, 6:50pm

HI @joshuacwnewton and thats for quick reply!

Although, I think you misunderstood what I am trying to achieve. I need success rate of SCT’s 5.3.0 PropSeg in 3.2.3 version. Meaning, I would like to replicate changes in PropSeg that are in SC5 5.3.0 in the SCT 3.2.3 (If you understand what I mean).

Here is the file generated from SCT 3.2.3. which is getting us INCORRECT segmentation. I would like to have SCT 3.2.3 to give a CORRET one isntead.propseg-log.txt (5.4 KB)

I am currently trying to edit sct_propseg.py from SCT 3.2.3 to mimic sct_propseg.py from SCT 5.3.0.

joshuacwnewton · June 6, 2021, 7:30pm

Thank you for the clarification. I understand what you mean.

I’ve looked into the possibility, but it is not straightforward to do this. Specifically, it turns out that the core algorithm for sct_propseg hasn’t actually changed between v3.2.3 and v5.3.0.

Binaries that are downloaded with v3.2.3:

Binaries that are downloaded with v5.3.0:

This means that the improvements you’ve noticed are due to updates to other parts of SCT. (For example, in 2019 there was a major refactoring of the OptiC centerline detection algorithm. OptiC is used in sct_propseg, so this may be one of the sources of improvement.)

Because of the scope of the changes that have been made over the years, it would be a nontrivial effort to stitch the improvements from v5.3.0 into v3.2.3.

You mention in an earlier comment that you were able to get v5.3.0 into a partially working state:

So, if we instead got “sct_propseg” to work for v5.3.0, would that be sufficient?

Kind regards,
Joshua

Stefan_Radonjic · June 6, 2021, 7:42pm

@joshuacwnewton oh, I understand!

But, yes! If we could somehow make sct_5.3.0 PropSeg to work that would be great! But, I cannot install it due to GLIBC version problem (which I cannot touch because it will mess with entire system). Any suggestions on how to do this?

I will generate logs from installation and send them in!

joshuacwnewton · June 6, 2021, 7:49pm

Of course! I think we might be able to work around this limitation.

There was an earlier assessment you made that I agree with: sct_propseg and sct_label_vertebrae do not rely on Keras. So, in theory, they should be able to be run in isolation, even with the older GLIBC version.

That’s why I made this earlier suggestion. If we look at the debugging information for sct_propseg, we might be able to determine why it was that it failed.

Stefan_Radonjic · June 6, 2021, 7:56pm

@joshuacwnewton oh, sorry, I did not understood what you meant by your earlier suggestion. GIve me one second to generate you logs.

Stefan_Radonjic · June 6, 2021, 8:01pm

@joshuacwnewton here you go, and really appreciate your help! Hope that this is possible. propseg_output_log.txt (143.2 KB)

joshuacwnewton · June 6, 2021, 8:07pm

Thank you!

This is interesting… When I look at this output log, I don’t see the error that was mentioned in your earlier comment, and the output files appear to be created OK:

e[33mFile /home/stefan/sdintegration/pythonPlugin/test/__main__.Test.test_verify/Output/analysisFolder/stiched_spine_seg.nii.gz already exists. Will overwrite it.e[0m
e[33mFile /home/stefan/sdintegration/pythonPlugin/test/__main__.Test.test_verify/Output/analysisFolder/stiched_spine_centerline.nii.gz already exists. Will overwrite it.e[0m

Could you double-check to see if it’s working OK for you?

Stefan_Radonjic · June 6, 2021, 8:12pm

@joshuacwnewton sorry, a lot happened from that day. Basically, that was my “failed” attempt of patching SCT_3.2.3.

I think we can return to installation of SCT_5.3.0 as you suggested. The output of PropSeg I’ve sent you is from MY LOCAL Machine, where I installed SCT_5.3.0 without issues. When I try to install it on our system, I get the following: sct_error_log.txt (66.4 KB)

Also, I’ve noticed that when downloading data it fails due to some network error.

joshuacwnewton · June 6, 2021, 8:15pm

Ahhh, I see!

The good news is: What you see in that error log is actually just a post-installation check. Meaning that, if you’re able get to that point, SCT has already installed itself.

So, on your remote machine, you should still be able to run the v5.3.0 version of sct_propseg (even with the GLIBC error).

Stefan_Radonjic · June 6, 2021, 8:24pm

@joshuacwnewton I just tried, and I get

--
Spinal Cord Toolbox (5.3.0)

sct_propseg -i /opt/proc_data/10008886_280521_V-2805_22664208/Output/analysisFolder/stiched_spine.nii.gz -c t2 -CSF -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION/stiched_spin$
--

Creating temporary folder (/tmp/sct-20210604033426.295974-label_vertebrae-8_lthurk)
Creating temporary folder (/tmp/sct-20210604033426.523197-propseg-wko3ao_l)
Creating temporary folder (/tmp/sct-20210604033427.024734-6h35v9i4)
Remove temporary files...
rm -rf /tmp/sct-20210604033427.024734-6h35v9i4
/mnt/algo/sct_5.3.0/bin/isct_propseg -t t2 -o /opt/proc_data/10008886_280521_V-2805_22664208/Output/SPINAL_CORD_SEGMENTATION -verbose -mesh -centerline-binary -CSF -centerline-coord -cross -init-tube -low-resol$
Automatic cord detection failed. Please initialize using -init-centerline or -init-mask (see help)

stderror is

Could this be because while I was INSTALLING sct_5.3.0 it COULD NOT download the data?

Stefan_Radonjic · June 6, 2021, 8:26pm

Here is an example:

sct_download_data -d PAM50 -o /mnt/algo/sct_5.3.0/data/PAM50
--

Removing existing destination folder '/mnt/algo/sct_5.3.0/data/PAM50'
Trying URL: https://github.com/sct-data/PAM50/releases/download/r20201104/PAM50-r20201104.zip
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream
Link download error, trying next mirror (error was: HTTPSConnectionPool(host='github-releases.githubusercontent.com', port=443): Max retries exceeded with url: /269508010/0a8f1980-1e1c-11eb-84b0-fa146c143d89?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210606T200404Z&X-Amz-Expires=300&X-Amz-Signature=1efc1d9363803537dee13f262c09a1c905fc7d32e01bb17a5aaca61d4cfafebf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=269508010&response-content-disposition=attachment%3B%20filename%3DPAM50-r20201104.zip&response-content-type=application%2Foctet-stream (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)))

joshuacwnewton · June 6, 2021, 8:26pm

Hi @Stefan_Radonjic ,

Thank you for the update. Could you try that command on your remote machine with the “-v 2” suggestion from earlier? The full output log is helpful for identifying exactly what went wrong.

Kind regards,
Joshua

Stefan_Radonjic · June 6, 2021, 8:32pm

propseg_log_5.3.0.txt (1.8 KB)

@joshuacwnewton ah, seems to be GLIBC issue.

joshuacwnewton · June 6, 2021, 8:54pm

Indeed…

This is a bit confusing to me. The error for v5.3.0 occurred inside “isct_propseg”, but that should be identical to the “isct_propseg” that was inside v3.2.3. (See the screenshots in my earlier comment.)

In other words, if the GLIBC on your work machine is causing issues for v5.3.0, then in theory you would experience this error on any version of SCT (v3.2.3 included)…

Stefan_Radonjic · June 6, 2021, 9:04pm

@joshuacwnewton strange. I think I already sent you SCT 3.2.3 output from the same machine. Btw, here we dont have any GLIBC issues, just INCORRECT segmentation.

Stefan_Radonjic · June 6, 2021, 9:10pm

propseg_log_3.2.3.txt (6.0 KB)

@joshuacwnewton here it is. This is generated by the same machine with the same data, only different version of SCT. In this case, 3.2.3.