How to use the updated sct_propseg (v5.3.0) for CentOS/RHEL 6?

Thank you for the kind words. :smile:

Truthfully, it’s difficult to speculate given that this is quite “uncharted territory” here… :sweat_smile:

After some searching, the “sct_download_data” issues may be due to an outdated version of OpenSSL. On your work machine, could you please run the following command and share the output?

openssl version

Kind regards,
Joshua

Hahaha, I understand :smiley:

Here it is: OpenSSL 1.1.1.h Sep 2020

Thank you!

Could you please try running the following command before using sct_download_data?

export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt

For transparency, my suggestion is based off of this recommendation and this information for RHEL 6.

@joshuacwnewton still the same issue :confused:

@joshuacwnewton it seems that data HAS been downloaded.

Well, then I am out of ideas why do we get 3x different results with these 3x approaches that should be the same since sct_propseg did not change from 3.2.3 - 5.3.0 :smiley:

Here is my current understanding for why there are three different results:

  • v3.2.3 with CentOS6 “isct_propseg” binary: Baseline performance (not satisfactory).
  • v5.3.0 with CentOS6 “isct_propseg” binary copied over: Better performance (even though the “isct_propseg” binary is the same) because it gets the improvements from years worth of updates to other parts of SCT.
  • v5.3.0 with the non-CentOS6 version of the “isct_propseg” binary: Best performance, but isn’t compatible with RHEL6 because of GLIBC requries.

Sorry for the trouble here, and thank you for the patience. :sweat_smile:

1 Like

@joshuacwnewton is it possible to download code for SCT v5.3.0 and compile it for CentOS6? :smiley:

With the current “hybrid” setup, you should already have most of SCT v5.3.0 – the only differences here are the binaries (“isct_propseg” and others). And, it’s difficult for me to guarantee that recompiling the binaries would have any effect.

I do have some alternate suggestions, though:

1. Tweaking the parameters of sct_propseg

Because you now have an improved segmentation, it might be a better use of your time to try to tweak the parameters of the original “sct_propseg” command. Our documentation currently has a tutorial for fixing segmentations called: Correcting sct_propseg, so some of the tips there may be of use.

We are also in the process of updating the tutorial parts of our documentation, so if anything is unclear, please ask! We will be happy to guide you accordingly, and incorporate any feedback into the documentation. :slightly_smiling_face:

2. Trying sct_deepseg_sc

sct_propseg was developed in 2015, and in the time since, SCT has also developed an improved spinal cord segmentation method called sct_deepseg_sc. It was developed using PyTorch (not Keras), so it may be possible to use it on your RHEL6 machine.

The corresponding command would be, for example:

sct_deepseg_sc -i {input} -c t2 -ofolder {ofolder}

Kind regards,
Joshua

1 Like

The code is in https://github.com/spinalcordtoolbox/spinalcordtoolbox/tree/master/dev/isct_propseg. You could do something like

git clone https://github.com/spinalcordtoolbox/spinalcordtoolbox/
cd spinalcordtoolbox
cd dev/isct_propseg
cmake .
cmake --build .

However we haven’t tested this process in years and we don’t know stable this is. Fixing that up is an ongoing project. If you do manage to make this work, you could help us out a lot by sharing your experiences with us on Github, it might accelerate making this more reliable for other upgraders in the future!

We dropped CentOS 6 support when it was officially dropped 8 months ago, so we can’t really vouch for the quality of any results obtained if you do manage to get it running there.

You might have better luck with a virtual machine. You can

yum install qemu-kvm

or maybe virtualbox (with this http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo)

and then install Ubuntu 20.04 or newer in that, and then install SCT in that.

Hi @anon33887296 @joshuacwnewton! I created a Docker image with CentOS6 where I am trying to recompile SCT’s propseg (as @anon33887296 described) with GLIBC 2.12. Athough, I am running into some issues:

mnt/algo/spinalcordtoolbox-5.3.0/dev/isct_propseg/Image3D.cpp:34:25: error: OrientImage.h: No such file or directory
/mnt/algo/spinalcordtoolbox-5.3.0/dev/isct_propseg/Image3D.cpp: In member function ‘void Image3D::TransformMeshToBinaryImage(Mesh*, std::string, OrientationType, bool, bool, CVector3*, CVector3*, CVector3*, CVector3*)’:
/mnt/algo/spinalcordtoolbox-5.3.0/dev/isct_propseg/Image3D.cpp:435: error: ‘OrientImage’ was not declared in this scope
/mnt/algo/spinalcordtoolbox-5.3.0/dev/isct_propseg/Image3D.cpp:435: error: expected primary-expression before ‘>’ token
/mnt/algo/spinalcordtoolbox-5.3.0/dev/isct_propseg/Image3D.cpp:435: error: ‘orientationFilter’ was not declared in this scope
gmake[2]: *** [CMakeFiles/isct_propseg.dir/Image3D.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/isct_propseg.dir/all] Error 2
gmake: *** [all] Error 2

This occurs when I try to execute cmake --build .

To come clean here, at this point you’re as educated about how to compile isct_propseg as any of us. The person who wrote it hasn’t been actively part of the team for some years now.

If you can figure it out that would be a helpful contribution and we would certainly merge updated build instructions into SCT. I think it requires building all of ITK too, first, which is a fairly large project that is difficult in its own way.

At this point, installing a virtual machine on that server is going to be as much trouble as figuring out how to build it. You can still benefit from all the CPUs, so long as you’re using qemu-kvm, with hardware-assisted virtualization.

Or, is there a reason your IT team can’t upgrade the server itself? RHEL6 ended 8 months ago: https://endoflife.software/operating-systems/linux/red-hat-enterprise-linux-rhel. SCT is probably not the only software your team is having trouble with on that server.

@anon33887296 Thank you for all the help! I think I am pretty close on rebuilding sct binaries, just need to know which ITK and VTK versions is SCT 5.3.0 depending on. Do you have that information by any chance?

Also, after I finish my work, and if I succeed, I will document it and send you in the details in case it proves useful to you guys.

1 Like

It looks like isct_propseg was compiled statically, copying ITK, VTK, and hdf5lib (and maybe others) into itself, because this doesn’t list them:

[spinalcordtoolbox]$ ldd bin/isct_propseg 
	linux-vdso.so.1 (0x00007ffe68f01000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007ff6ea4df000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ff6ea4be000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ff6ea4b7000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ff6ea2a1000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ff6ea286000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007ff6ea0ba000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff6ea65c000)

I found a clue here: https://github.com/spinalcordtoolbox/spinalcordtoolbox/blob/a685d7f0d8032f9df685a08d22947947ac416e71/dev/isct_propseg/CMakeLists.txt#L10-L12

set(CMAKE_INCLUDE_PATH ${ITK_DIR}/include/ITK-4.5)
set(CMAKE_INCLUDE_PATH ${VTK_DIR}/include/vtk-6.1)
set(CMAKE_INCLUDE_PATH ${ITK_DIR}/include/ITK-4.7)

but a bit later it just says https://github.com/spinalcordtoolbox/spinalcordtoolbox/blob/a685d7f0d8032f9df685a08d22947947ac416e71/dev/isct_propseg/CMakeLists.txt#L46-L50


FIND_PACKAGE(ITK REQUIRED)
INCLUDE(${ITK_USE_FILE})

FIND_PACKAGE(VTK REQUIRED)
INCLUDE(${VTK_USE_FILE})

find_package has an optional second version parameter. But this cmake file doesn’t specify it, which means either:

  • this code is supposed to be compatible with every ITK and VTK version or
  • we need to know what version was last used, because that’s probably all they were testing with

I had an insight and tried strings, which found:

[spinalcordtoolbox]$ strings -n 10 bin/isct_propseg  | egrep "*tk version" 
itk version 4.6.0
Running itk version :
vtk version 6.1.0
Running vtk version :

This partially conflicts above: CMakeLists.txt says ITK 4.5 or 4.7, not 4.6. Maybe whoever compiled isct_propseg last didn’t commit their changes to the file? :confused:

Thanks a lot for diving in to this. It’s work we haven’t made time for over here. Does this help get you further? It seems like you’re on the right track.

Hi @anon33887296! I actually found out about all of the mentioned above and managed to finalize SCT 5.3.0 compilation on CentOS 6 with GLIBC 2.12.! I will share documents with you as soon as I write them.

Thing that bugs me is that segmentation is NOT as good as the one with ORIGINAL SC 5.3.0. Here is what I mean:

Left is original SCT 5.3.0 while on the right is SCT 5.3.0 with isct_propseg binary compiled on CentOS6 with GLIBC 2.12.

I am not quite sure what could be causing this issue. Perhaps GLIBC itself? What do you think?

I wouldn’t know where to start. I might try to trace every single variable in both versions of the program and then run diff -u to find where they start to diverge. But you can’t insert debug prints into the pre-compiled version, so you’re stuck. Maybe you could script it with gdb? There’s probably some reverse engineering tool out there that I’m unaware of that can help.

Do you get the same results if you compile isct_propseg on your macOS/Ubuntu desktop?

You might try running our test suite. Maybe something will stand out. This will let you run them:

. python/bin/activate
conda activate venv_sct
pytest -v

Hi guys!

Sadly, still no luck with the complete recompilation. But I will not give up.

Should we close this for now and reopen if anything new pops up, or?

1 Like

That sounds reasonable to me!

I have edited the title and marked one of the comments as a solution, just so that anyone coming to this thread in the future will be able to understand the situation a little easier.

Thank you for your patience throughout this thread. My sincere apologies that we are unable to fully support RHEL6, but I hope you are able to find a solution that works for you.

Kind regards,
Joshua

tagging @joshuacwnewton’s previous message as “solution” just to help our internal management/triage of issues