Where is the C++ source code for the binaries?

siavashk · September 1, 2023, 5:43pm

Problem description:
I am looking for the C++ source code for the binaries under the $SCT_DIR/$BIN_DIR. These folders are defined under the install_sct script and consist of isct_ComposeMultiTransform, isct_antsApplyTransforms, etc.

I am considering a project that adds pybinding to the source code so that relevant code can be called seamlessly from python.

I did some digging and found some of the source code here. Where is everything else?

Commands and terminal output:
NA
System information.
NA
File upload.
NA

joshuacwnewton · September 1, 2023, 5:58pm

Hi @siavashk,

We have an internal developer page that describes where the binaries are sourced from: Packaging: How to compile C binaries (ANTs, isct) · spinalcordtoolbox/spinalcordtoolbox Wiki · GitHub

To summarize that page:

The source code for tools built internally (i.e. from the dev folder) have been migrated to a public archive: GitHub - spinalcordtoolbox/spinalcordtoolbox-dev: A backup of the `dev/` folder from the Spinal Cord Toolbox repository.
antsComposeMultiTransform and antsApplyTransforms come from the ANTs library. We open-source our build workflows, which should hopefully make the process easy for you to reproduce.
The stitching binary comes from biomedia-mira. Again, we have open-sourced our build workflow to make things easier to reproduce.
We do not currently have access to the source code for spine_detect and spine_train_svm, however we are in the process of reaching out to the original developer as per Access to source code of optic functions · Issue #3438 · spinalcordtoolbox/spinalcordtoolbox · GitHub.

Hope this helps,
Joshua

siavashk · September 1, 2023, 6:08pm

This is a great answer. Thank you.

My intention is to opensource my development and have a reciprocal relationship with the community. We will also comply with the license agreement for each piece of source code that we use.

The project is not green lit yet as we are still figuring out if the effort is worth it.

joshuacwnewton · September 1, 2023, 6:10pm

Thank you for the additional context! I am very happy to read your intentions, and I would be more than happy to help in any way that I can to foster a working relationship and comply with the relevant licenses.

Just let me know if you have any additional questions or concerns.

Kind regards,
Joshua

siavashk · September 5, 2023, 9:03pm

I am having a hard time finding the alglib used for isct_propseg. Would you mind telling me where I can find this? The current version of the official release of alglib does not have some of the header files used in this project.

github.com

spinalcordtoolbox/spinalcordtoolbox-dev/blob/master/dev/isct_propseg/util/MatrixNxM.h#L16


      
          * Description: Definition et Implementation de la classe generique Matrice

          ****************************************************************************/

          

          #ifndef _MATRICE_H_

          #define _MATRICE_H_

          

          #include <iostream>

          #include <cmath>

          

          

          #include <alglib/ap.h>

          

          class Matrice

          {

          public:

          	Matrice();

          	Matrice(unsigned int lignes, unsigned int colonnes);

          	Matrice(ap::real_2d_array m, unsigned int lignes, unsigned int colonnes, bool newMatrice=false);

          	Matrice(const Matrice& uneMatrice);

          	~Matrice();

joshuacwnewton · September 5, 2023, 10:13pm

Hi @siavashk,

Just as a quick note before I answer: I joined the NeuroPoly team in 2020, and sct_propseg was developed in 2012-2013. So, my initial answer won’t be based on firsthand experience, but I will still try my best to answer.

I’ve tried building sct_propseg once before, specifically when adding native support for Windows (#3694). And, like you, I also ran into the fact that some header files (e.g. “blas.h”) appear to missing from the Free Edition of ALGLIB. Regrettably, efforts stalled and were put on the backburner at the time.

That said, you’ve given me reason to dig a bit further. I presume we want the source closest to when sct_propseg was developed (March 2012 - August 2013). Looking at ALGLIB’s news archive, that could be anywhere from v3.5.0 to v3.8.0).

Of course, as you’ve probably noticed, ALGLIB seems to have no available download links for past versions. (Why would they do this!! Grr.) That said, I tried manually editing the version number in the current download link: https://www.alglib.net/translator/re/alglib-4.00.0.cpp.gpl.zip. And, luckily, they still work? But only until https://www.alglib.net/translator/re/alglib-3.9.0.cpp.gpl.zip. (No dice for 3.8.0 and below…) And, unfortunately, 3.9.0 does not seem to contain blas.h either.

That said, there is always GitHub! Curiously, I found a copy of 3.4.0 that does not contain blas.h. This suggests sct_propseg possibly relied on an even earlier version (or that blas.h is not included in the free version). Despite this, there are multiple redistributed copies of ALGLIB that do contain blas.h (example). From what I can tell, these copies all seem to contain a copyright of 2005-2007 – This would suggest that a much earlier copy of ALGLIB was used, potentially even pre-3.0.0. In that case, we may actually want the official “Legacy release” of ALGLIB from the very bottom of this page:

v2.6.0 (MPFR) does indeed contain blas.h. So… maybe try that for now, and see how it goes?

Best wishes,
Joshua

siavashk · September 5, 2023, 10:20pm

Thank you very much for your detailed answers. I am coding in the middle of meetings so sorry for brevity in my questions.

siavashk · September 6, 2023, 6:35pm

Hi @joshuacwnewton I just wanted to let you know that isct_propseg does not compile with legacy alglib 2.6. The issue is here.

Basically, the template function rmatrixsvd takes a template_2d_array< amp::ampf<Precision> > as its first argument but the member variable is of type template_2d_array<T,aligned=True>.

I do not think you can recreate your binaries easily. If you can dig on your side in a build instance for a proper version of alglib it would be helpful. In the mean time I can try to refactor the code so that the build passes.

joshuacwnewton · September 6, 2023, 6:40pm

I will definitely see what I can do.

Also, thank you so much for going above and beyond with your investigations. This source code has laid dormant for so many years now, so I very much appreciate having another set of eyes on it. It would be lovely to solve issue #3694 once and for all.

Warm regards,
Joshua

siavashk · September 7, 2023, 6:01pm

Hi @joshuacwnewton, I refactored the code and was able to build isct_propseg.
The project now supports VTK 9.3, ITK 5.4 and Alglib 4.0.0.

Please see the screenshot below. I was also able to run the executable with --help and print the help.
Do you have a sample input / output that I can run with this executable to verify correctness?

joshuacwnewton · September 7, 2023, 6:07pm

This is wonderful news! Thank you so much for your effort. I would love to incorporate your changes into the mainline SCT repo, if possible.

We have a set of tests in our test suite for sct_propseg. These tests operate on the sct_testing_data sample dataset.

Though, I should note that this test presumes:

A) SCT has been installed
B) The isct_propseg binary is present in the $SCT_DIR/bin directory

This is because the CLI script sct_propseg wraps the internal binary isct_propseg and performs some additional Python-based pre- and post-processing steps.

Kind regards,
Joshua

siavashk · September 7, 2023, 6:22pm

I have pushed my development to a public fork of spinalcordtoolbox-dev. You can find it under the pysct repository. If you want I can add you as a collaborator to the project.

siavashk · September 7, 2023, 7:05pm

@joshuacwnewton it seems like the refactored isct produces a correct segmentation. This was generated with isct_propseg -i t2.nii.gz -t t2. I got t2.nii.gz from here

siavashk · September 13, 2023, 11:22pm

@joshuacwnewton I updated our Github conversation with information about my development. I am now looking into incorporating this tutorial into my fork. If I understand correctly, sct_label_vertebrae uses isct_spine_detect under the hood but I could not find the source code for it.

I looked at this conversation that you mentioned earlier and it seems that you guys got access to it at some point. Can I have access to this? As before, everything that I build is accessible to you and your group.

siavashk · September 14, 2023, 12:08am

@joshuacwnewton also I could not find the source code for isct_ComposeMultiTransform. There seems to be a talk about antsComposeMultiTransform but I did not find isct_ComposeMultiTransform. Are you guys renaming antsComposeMultiTransform to isct_ComposeMultiTransform?

joshuacwnewton · September 18, 2023, 4:26pm

My apologies for the late reply. I was out of office for much of last week.

My apologies, but as per the agreement with our collaborator sdika, we are obliged to keep the source code private to only the SCT team.

The full text of the license agreement provided with the ctrDetect binaries (via sdika’s website [1], [2]) is as follows:

By downloading, copying, installing or using the software you agree to 
this license. If you do not agree to this license, do not download, install,
copy or use the software.


                          License Agreement

Copyright (C) 2017, CNRS Institute, all rights reserved.

Redistribution and use in binary forms, are permitted provided that the 
following conditions are met:

  * Redistributions in binary form must reproduce the above copyright notice,
    this list of conditions and the following disclaimer in the documentation
    and/or other materials provided with the distribution.

  * Neither the names of the copyright holders nor the names of the contributors
    may be used to endorse or promote products derived from this software
    without specific prior written permission.

This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are disclaimed.
In no event shall copyright holders or contributors be liable for any direct,
indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.

Precisely. (See here.)

siavashk · September 18, 2023, 4:37pm

@joshuacwnewton with regards to isct_spine_detect, it is OK. I just wanted to know the bounds of reproducibility / open-source nature of the spinal cord toolbox.

jcohenadad · September 18, 2023, 4:50pm

FWIW, isct_spine_detect will soon(ish) be replaced by deep learning methods

siavashk · September 18, 2023, 4:56pm

@jcohenadad we already have a deep learning approach that was presented in ISMRM 2023.
ISMRM23 - ML/AI for General Image Analysis & Post-Processing II (poster 4909, item 143).

siavashk · September 18, 2023, 4:59pm

@jcohenadad I was going to replace the isct_spine_detect portion of the workflow with our own eventually but in the first iteration I was going to keep everything constant and just remove the python - subprocess calls.