Sct_fmri_moco very, very slow

SCT
1

Hi everyone!

I’m trying to update the sct version of my previous pipeline from v3.2.7 to v4.2.2 and am finding in sct_fmri_moco a previously nonexistent bottleneck timewise.

When looping through Z, what before was 1.5s/iter approximately now is 64s/iter

So when running:

sct_fmri_moco -i fmri.nii.gz -m Mask/mask_fmri.nii.gz -g 1 -r 0 -param poly=1,smooth=2 -ofolder Moco

I now get stuck in:

-------------------------------------------------------------------------------
  Estimating motion...
-------------------------------------------------------------------------------

Input parameters:
  Input file ............fmri_averaged_groups.nii
  Reference file ........fmri_mean_0.nii.gz
  Polynomial degree .....1
      Smoothing kernel ......2
      Gradient step .........1
      Metric ................MeanSquares
      Sampling ..............0.2
      Todo ..................estimate_and_apply
      Mask  .................Mask/mask_fmri.nii.gz
      Output mat folder .....mat_groups

Data dimensions:
  144 x 44 x 32 x 164

Copy file_target to a temporary file...
sct_convert -i fmri_mean_0.nii.gz -o target.nii.gz
sct_convert -i /Mask/mask_fmri.nii.gz -o mask.nii

Register. Loop across Z (note: there is only one Z if orientation is axial
Z=0/0:   5%|#7                              | 9/164 [10:08<2:57:56, 68.88s/iter]

when previously I had something more like 1.5s/iter.

I noticed there is an open issue which may be related here.

I imagine I have to adapt in some way to get back to the previous more reasonable time/iteration.

Thanks!

Sergio

2

Hi Sergio,

Thank you for the feedback. This is a puzzling problem, and indeed, related to issue #947.

To help us understand the cause of this issue, could you please copy/paste the output of sct_check_dependencies (to know your hardware/OS config).

Thanks!
Julien

3

Hi Julien,

Thanks for answering so promptly!

Here is the output of the sct_check_dependencies:

sct_check_dependencies

--
Spinal Cord Toolbox (4.2.2)

SCT info:
- version: 4.2.2
- path: /home/shernand/Lumbar_Cord/sct_4.2.2
OS: linux (Linux-4.4.0-104-generic-x86_64-with-debian-stretch-sid)
CPU cores: Available: 64, Used by SCT: 64
RAM: MemTotal:       527763148 kB
total        used        free      shared  buff/cache   available
Mem:         515393      246473      265178          15        3741      267307
Swap:         30517       17858       12659
Check Python executable.............................[OK]
  Using bundled python 3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 21:14:29) 
[GCC 7.3.0] at /home/shernand/sct_4.2.2/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if numpy is installed.........................[OK] (1.17.4)
Check if cryptography is installed..................[OK] (2.8)
Check if dipy is installed..........................[OK] (1.0.0)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (2.10.0)
Check if Keras is installed.........................[OK] (2.3.1)
Check if matplotlib is installed....................[OK] (3.1.2)
Check if nibabel is installed.......................[OK] (3.0.0)
Check if pandas is installed........................[OK] (0.25.3)
Check if psutil is installed........................[OK] (5.6.7)
Check if pyqt5 (5.11.3) is installed................[OK]
Check if pytest is installed........................[OK] (5.3.2)
Check if pytest-cov is installed....................[OK] (2.8.1)
Check if raven is installed.........................[OK]
Check if requests is installed......................[OK] (2.22.0)
Check if scipy is installed.........................[OK] (1.4.0)
Check if scikit-image is installed..................[OK] (0.16.2)
Check if scikit-learn is installed..................[OK] (0.22)
Check if tensorflow is installed....................[OK] (2.0.1)
Check if xlrd is installed..........................[OK] (1.2.0)
Check if xlutils is installed.......................[OK]
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.40.2)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3[secure] is installed...............[OK] (1.25.7)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if DISPLAY variable is set....................[OK]
Check if figure can be opened with PyQt.............[OK]

Thanks,

Sergio

4

Hi Sergio,

So, after digging a bit more, it looks like we did find a solution. The fix is not implemented in the master branch yet, but I though maybe you could give it a try to confirm it solves the issue on your end. If it does, we will go ahead and merge the changes on the stable release.

Could you download these binaries: https://osf.io/568y4/

And copy the files in your folder $SCT_DIR/bin (replace the old files).

Then, re-run moco and let us know if it is significantly faster.

Cheers,
Julien

5

Hi Julien!

Thanks a lot!

I get an unfortunate error now, probably related to the gcc version of the machine from my side and the gcc version used to compile the binaries.

sct_utils.RunError: /home/shernand/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /home/shernand/Lumbar_Cord/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration)

The output of ldd --version is:
ldd (Ubuntu GLIBC 2.23-0ubuntu11) 2.23

So I gues I need to update glibc…

I was also getting the errors:

sct_utils.RunError: /home/shernand/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/shernand/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration)
/home/shernand/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /home/shernand/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration)

and fixed them by going to the conda bin of sct at $SCT_DIR/python/bin and installing libgcc by running:

./conda install libgcc

and then pointing the conda lib with LD_LIBRARY_PATH with:
LD_LIBRARY_PATH=$SCT_DIR/python/lib:$LD_LIBRARY_PATH

6

@shernand,

Would you be willing to give this another shot? I have recompiled the binaries again but this time with -static, so they shouldn’t have any library issues: https://osf.io/r59va.

I don’t know if this is a good long-term solution, but it would help us to know if it solves the performance problem without needing special user intervention.

Thanks for your time and your report and I hope you’re keeping safe.
-Nick

7

Dear Nick,

Thanks a lot! The link, https://osf.io/r59va, seems to fail. It looks like the OSF site is down.

I’ll test them as soon as I get the opportunity.

Sergio

8

Oh wow, you’re right: Osf.io down? Current problems and status.

2020-03-27-134842_695x321_scrot
2020-03-27-134842_695x321_scrot695×321 15.6 KB

I was there 20 minutes ago. I’m sure it’s just…maintenance. I’ve uploaded a second copy at https://gist.github.com/kousu/20ed86d2da84a14267b4fcccd07b114a/raw/0f26f850260c69673e8afab642c295ba30da17f0/20200327_sct_static_binaries_linux.tar.gz instead.

9

Hey Nick,

Well the library errors are gone!
But I get this error (and output) instead:

Register. Loop across Z (note: there is only one Z if orientation is axial
Z=0/0:   0%|                                          | 0/164 [00:00<?, ?iter/s]
Traceback (most recent call last):
  File "/home/shernand/sct_4.2.2/scripts/sct_fmri_moco.py", line 453, in <module>
    main()
  File "/home/shernand/sct_4.2.2/scripts/sct_fmri_moco.py", line 213, in main
    fmri_moco(param)
  File "/home/shernand/sct_4.2.2/scripts/sct_fmri_moco.py", line 350, in fmri_moco
    file_mat = moco.moco(param_moco)
  File "/home/shernand/sct_4.2.2/scripts/msct_moco.py", line 152, in moco
    file_data_splitZ_splitT_moco[it], im_mask=input_mask)
  File "/home/shernand/sct_4.2.2/scripts/msct_moco.py", line 276, in register
    status, output = sct.run(cmd, verbose=0, **kw)
  File "/home/shernand/sct_4.2.2/scripts/sct_utils.py", line 311, in run
    raise RunError(output)
sct_utils.RunError

Maybe there are some more files to update?

Thanks in advance!

Sergio

10

I’m looking into this. I’ll have to run some tests with our testing data. Maybe the static compilation broke something subtle about this usage.

11

I reproduced the crash!

In an empty Ubuntu container, I installed a fresh copy:

sudo apt-get install -y git wget build-essential
git clone --depth 1 https://github.com/neuropoly/spinalcordtoolbox
cd spinalcordtoolbox
./install_sct

I didn’t have an input file or a mask to test with, but I followed ./batch_processing.sh to get one:

mkdir forum365; cd forum365
sct_download_data -d sct_example_data
sct_maths -i sct_example_data/fmri/fmri.nii.gz -mean t -o fmri_mean.nii.gz
sct_get_centerline -i fmri_mean.nii.gz -c t2s
sct_create_mask -i fmri_mean.nii.gz -p centerline,fmri_mean_centerline.nii.gz -size 35mm

This gives mask_fmri_mean.nii.gz (and other stuff that we do not need here).

The next step in batch_processing.sh is sct_crop_image but instead I’ll try your command line:

$ sct_fmri_moco -i sct_example_data/fmri/fmri.nii.gz -m mask_fmri_mean.nii.gz -g 1 -r 0 -param poly=1,smooth=2 -ofolder Moco

--
Spinal Cord Toolbox (4.2.2)


Input parameters:
  input file ............sct_example_data/fmri/fmri.nii.gz

Create temporary folder (/tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw)...

Copying input data to tmp folder and convert to nii...
sct_convert -i /home/ubuntu/forum365/sct_example_data/fmri/fmri.nii.gz -o /tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw/fmri.nii

Get dimensions of data...
  128 x 28 x 25 x 50

Data orientation: RPI
  Treated as axial

Split along T dimension...
Merge within groups:   0%|                             | 0/50 [00:00<?, ?iter/s]sct_convert -i fmri_0.nii -o fmri_mean_0.nii.gz
sct_convert -i fmri_1.nii -o fmri_mean_1.nii.gz
sct_convert -i fmri_2.nii -o fmri_mean_2.nii.gz
sct_convert -i fmri_3.nii -o fmri_mean_3.nii.gz
sct_convert -i fmri_4.nii -o fmri_mean_4.nii.gz
sct_convert -i fmri_5.nii -o fmri_mean_5.nii.gz
sct_convert -i fmri_6.nii -o fmri_mean_6.nii.gz
sct_convert -i fmri_7.nii -o fmri_mean_7.nii.gz
Merge within groups:  16%|###3                 | 8/50 [00:00<00:00, 76.23iter/s]sct_convert -i fmri_8.nii -o fmri_mean_8.nii.gz
sct_convert -i fmri_9.nii -o fmri_mean_9.nii.gz
sct_convert -i fmri_10.nii -o fmri_mean_10.nii.gz
sct_convert -i fmri_11.nii -o fmri_mean_11.nii.gz
sct_convert -i fmri_12.nii -o fmri_mean_12.nii.gz
sct_convert -i fmri_13.nii -o fmri_mean_13.nii.gz
sct_convert -i fmri_14.nii -o fmri_mean_14.nii.gz
sct_convert -i fmri_15.nii -o fmri_mean_15.nii.gz
Merge within groups:  32%|######4             | 16/50 [00:00<00:00, 76.46iter/s]sct_convert -i fmri_16.nii -o fmri_mean_16.nii.gz
sct_convert -i fmri_17.nii -o fmri_mean_17.nii.gz
sct_convert -i fmri_18.nii -o fmri_mean_18.nii.gz
sct_convert -i fmri_19.nii -o fmri_mean_19.nii.gz
sct_convert -i fmri_20.nii -o fmri_mean_20.nii.gz
sct_convert -i fmri_21.nii -o fmri_mean_21.nii.gz
sct_convert -i fmri_22.nii -o fmri_mean_22.nii.gz
sct_convert -i fmri_23.nii -o fmri_mean_23.nii.gz
Merge within groups:  48%|#########6          | 24/50 [00:00<00:00, 76.98iter/s]sct_convert -i fmri_24.nii -o fmri_mean_24.nii.gz
sct_convert -i fmri_25.nii -o fmri_mean_25.nii.gz
sct_convert -i fmri_26.nii -o fmri_mean_26.nii.gz
sct_convert -i fmri_27.nii -o fmri_mean_27.nii.gz
sct_convert -i fmri_28.nii -o fmri_mean_28.nii.gz
sct_convert -i fmri_29.nii -o fmri_mean_29.nii.gz
sct_convert -i fmri_30.nii -o fmri_mean_30.nii.gz
sct_convert -i fmri_31.nii -o fmri_mean_31.nii.gz
Merge within groups:  64%|############8       | 32/50 [00:00<00:00, 77.01iter/s]sct_convert -i fmri_32.nii -o fmri_mean_32.nii.gz
sct_convert -i fmri_33.nii -o fmri_mean_33.nii.gz
sct_convert -i fmri_34.nii -o fmri_mean_34.nii.gz
sct_convert -i fmri_35.nii -o fmri_mean_35.nii.gz
sct_convert -i fmri_36.nii -o fmri_mean_36.nii.gz
sct_convert -i fmri_37.nii -o fmri_mean_37.nii.gz
sct_convert -i fmri_38.nii -o fmri_mean_38.nii.gz
sct_convert -i fmri_39.nii -o fmri_mean_39.nii.gz
Merge within groups:  80%|################    | 40/50 [00:00<00:00, 77.41iter/s]sct_convert -i fmri_40.nii -o fmri_mean_40.nii.gz
sct_convert -i fmri_41.nii -o fmri_mean_41.nii.gz
sct_convert -i fmri_42.nii -o fmri_mean_42.nii.gz
sct_convert -i fmri_43.nii -o fmri_mean_43.nii.gz
sct_convert -i fmri_44.nii -o fmri_mean_44.nii.gz
sct_convert -i fmri_45.nii -o fmri_mean_45.nii.gz
sct_convert -i fmri_46.nii -o fmri_mean_46.nii.gz
sct_convert -i fmri_47.nii -o fmri_mean_47.nii.gz
Merge within groups:  96%|###################2| 48/50 [00:00<00:00, 77.62iter/s]sct_convert -i fmri_48.nii -o fmri_mean_48.nii.gz
sct_convert -i fmri_49.nii -o fmri_mean_49.nii.gz
Merge within groups: 100%|####################| 50/50 [00:00<00:00, 77.43iter/s]

Merging volumes...

-------------------------------------------------------------------------------
  Estimating motion...
-------------------------------------------------------------------------------

Input parameters:
  Input file ............fmri_averaged_groups.nii
  Reference file ........fmri_mean_0.nii.gz
  Polynomial degree .....1
  Smoothing kernel ......2
  Gradient step .........1
  Metric ................MeanSquares
  Sampling ..............0.2
  Todo ..................estimate_and_apply
  Mask  ................./home/ubuntu/forum365/mask_fmri_mean.nii.gz
  Output mat folder .....mat_groups

Data dimensions:
  128 x 28 x 25 x 50

Copy file_target to a temporary file...
sct_convert -i fmri_mean_0.nii.gz -o target.nii.gz
sct_convert -i /home/ubuntu/forum365/mask_fmri_mean.nii.gz -o mask.nii

Register. Loop across Z (note: there is only one Z if orientation is axial
Z=0/0: 100%|##################################| 50/50 [00:52<00:00,  1.04s/iter]
mv fmri_averaged_groups_moco.nii fmri_moco.nii
WARNING: File /tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw/fmri_moco.nii already exists. Will overwrite it.

Averaging data...

Generate output files...
sct_convert -i /tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw/fmri_moco.nii -o Moco/fmri_moco.nii.gz
  File created: Moco/fmri_moco.nii.gz
sct_convert -i /tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw/fmri_moco_mean.nii -o Moco/fmri_moco_mean.nii.gz
  File created: Moco/fmri_moco_mean.nii.gz
sct_convert -i /tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw/fmri_moco_params_X.nii -o Moco/fmri_moco_params_X.nii.gz
  File created: Moco/fmri_moco_params_X.nii.gz
sct_convert -i /tmp/sct-20200327203622.789997-fmri_moco-u6tqecyw/fmri_moco_params_Y.nii -o Moco/fmri_moco_params_Y.nii.gz
  File created: Moco/fmri_moco_params_Y.nii.gz

Finished! Elapsed time: 56s

Great.

Now if I replace this with my statically linked binaries:

curl -LOJ https://osf.io/r59va/download
tar -zxvf 20200327_sct_static_binaries_linux.tar.gz
mv isct_* $SCT_DIR/bin/

and try again:

$ sct_fmri_moco -i sct_example_data/fmri/fmri.nii.gz -m mask_fmri_mean.nii.gz -g 1 -r 0 -param poly=1,smooth=2 -ofolder Moco

--
Spinal Cord Toolbox (4.2.2)

Folder Moco has been created.

Input parameters:
  input file ............sct_example_data/fmri/fmri.nii.gz

Create temporary folder (/tmp/sct-20200328060538.432796-fmri_moco-g85h1pz1)...

Copying input data to tmp folder and convert to nii...
sct_convert -i /home/ubuntu/forum365/sct_example_data/fmri/fmri.nii.gz -o /tmp/sct-20200328060538.432796-fmri_moco-g85h1pz1/fmri.nii

Get dimensions of data...
  128 x 28 x 25 x 50

Data orientation: RPI
  Treated as axial

Split along T dimension...
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0000.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0001.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0002.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0003.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0004.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0005.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0006.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0007.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0008.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0009.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0010.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0011.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0012.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0013.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0014.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0015.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0016.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0017.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0018.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0019.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0020.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0021.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0022.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0023.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0024.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0025.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0026.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0027.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0028.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0029.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0030.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0031.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0032.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0033.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0034.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0035.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0036.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0037.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0038.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0039.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0040.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0041.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0042.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0043.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0044.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0045.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0046.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0047.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0048.nii.gz already exists. Will overwrite it.
WARNING: File /home/ubuntu/forum365/sct_example_data/fmri/fmri_T0049.nii.gz already exists. Will overwrite it.
Merge within groups:   0%|                             | 0/50 [00:00<?, ?iter/s]sct_convert -i fmri_0.nii -o fmri_mean_0.nii.gz
sct_convert -i fmri_1.nii -o fmri_mean_1.nii.gz
sct_convert -i fmri_2.nii -o fmri_mean_2.nii.gz
sct_convert -i fmri_3.nii -o fmri_mean_3.nii.gz
sct_convert -i fmri_4.nii -o fmri_mean_4.nii.gz
sct_convert -i fmri_5.nii -o fmri_mean_5.nii.gz
sct_convert -i fmri_6.nii -o fmri_mean_6.nii.gz
sct_convert -i fmri_7.nii -o fmri_mean_7.nii.gz
Merge within groups:  16%|###3                 | 8/50 [00:00<00:00, 76.06iter/s]sct_convert -i fmri_8.nii -o fmri_mean_8.nii.gz
sct_convert -i fmri_9.nii -o fmri_mean_9.nii.gz
sct_convert -i fmri_10.nii -o fmri_mean_10.nii.gz
sct_convert -i fmri_11.nii -o fmri_mean_11.nii.gz
sct_convert -i fmri_12.nii -o fmri_mean_12.nii.gz
sct_convert -i fmri_13.nii -o fmri_mean_13.nii.gz
sct_convert -i fmri_14.nii -o fmri_mean_14.nii.gz
sct_convert -i fmri_15.nii -o fmri_mean_15.nii.gz
Merge within groups:  32%|######4             | 16/50 [00:00<00:00, 74.70iter/s]sct_convert -i fmri_16.nii -o fmri_mean_16.nii.gz
sct_convert -i fmri_17.nii -o fmri_mean_17.nii.gz
sct_convert -i fmri_18.nii -o fmri_mean_18.nii.gz
sct_convert -i fmri_19.nii -o fmri_mean_19.nii.gz
sct_convert -i fmri_20.nii -o fmri_mean_20.nii.gz
sct_convert -i fmri_21.nii -o fmri_mean_21.nii.gz
sct_convert -i fmri_22.nii -o fmri_mean_22.nii.gz
sct_convert -i fmri_23.nii -o fmri_mean_23.nii.gz
Merge within groups:  48%|#########6          | 24/50 [00:00<00:00, 74.01iter/s]sct_convert -i fmri_24.nii -o fmri_mean_24.nii.gz
sct_convert -i fmri_25.nii -o fmri_mean_25.nii.gz
sct_convert -i fmri_26.nii -o fmri_mean_26.nii.gz
sct_convert -i fmri_27.nii -o fmri_mean_27.nii.gz
sct_convert -i fmri_28.nii -o fmri_mean_28.nii.gz
sct_convert -i fmri_29.nii -o fmri_mean_29.nii.gz
sct_convert -i fmri_30.nii -o fmri_mean_30.nii.gz
Merge within groups:  62%|############4       | 31/50 [00:00<00:00, 72.21iter/s]sct_convert -i fmri_31.nii -o fmri_mean_31.nii.gz
sct_convert -i fmri_32.nii -o fmri_mean_32.nii.gz
sct_convert -i fmri_33.nii -o fmri_mean_33.nii.gz
sct_convert -i fmri_34.nii -o fmri_mean_34.nii.gz
sct_convert -i fmri_35.nii -o fmri_mean_35.nii.gz
sct_convert -i fmri_36.nii -o fmri_mean_36.nii.gz
sct_convert -i fmri_37.nii -o fmri_mean_37.nii.gz
sct_convert -i fmri_38.nii -o fmri_mean_38.nii.gz
Merge within groups:  78%|###############6    | 39/50 [00:00<00:00, 72.91iter/s]sct_convert -i fmri_39.nii -o fmri_mean_39.nii.gz
sct_convert -i fmri_40.nii -o fmri_mean_40.nii.gz
sct_convert -i fmri_41.nii -o fmri_mean_41.nii.gz
sct_convert -i fmri_42.nii -o fmri_mean_42.nii.gz
sct_convert -i fmri_43.nii -o fmri_mean_43.nii.gz
sct_convert -i fmri_44.nii -o fmri_mean_44.nii.gz
sct_convert -i fmri_45.nii -o fmri_mean_45.nii.gz
sct_convert -i fmri_46.nii -o fmri_mean_46.nii.gz
Merge within groups:  94%|##################7 | 47/50 [00:00<00:00, 73.75iter/s]sct_convert -i fmri_47.nii -o fmri_mean_47.nii.gz
sct_convert -i fmri_48.nii -o fmri_mean_48.nii.gz
sct_convert -i fmri_49.nii -o fmri_mean_49.nii.gz
Merge within groups: 100%|####################| 50/50 [00:00<00:00, 73.27iter/s]

Merging volumes...

-------------------------------------------------------------------------------
  Estimating motion...
-------------------------------------------------------------------------------

Input parameters:
  Input file ............fmri_averaged_groups.nii
  Reference file ........fmri_mean_0.nii.gz
  Polynomial degree .....1
  Smoothing kernel ......2
  Gradient step .........1
  Metric ................MeanSquares
  Sampling ..............0.2
  Todo ..................estimate_and_apply
  Mask  ................./home/ubuntu/forum365/mask_fmri_mean.nii.gz
  Output mat folder .....mat_groups

Data dimensions:
  128 x 28 x 25 x 50

Copy file_target to a temporary file...
sct_convert -i fmri_mean_0.nii.gz -o target.nii.gz
sct_convert -i /home/ubuntu/forum365/mask_fmri_mean.nii.gz -o mask.nii

Register. Loop across Z (note: there is only one Z if orientation is axial
Z=0/0:   0%|                                           | 0/50 [00:00<?, ?iter/s]
Traceback (most recent call last):
  File "/home/ubuntu/sct_4.2.2/scripts/sct_fmri_moco.py", line 453, in <module>
    main()
  File "/home/ubuntu/sct_4.2.2/scripts/sct_fmri_moco.py", line 213, in main
    fmri_moco(param)
  File "/home/ubuntu/sct_4.2.2/scripts/sct_fmri_moco.py", line 350, in fmri_moco
    file_mat = moco.moco(param_moco)
  File "/home/ubuntu/sct_4.2.2/scripts/msct_moco.py", line 152, in moco
    file_data_splitZ_splitT_moco[it], im_mask=input_mask)
  File "/home/ubuntu/sct_4.2.2/scripts/msct_moco.py", line 276, in register
    status, output = sct.run(cmd, verbose=0, **kw)
  File "/home/ubuntu/sct_4.2.2/scripts/sct_utils.py", line 311, in run
    raise RunError(output)
sct_utils.RunError

By tweaking that sct.run() line I was able to find that the actual command that’s crashing is this long isct_antsSliceRegularizedRegistration invocation

$ pushd /tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu; /home/ubuntu/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration --polydegree 1 --transform 'Translation[1]' --metric 'MeanSquares[target.nii.gz,/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/fmri_averaged_groups_T0000.nii,1,4,Regular,0.2]' --iterations 10 --shrinkFactors 1 --smoothingSigmas 2 --verbose 1 --output '[mat_groups/mat.Z0000T0000,/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/fmri_averaged_groups_T0000_moco.nii]' -n Linear --mask /tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/mask.nii; popd
/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu ~/forum365
Segmentation fault (core dumped)
~/forum365

or equivalently

$ /home/ubuntu/sct_4.2.2/bin/isct_antsSliceRegularizedRegistration --polydegree 1 --transform 'Translation[1]' --metric 'MeanSquares[/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/target.nii.gz,/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/fmri_averaged_groups_T0000.nii,1,4,Regular,0.2]' --iterations 10 --shrinkFactors 1 --smoothingSigmas 2 --verbose 1 --output '[mat_groups/mat.Z0000T0000,/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/fmri_averaged_groups_T0000_moco.nii]' -n Linear --mask /tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/mask.nii
Segmentation fault (core dumped)

Trying this with the original isct_antsSliceRegularizedRegistration works:

$ pushd /tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu; /home/ubuntu/isct_bin.bak/isct_antsSliceRegularizedRegistration --polydegree 1 --transform 'Translation[1]' --metric 'MeanSquares[/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/target.nii.gz,/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/fmri_averaged_groups_T0000.nii,1,4,Regular,0.2]' --iterations 10 --shrinkFactors 1 --smoothingSigmas 2 --verbose 1 --output '[mat_groups/mat.Z0000T0000,/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/fmri_averaged_groups_T0000_moco.nii]' -n Linear --mask /tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/mask.nii; popd
/tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu ~/forum365
 setting mask /tmp/sct-20200328060626.256311-fmri_moco-6j8yqdgu/mask.nii
Loop0 polyerr: 0 image-metric 0
 polyx 0 iceptx 0
 polyy 0 icepty 0
Loop1 polyerr: 0 image-metric 0
 polyx 0 iceptx 0
 polyy 0 icepty 0
~/forum365

So it’s probably a memory corruption bug. It’s possible that statically linking exposed a bug, or maybe something else is going on. I’ll try to investigate more tomorrow.

1 Like
12

Hi again,

I don’t think static linking is going to be a good solution for us in the long run. Thanks for helping us experiment with it.

I am going to make a point of building a fully cross-platform set of packages.

In the meantime, if you want to use the latest code, please look at the Docker or VM versions.

If you simply migrate your datasets into a docker container using --mount, they should be able to take advantage of your multicore machine without any further tweaking.

For posterity, the crash I gifted you (sorry about that!) is happening in itk::ProcessObject::UpdateOutputData(), specifically on this assembly instruction:

  edf24d:       e8 ae 0d 12 ff          callq  0 <_nl_current_LC_CTYPE>

You can’t call NULL. This is a missing linkup to the (internal?) libc function _nl_current_LC_CTYPE(). I had asked the build process to make a static binary, but because the ANTS build in turn automatically builds the ITK library from scratch, I wasn’t able to also ask ITK to build statically, and – I’m guessing here – statically linking in a non-statically-linked library doesn’t go and rewrite that library statically. Hence the crash.

ANTS isn’t tested or supported in this configuration so I’m going to stop chasing this rabbit hole and focus on doing a multi-platform build that should keep our packages up to date and avoid regressions like this in the future.

Thanks again for the report!

1 Like