Problem
When running SCT batch processing (sct_run_batch) in parallel with -jobs 16, it causes the QC .json files of many subjects to be missing from the QC report (index.html). The QC images are generated correctly but the JSON entries for index.html are not written.
I think the cause is that when multiple subjects run in parallel and reach the QC writing step simultaneously, they compete for a file lock (portalocker), sct raises AlreadyLocked and crashes out of the QC step, so the JSON entry is never written for those subjects.
Error message from log: portalocker.exceptions.AlreadyLocked File ".../spinalcordtoolbox/utils/fs.py", line 282, in mutex semaphore.acquire()
I think reducing -jobs to 1 would eliminate this problem entirely, but with each subject taking ~1 hour to process (based on the time documented in the log), 267 subjects would take approximately 11 days to process. And reducing -jobs to a moderate number (4–8) might help, but not solve this problem.
Thank you so much for raising this issue. I am surprised to see that the lock file is resulting in an exception being raised – we adopted that solution specifically to allow multiple processes to run concurrently and wait until the lock becomes available! So, we definitely would consider this a bug, and definitely don’t recommend -jobs 1 as a solution (at least, it shouldn’t be necessary).
I will open an issue on SCT’s GitHub repository and take a look at this as soon as I am able. Hopefully I can come up with a quick solution that allows you to proceed with your analysis in the short-term until we can arrive at a long-term fix.
Thank you for your patience, and thank you for your attention to detail in your report. (I’m about to take a look at your other QC-related post as well!)
Update: It seems the lock is working as intended, but the issue is that we’re running into the 60s timeout built into the lock.
Why we’re running into the timeout I’m not entirely sure (it could be due to the unique combination of -jobs 16 + 267 subjects – perhaps regenerating the HTML file takes a very long time once the number of subjects gets to a certain point, holding onto the lock for longer than anticipated.
This seems like we may need to rethink our structure for parallelism and QC reports. I have some ideas for that. But, in the short-term, it may be fruitful to simply extend the timeout as a quick workaround.
Navigate to L281 (shown above) and change the value from 60 to, say, 600.
Note: We shouldn’t actually need to lock any of the processes for ~10min, but what I’m expecting is that the “real” wait time is actually much smaller (e.g. <10s), and that we’re only exceeding the timeout due to multiple subjects queued up at once (e.g. 6*10s). At most, I expect the wait time to be e.g. 16*10s before all of the subjects begin their 1hr processing time once more.
Of course rather than an increased timeout I expect there to be a more sustainable solution (e.g. writing the JSON entries independently, then triggering a single HTML write at the end of the batch, eliminating the need for a lock entirely). But, this might alleviate the run failures in the short-term.