Parallel processing with dstools-extract-ds

[JacksonJMB]

Currently dstools-extract-ds deletes all temporary files in the MeasurementSet's parent directory. In extract_ds.py, main ends with

os.system(f"rm -r {ms.path.parent}/*dstools-temp*.*ms 2>/dev/null")

This causes difficulties for parallel processing, since when the command finishes executing in one process, it causes a failure in the other processes. A temporary fix for my purposes is to separate my data into multiple directories, but would it be possible to change extract_ds.py so that only the files created by the given process are removed?

[krose098]

@joshoewahp would the solution we were discussing the other day for parallel processing of the same MeasurementSet work for this as well? i.e. generating a a temporary directory with a copy of the visibilities?

[joshoewahp]

Thanks @JacksonJMB!

Yeah the ideal fix for this is something along the lines of your suggestion @krose098. Store intermediate products in a unique temp directory for each run (e.g. using the builtin tempfile module) and clear that rather than by using wildcards.

Will aim to fix this for v2.1