[Question] Inference time TTA and results aggregation

[bastienchesaux]

❓ Question

Hi,
Thank you for your work on nnDetection.

I've been training nnDetection on my own dataset of mice lung ct-scans, that can present up to ~100 tumors (objects) per image. The dataset is a bit tricky because of image artifacts and some inconsistencies in the annotations.

I've managed to make preprocessing/training/inference worko on this dataset and I obtain similar detection metrics on the test set compared to what nnUNet achieves on this dataset.

However, I've noticed that using all TTAs during inference actually drops the F1 detection metric by 3%, on a test set of >150 images, compared to using no TTAs.

I find this quite surprising and wonder if it means that the aggregation settings found by the sweep are not ideal for my dataset. I've tried to use a different sweep metric (FROC instead of AP) but it results on the same set of hyper-parameters.

FYI for computing the F1 metric a detection is considered a TP if it's IOU is >0.4 with a GT instance's bbox. We use greedy assignement, i.e. there can be only one TP per GT instance.

Have you got any experience with datasets that contain this many objects? Does this make any sense to you?

Thank you for your input.

[andres2631996]

Hello,

In my opinion, here could be the causes from more relevant to less relevant:

• Since you are working with approximately 100 objects per image, it could be important to check the parameter "detections_per_img", since it is 100 by default. This could introduce some limitations, and perhaps increasing that parameter would be help, either modifying it here: https://github.com/MIC-DKFZ/nnDetection/blob/main/nndet/ptmodule/retinaunet/base.py#L436, or in the architecture plans.
  Related to TTA, the predictions from each TTA view are clipped to the number provided by "detections_per_img" before aggregation. Consequently, it may happen that some predictions are lost when using Weighted Box Clustering. Without TTA, there is no aggregation process and thus may lead to better performance. Hence, it could be maybe good to increase that parameter.

• If your IoU threshold to establish true positives is 0.40, could it be better to conduct sweeping with "AP_IoU_0.40" or "FROCwp_IoU_IoU_0.40"? If these metrics are not available, you can use "AP_IoU_0.30" / "FROCwp_IoU_IoU_0.30" or "AP_IoU_0.50" / "FROCwp_IoU_IoU_0.50". Perhaps it is better to directly optimize with the conditions that you are really interested in.

• Are the tumors maybe encoding whether they occur in the left or right lung? If that is the case, it may be that some mirroring transforms in the TTA damage performance, especially in the left-right axis.

• Which mirroring transforms do you have in your augmentation scheme? The default scheme has mirrorings across all axes.

You can let me know if any of this would be of help.

[bastienchesaux]

Hello,
thank you for your answer. A few following questions and comments:

• My understanding was the detections_per_img parameter was also being optimised for my task during the sweep process. If that's not the case then yes I might want to modify it. In which configuration file can I modify this setting? Also, the caseid_boxes.pkl files resulting from inference usually contain more than 100 detections. How's that?

• I'm using the default augmentation scheme so yes mirroring of all the axes are included in train-time augmentations.

• Are the tumors maybe encoding whether they occur in the left or right lung? : That's is a possibility yes, but in that case I guess the model would benefit from removing axis mirroring from the training transforms. I expect that if the model can handle those transforms during training it should be able to handle them at inference.

[andres2631996]

Hello,

The easiest way to modify detections_per_img for now is directly in nndet/ptmodule/retinaunet/base.py#L436. The parameter refers to the outputted predictions per inferred patch, which is why you see more than 100 boxes after patch stitching.
That said, this can still affect TTA performance: each TTA forward pass on a patch is independently truncated to the top 100 before aggregation, so borderline TPs in dense patches can flicker in and out across views and lose support during Weighted Box Clustering.

[bastienchesaux]

Hello @andres2631996

Thanks for the details and sorry for the delayed answer I got caught up in some other task.

Quick question about this modification, do I need to run the 5 folds traning again with detections_per_img=500? Or does it only impact sweep and/or consolidation?

[andres2631996]

No worries. No, you dont need to retrain, It only impacts sweep and consolidation

[bastienchesaux]

Hey @andres2631996,
One last precision, I made the modifications we discussed but it seems that the number of detections is still bounded to 100.
Are the parameters detection_per_img and model_detections_per_image in inference/ensembler/detection.py#L156 the same parameter? should I also modify the latter ?

[andres2631996]

Maybe try to modify both. I am away this week, with límited access