Memory keeps growing epoch by epoch and OOM after 7~8 epochs

[zhangsongrong]

Hi authors,

Thanks a lot for your excellent work and open-sourcing this project!

I encountered a serious memory issue when reproducing your code. My server is equipped with 96GB system RAM. During training, the overall system memory keeps accumulating epoch by epoch, and the program will run out of memory after around 7 to 8 epochs.

After days of debugging and observation, I found the root behaviors:

1. The memory usage of the main process keeps rising continuously during the entire training phase.
2. When a new epoch starts, every DataLoader worker process will copy the full memory of the main process via fork, which further amplifies the memory usage.

This problem has troubled me for several days. Would you mind sharing some solutions or suggestions to resolve this memory accumulation issue?

Thank you very much for your help!