DP vs DDP

Update: 02/18/2024
https://huggingface.co/docs/transformers/v4.20.1/en/perf_train_gpu_many#:~:text=DP%20copies%20data%20within%20the,not%20the%20case%20with%20DP.

————/
DDP를 써라!
핵심은 DP는 single-process multi-thread이고, DDP는 multi-process라는 것. python GIL overhead를 피하려면 multi-thread가 아닌 multi-process를 쓰는게 효율적임.

https://pytorch.org/docs/master/notes/cuda.html#use-nn-parallel-distributeddataparallel-instead-of-multiprocessing-or-nn-dataparallel

CUDA semantics — PyTorch master documentation

DDP overview
https://pytorch.org/tutorials/beginner/dist_overview.html

PyTorch Distributed Overview — PyTorch Tutorials 1.9.0+cu102 documentation

DDP, DP와 batch norm
https://discuss.pytorch.org/t/how-does-dataparallel-handels-batch-norm/14040

How does Dataparallel handels batch norm?

참고 dataloader num_workers는 subprocess가 아니라 thread 수를 지정해주는것임.
process vs thread 참고: https://stackoverflow.com/questions/37430255/difference-between-subprocess-and-thread