[BUG]The higher the zero_stage, the GPU memory is consumed.

[yangshenchang]

Thank you for your wonderful work. When I used a small model to test the zero strategy of deepspeed, the higher the stage, the more video memory is consumed. Is there any other operation that needs to be enabled?
The model used is resnet152

[tjruwase]

@yangshenchang, ZeRO is designed for data-parallel training of multi-billion parameter models that are larger than GPU memory. ZeRO allocates some extra buffers for its memory optimizations. In other words, ZeRO is not recommended for small models like resnet152.

[yangshenchang]

Ok，Thanks a lot