[Question]: No more teams available

[MARD1NO]

Question

Recently I use nvshmem v3.7 to run deepep lowlatency mode, I apply the patch from: deepseek-ai/DeepEP#696

normal mode is success, but in low latency mode, I encouter:

/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:1821: error status: 2 (NVSHMEMX_ERROR_OUT_OF_MEMORY) No more teams available (max = 7), try setting NVSHMEM_MAX_TEAMS environment variable

/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:nvshmemi_team_allocate_resources:1975: Failed to set team index for team -1 is_dupl: 0

/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:1821: error status: 2 (NVSHMEMX_ERROR_OUT_OF_MEMORY) No more teams available (max = 7), try setting NVSHMEM_MAX_TEAMS environment variable

/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:nvshmemi_team_allocate_resources:1975: Failed to set team index for team -1 is_dupl: 0

/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:1821: error status: 2 (NVSHMEMX_ERROR_OUT_OF_MEMORY) No more teams available (max = 7), try setting NVSHMEM_MAX_TEAMS environment variable

/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:nvshmemi_team_allocate_resources:1975: /dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:1821: error status: 2 (NVSHMEMX_ERROR_OUT_OF_MEMORY) /dvs/p4/build/sw/rel/gpgpu/toolkit/r13.2/main_nvshmem/src/host/team/team_internal.cpp:1821: error status: 2 (NVSHMEMX_ERROR_OUT_OF_MEMORY) Failed to set team index for team -1 is_dupl: 0

Can you give me some suggestion to debug?

[benhg]

It seems like you are running out of memory. Can you monitor nvidia-smi or nvtop while the program runs and/or run NSys to see where the memory is going? I believe we also have a knob for printing whenever we allocate memory.

[MARD1NO]

It seems like you are running out of memory. Can you monitor nvidia-smi or nvtop while the program runs and/or run NSys to see where the memory is going? I believe we also have a knob for printing whenever we allocate memory.

I just running deepep unittest, I think the memory is enough. Which knob can I use to debug the memory allocation?

PS: I found it works well on nvshmem==3.4.5

[benhg]

Set NVSHMEM_DEBUG=DEBUG and NVSHMEM_DEBUG_SUBSYS=MEM