Frequent Hang Issue in test_vnet and test_ahnet during daily tests

[KumoLiu]

[2025-09-20T21:06:34.990Z] Starting test: test_ahnet_shape_2d_0 (tests.networks.nets.test_ahnet.TestAHNET)...

[2025-09-20T22:20:44.737Z] Sending interrupt signal to process

[2025-09-20T22:20:44.737Z] Killing processes

[2025-09-20T22:20:45.018Z] kill finished with exit code 0

[2025-09-20T22:20:58.768Z] script returned exit code 143

[2025-09-19T20:39:29.985Z] Starting test: test_vnet_shape_0 (tests.networks.nets.test_vnet.TestVNet)...

[2025-09-19T22:20:43.981Z] Sending interrupt signal to process

[2025-09-19T22:20:43.988Z] Killing processes

[2025-09-19T22:20:44.300Z] kill finished with exit code 0

[2025-09-19T22:20:54.367Z] script returned exit code 143

[KumoLiu]

I found the issue only happen when pytorch==2.8.0.
I fixed the pytorch version to 2.7.1 in blossom pip pipeline.

[ericspod]

#8561 will want to test with PyTorch 2.8.0. The tests there seem to be completing successfully so it's possible this issue is resolved.

[ericspod]

I'm going to close this as resolved now with other changes that have been made since raising.