[BUG] RIFE Interpolation engine hangs randomly #37

Hello! I am trying to use TensorRT-RTX but I have a very interesting bug.

NOTE: TensorRT 11 is fine, works perfectly with zero problems, only TensorRT-RTX 1.5 hangs randomly. I DO want to use the RTX version because it is more interesting: Portable engines, much lower binary size are important for me. Anyway the problem is this:

A RIFE v4.6 engine built with TensorRT for RTX 1.5.0.114 hangs forever inside enqueueV3 at the input shape 1x11x576x704, but only when the GPU is allowed to go idle between consecutive enqueues! The hang is reproducible with tensorrt_rtx alone (no application code) by adding --idleTime.

The same ONNX, same shape, same idle pattern, same GPU runs good on mainline TensorRT 11.1.0.106, which is I believe this is an RTX-specific problem.

rife_v4.6_fp16.onnx.zip

NOTE: This is the fp16 version of the model which I AutoCast (but kept some "fp32 islands" with surgeries), the original FP32 model can be found here

rife_v4.6.onnx.zip

When it hangs, the GPU sits at 100% utilization but only ~39 W, with 0-1% memory utilization and one CPU core spinning at 100%.

Real inference on this engine draws 150-220 W normally. I left it in this state for 4.5 hours (well, I slept expecting it to complete in an hour 😅) with zero progress.

Environment

| Model | rife_v4.6 ONNX (fp16, opset 16), from vs-mlrt external-models. 11-channel input [1,11,H,W] |

Reproduction

Build the engine (dynamic shapes, these are the ones I use):

tensorrt_rtx --onnx=rife_v4.6_fp16.onnx --saveEngine=rife_v4.6_fp16.trt \
             --minShapes=input:1x11x128x128 \
             --optShapes=input:1x11x1088x1920 \
             --maxShapes=input:1x11x2176x3840

Run it at 576x704 with an idle gap between enqueues:

# HANGS: never terminates, must be killed
tensorrt_rtx --loadEngine=rife_v4.6_fp16.trt --shapes=input:1x11x576x704 \
             --iterations=800 --duration=0 --idleTime=30

Expected: completes 800 iterations (roughly 1.3 ms of compute each) and reports PASSED.

Actual: stops making progress after a few hundred iterations and hangs indefinitely. The last log line is [I] Starting inference.

The same command without the idle gap passes:

# PASSES (781 qps, exit 0). The GPU never drains between enqueues.
tensorrt_rtx --loadEngine=rife_v4.6_fp16.trt --shapes=input:1x11x576x704 \
             --iterations=300 --duration=0 --idleTime=0

The hang is random

It happens after roughly 1 in 600 to 1500 enqueues.

Two consecutive runs of the identical command (--iterations=1500 --idleTime=30) gave HANG then ok.

A single passing run therefore proves nothing. I recommend at least 8000 enqueues per configuration when verifying.

The original FP32 version may hang MUCH later but it WILL hang, and it may also PASS sometimes. generally this is very random! But IF it happens then the GPU is stuck!

What we ruled out (all measured)

So the trigger is very narrow: this one input shape on this graph (of course there may be many more), plus an idle GPU between enqueues.

Neighbouring shapes (including the same height at other widths, and the same width at other heights) are good.

Additionally, from the application side (before we reduced it to trtexec), we ruled out:

• Not our memory. The TRT workspace and all application buffers were verified non-overlapping.
• Not a race with application kernels. A full cudaStreamSynchronize immediately before enqueueV3 (empty stream, cudaSuccess) still hangs. The stuck kernel belongs to the engine.
• Not out-of-bounds access. compute-sanitizer memcheck reports zero errors.

Idle gap is important!

This is not a synthetic condition.

In our video pipeline, RIFE enqueues are separated by an idle GPU whenever a CPU-side stage is the bottleneck (software video decode, or an x265 CPU encode applying backpressure).

Under a saturated GPU pipeline (hardware decode and encode) the same shape runs fine indefinitely, which is why this only surfaced on one code path!!

Impact

Any application feeding this engine a 576x704 input on a non-saturated GPU hangs permanently, with no error, no exception, and no timeout.

It cannot be caught or recovered from in-process!

Our workaround, avoiding the shape, is not something an application can do in general, since the "bad" shape is not predictable.