I am trying to transcode HEVC encoded video to AV1 and facing VCEnc crashes related to VRAM allocation.
Few minutes (not measured the exact time) after the total available VRAM is consumed crash happens.

I tried drivers 25.9.1, 25.9.2 and 25.10.1 BETA with no luck. No differences observed, always ends up crashing.

The environment is:

Windows    : Windows 10.0.26100 24H2 Professional
Language   : English (United States)
CPU        : AMD Ryzen 7 5800X 8-Core Processor
RAM        : 64GB
GPU        : AMD Radeon RX 7900 GRE

StaxRip is used as a frontend for VCEEnc.

Encoder is executed with following parameters (asterix to mask filenames):
Latest (9.01) version used.

VCEEncC64.exe 
--avhw 
--vbr 24000 
--codec av1 
--output-depth 10 
--preset slower 
--b-pyramid 
--max-bitrate 52000 
--pe 
--aq-mode caq 
--cdef-mode on 
--cdf-update 
--cdf-frame-end-update 
--dolby-vision-profile 10.1 
--dolby-vision-rpu *
--master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)" 
--colormatrix bt2020nc 
--colorprim bt2020 
--transfer smpte2084 
--max-cll "303,68" 
--chromaloc 2 
--ssim 
--psnr 
--motion-est full-pel 
--chapter-copy 
--vbaq 
--crop 0,276,0,276 -i * -o *

The log is pretty much huge so attaching just the messages directly preceding crash.
This particular log is taken with drivers 25.9.2 installed.

log-snippet.txt

I have also noticed when VRAM is not fully allocated then GPU is utilized in full...and it reverses on total VRAM usage.
Windows Task Manager screenshots below:

1. GPU fully utilized (VRAM not fully allocated)

2. GPU not fully utilized (all VRAM allocated)

3. After crash