Steps to Reproduce
  • Create new Open World Map
  • Select "Build->Build HLODs" (works fine)
  • In 'World Settings' set the 'Cell Size' and 'Loading Range" to 2000
  • Select "Build->Build HLODs" (error in building)
  • This error can also be caught in VS by selecting 1 cell, and building its HLOD (from properties tab)
Callstack
LogD3D12RHI: Warning: GPU timeout: A payload (0x00000A1073040A80) on the [0x00000A1053669B80, 3D] queue has not completed after 5.000984 seconds.
D3D12: Removing Device.
LogD3D12RHI: Error: Aftermath: Writing Aftermath dump to: ../../../ProjectName/Saved/Logs/UEAftermathD3D12.nv-gpudmp
[2024.01.08-20.08.33:966][  1]LogD3D12RHI: Error: CurrentQueue.Fence.D3DFence->GetCompletedValue() failed 
 at ...Engine\Source\Runtime\D3D12RHI\Private\D3D12Submission.cpp:984 
 with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG

LogRHI: Error: [Aftermath] Status: Timeout
LogRHI: Error: [Aftermath] Failed to get Aftermath stack data
LogD3D12RHI: Error: [GPUBreadCrumb] Last tracked GPU operations:
LogD3D12RHI: Error: [GPUBreadCrumb]	3D Queue 0
Breadcrumbs: | BufferPoolCopyOps [Finished]
Breadcrumbs: | TexturePoolCopyOps [Finished]
Breadcrumbs: | CanvasClear [Finished]
Breadcrumbs: | SendAllEndOfFrameUpdates [Finished]
Breadcrumbs: | GPUResourceUpdate [Finished]
Breadcrumbs: | ClearGPUMessageBuffer [Finished]
Breadcrumbs: | UpdateAllPrimitiveSceneInfos [Finished]
Breadcrumbs: | ClearBuffer(Shadow.Virtual.CacheInstanceAsStatic Size=524288bytes) [Finished]
Breadcrumbs: | ClearBuffer(Shadow.Virtual.LastInstanceInvalidatedFrame Size=524288bytes) [Finished]
Breadcrumbs: | ClearBuffer(Shadow.Virtual.PhysicalPageMetaData Size=98304bytes) [Finished]
Breadcrumbs: | ClearBuffer(Shadow.Virtual.PhysicalPageMetaData Size=98304bytes) [Finished]
Breadcrumbs: | ShaderPrint::UploadParameters [Finished]
Breadcrumbs: > Scene [Active]
  Breadcrumbs: | AccessModePass[Graphics] (Textures: 0, Buffers: 1) [Finished]
  Breadcrumbs: | BuildRenderingCommandsDeferred(Culling=On) [Finished]
  Breadcrumbs: | AccessModePass[Graphics] (Textures: 1, Buffers: 1) [Finished]
  Breadcrumbs: | ClearDepthStencil (SceneDepthZ) [Finished]
  Breadcrumbs: | PrePass DDM_AllOpaqueNoVelocity (Forced by Nanite) [Finished]
  Breadcrumbs: | FVirtualShadowMapArray::UpdatePhysicalPageAddresses [Finished]
  Breadcrumbs: | ComputeLightGrid [Finished]
  Breadcrumbs: | LightFunctionAtlas Generation [Finished]
  Breadcrumbs: | BuildHZB(ViewId=0) [Finished]
  Breadcrumbs: | SubmitCommands [Finished]
  Breadcrumbs: | FenceOcclusionTests [Finished]
  Breadcrumbs: | SkyAtmosphereLUTs [Finished]
  Breadcrumbs: | CaptureConvolveSkyEnvMap [Finished]
  Breadcrumbs: | AccessModePass[Graphics] (Textures: 1, Buffers: 1) [Finished]
  Breadcrumbs: | CopyBuffer(NaniteRayTracing.AuxiliaryDataBuffer Size=32bytes) [Finished]
  Breadcrumbs: | GBufferClear [Finished]
  Breadcrumbs: | BasePass [Finished]
  Breadcrumbs: | LumenFrontLayerTranslucencyReflections [Finished]
  Breadcrumbs: | FVirtualShadowMapArray::BuildPageAllocation [Finished]
  Breadcrumbs: | ShadowDepths [Finished]
  Breadcrumbs: | DispatchToRHI [Finished]
  Breadcrumbs: | LightCompositionTasks_PreLighting [Finished]
  Breadcrumbs: | ClearStencil (SceneDepthZ) [Finished]
  Breadcrumbs: | DiffuseIndirectAndAO [Finished]
  Breadcrumbs: | Lights [Finished]
  Breadcrumbs: > ReflectionIndirect [Active]
    Breadcrumbs: > ReflectionEnvironmentAndSky(16384x16384,SubstrateMat=Off) [Active]
  Breadcrumbs: > SkyAtmosphere [Active]
  Breadcrumbs: > VolumetricCloud [Active]
    Breadcrumbs: > CloudView (CS) 16384x16384 [Active]
  Breadcrumbs:   PostProcessing [Not started]
  Breadcrumbs:   ReleaseRayTracingResources [Not started]
  Breadcrumbs:   RenderFinish [Not started]
  Breadcrumbs:   ExtractUniformBuffer [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 8, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]
Breadcrumbs:   ClearGPUMessageBuffer [Not started]
Breadcrumbs:   UpdateAllPrimitiveSceneInfos [Not started]
Breadcrumbs:   ShaderPrint::UploadParameters [Not started]
Breadcrumbs:   Scene [Not started]
Breadcrumbs:   EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs:   AccessModePass[Graphics] (Textures: 7, Buffers: 2) [Not started]

LogD3D12RHI: Error: [GPUBreadCrumb]	Copy Queue 0

LogD3D12RHI: Error: [GPUBreadCrumb]	Compute Queue 0

LogD3D12RHI: Error: DRED: Last tracked GPU operations:
LogD3D12RHI: Error: DRED: Commandlist "FD3D12CommandList (GPU 0)" on CommandQueue "3D Queue (GPU 0)", 40 completed of 127
LogD3D12RHI: Error: 	Op: 0, Unknown Op
LogD3D12RHI: Error: 	Op: 1, EndEvent
LogD3D12RHI: Error: 	Op: 2, BeginEvent [LightCompositionTasks_PreLighting]
LogD3D12RHI: Error: 	Op: 3, BeginEvent [AmbientOcclusionSetup 8192x8192]
LogD3D12RHI: Error: 	Op: 4, EndEvent
LogD3D12RHI: Error: 	Op: 5, BeginEvent [AmbientOcclusionPS 8192x8192 SetupAsInput=1 Upsample=0 ShaderQuality=2]
LogD3D12RHI: Error: 	Op: 6, EndEvent
LogD3D12RHI: Error: 	Op: 7, BeginEvent [AmbientOcclusionPS 16384x16384 SetupAsInput=0 Upsample=1 ShaderQuality=2]
LogD3D12RHI: Error: 	Op: 8, EndEvent
LogD3D12RHI: Error: 	Op: 9, EndEvent
LogD3D12RHI: Error: 	Op: 10, BeginEvent [ClearStencil (SceneDepthZ)]
LogD3D12RHI: Error: 	Op: 11, EndEvent
LogD3D12RHI: Error: 	Op: 12, BeginEvent [DiffuseIndirectAndAO]
LogD3D12RHI: Error: 	Op: 13, BeginEvent [DiffuseIndirectComposite(DiffuseIndirect=Disabled ApplyAOToSceneColor) 16384x16384]
LogD3D12RHI: Error: 	Op: 14, EndEvent
LogD3D12RHI: Error: 	Op: 15, EndEvent
LogD3D12RHI: Error: 	Op: 16, BeginEvent [Lights]
LogD3D12RHI: Error: 	Op: 17, BeginEvent [DirectLighting]
LogD3D12RHI: Error: 	Op: 18, BeginEvent [VirtualShadowMapProjectionMaskBits]
LogD3D12RHI: Error: 	Op: 19, BeginEvent [VirtualShadowMapProjection(RayCount:8(Adaptive),SamplesPerRay:4,Input:GBuffer)]
LogD3D12RHI: Error: 	Op: 20, EndEvent
LogD3D12RHI: Error: 	Op: 21, EndEvent
LogD3D12RHI: Error: 	Op: 22, BeginEvent [UnbatchedLights]
LogD3D12RHI: Error: 	Op: 23, BeginEvent [map_00682773_wp.DirectionalLight]
LogD3D12RHI: Error: 	Op: 24, BeginEvent [ClearRenderTarget(ShadowMaskTexture, slice 0, mip 0) 16384x16384 ClearAction]
LogD3D12RHI: Error: 	Op: 25, EndEvent
LogD3D12RHI: Error: 	Op: 26, BeginEvent [ShadowProjectionOnOpaque]
LogD3D12RHI: Error: 	Op: 27, BeginEvent [VirtualShadowMapProjection(RayCount:8(Adaptive),SamplesPerRay:4,Input:GBuffer)]
LogD3D12RHI: Error: 	Op: 28, EndEvent
LogD3D12RHI: Error: 	Op: 29, BeginEvent [CompositeVirtualShadowMapMask]
LogD3D12RHI: Error: 	Op: 30, EndEvent
LogD3D12RHI: Error: 	Op: 31, EndEvent
LogD3D12RHI: Error: 	Op: 32, BeginEvent [Light::StandardDeferred]
LogD3D12RHI: Error: 	Op: 33, EndEvent
LogD3D12RHI: Error: 	Op: 34, EndEvent
LogD3D12RHI: Error: 	Op: 35, EndEvent
LogD3D12RHI: Error: 	Op: 36, EndEvent
LogD3D12RHI: Error: 	Op: 37, EndEvent
LogD3D12RHI: Error: 	Op: 38, BeginEvent [ReflectionIndirect]
LogD3D12RHI: Error: 	Op: 39, BeginEvent [ReflectionEnvironmentAndSky(16384x16384,SubstrateMat=Off)] - LAST COMPLETED
LogD3D12RHI: Error: 	Op: 40, EndEvent
LogD3D12RHI: Error: 	Op: 41, EndEvent
LogD3D12RHI: Error: 	Op: 42, BeginEvent [SkyAtmosphere]
LogD3D12RHI: Error: 	Op: 43, EndEvent
LogD3D12RHI: Error: 	Op: 44, BeginEvent [VolumetricCloud]
LogD3D12RHI: Error: 	Op: 45, BeginEvent [CloudView (CS) 16384x16384]
LogD3D12RHI: Error: 	Op: 46, EndEvent
LogD3D12RHI: Error: 	Op: 47, EndEvent
LogD3D12RHI: Error: 	Op: 48, BeginEvent [PostProcessing]
LogD3D12RHI: Error: 	Op: 49, BeginEvent [PostProcessMaterial 16384x16384 Material=BaseColor]
LogD3D12RHI: Error: 	Op: 50, BeginEvent [BaseColor]
LogD3D12RHI: Error: 	Op: 51, EndEvent
LogD3D12RHI: Error: 	Op: 52, EndEvent
LogD3D12RHI: Error: 	Op: 53, BeginEvent [Downsample(PostProcessing.SceneColor.HalfRes Quality=Bilinear PS) 16384x16384 -> 8192x8192]
LogD3D12RHI: Error: 	Op: 54, EndEvent
LogD3D12RHI: Error: 	Op: 55, BeginEvent [SceneDownsample]
LogD3D12RHI: Error: 	Op: 56, BeginEvent [Downsample(Scene(1/4) Quality=High PS) 8192x8192 -> 4096x4096]
LogD3D12RHI: Error: 	Op: 57, EndEvent
LogD3D12RHI: Error: 	Op: 58, BeginEvent [Downsample(Scene(1/8) Quality=High PS) 4096x4096 -> 2048x2048]
LogD3D12RHI: Error: 	Op: 59, EndEvent
LogD3D12RHI: Error: 	Op: 60, BeginEvent [Downsample(Scene(1/16) Quality=High PS) 2048x2048 -> 1024x1024]
LogD3D12RHI: Error: DRED: No PageFault data.
LogD3D12RHI: Error: Memory Info from frame ID 2:
LogD3D12RHI: Error: 	Budget:	10273.00 MB
LogD3D12RHI: Error: 	Used:	 179.35 MB
LogD3D12RHI: Error: GPU Crashed or D3D Device Removed.

Have Comments or More Details?

There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-203446 in the post.

0
Login to Vote

Fixed
Fix Commit30524386
Main Commit30525291
CreatedJan 8, 2024
ResolvedJan 10, 2024
UpdatedFeb 20, 2024