Packaged Builds with DX12 Crash on 2nd Start

Running a packaged game with DX12 will crash on launch every second start and after. If you delete the files with the .ushaderprecache extension that are in the ProjectName/Saved folder the exe starts again, but this has to be done each time.

Working as expected in 4.19 CL# 4033788

Found in 4.20 CL# 4369336, 4.21 CL# 4472090, 4.22 CL# 4481780

This is a regression.

1. Open the Editor
2. Create a new level and set it as the start level
3. Package the project
4. Run the project executable with -dx12
5. Close the project
6. Run it again with -dx12

Results:
The packaged project will crash on launch after running and closing it for the first time

Expected:
The packaged project would open