Description

As with shader compilation, some third-party plugins need to accumulate information about all UObjects cooked in a cook session, and write their data to a database at the end of the cook; the database in general is in a custom format and is staged to the runtime and loaded with custom code. One way to implement this is allowing them to store their data in a UObject and specify that it should be saved after all of their other assets have been saved. But that adds complexity to the cook. Can we recommend another solution that satisfies all of the optimal-packaging needs that they get from bulkdata in a UObject?

An example of use:

In Editor/Contents folder, we have references inside UAssets, such as a UPluginAsset.
We have a system used during cooking, that looks at our Database type, and tells "UPluginAsset A needs the following DatabaseDatas from our database: X,Y,Z".
If a DatabaseData is ONLY used in ONE UPluginAsset, it's packaged in a FByteBulkData.
If a DatabaseData is used in MANY UPluginAssets (that's typically a multi-megabyte file, or a streaming file), it's absolutely inadequate to copy the data on all the UPluginAssets, so we combine them into database used for the runtime and store them in a UDatabaseAsset.
We also have to extract languages from the UPluginAssets, so we can create chunks for languages (eg. chunk 100 might be multilingual) by extracting them and putting them somewhere else.
We also have to extract DLCs, or modular gameplays.

We store them in a UObject because want to be as close as possible to "regular" UAsset. Our shared clients have very broad usage of Unreal, and we want to be 100% certain we package their data into assets that work as well as possible in Unreal. If UAssets are in IOStore, we'll go to IOStore, or feel the wrath of users asking why their previously-nearly-empty .pak files are suddenly 2gb with all of our DatabaseData.
So ... we know for performance considerations that asset libraries could reside in a pak, or honestly, outside as external files. But we don't want that.
It's also a way for us to future proof or offerings. By precisely supporting UAssets, we live in a world where we have an option to use AdditionalFilesOverride, or to put them inside UAssets as individual Bulk Datas, or to put them inside libraries as a monolithic BulkData.
From that point on, if Unreal and our toolchain for Unreal eventually reaches a point where we can create packaged UAssets that don't have an Editor's alter-ego (for example by faking an uasset with a precise GUID as existing through a fake virtual asset repository), we could automatically create UAssets with only the necessary chunks of data as smaller libraries and tie them directly into the requester UAsset, and that will use the IO Store System with its intended optimizations. This system being an extension of the current one, if we arrive at a solution to do that, the infrastructure will already be paved.

However, we cannot tell whether any DatabaseAsset in the project database is actually used or not in the project, so we can only ask the user to please fill in what they want in the package, and filter out their debug stuff. This is what we are doing for our next major release. It's useable, but manual.

We therefore would like the ability to save our UDatabaseAsset at the end of the cook, after all of the UPluginAssets have been cooked.

Have Comments or More Details?

There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-209045 in the post.

1
Login to Vote

Unresolved
CreatedMar 6, 2024
UpdatedMar 11, 2024