PAYDAY 2's asset loading

(Note: the complete API definition can be found on GitLab)

Before modifying it, it's important to understand how PAYDAY 2 structures it's assets.

In the context of this, 'asset' refers to any files embedded into the game - textures, units, sounds etc.

Files are referred to not by name, but by a 64-bit identifier. This identifier is generated by hashing a string using lookup8, a non-cryptographic high-speed hash function. This is because computers can sort and search numbers much faster than strings, so this is fantastic for performance - however it poses something of a difficulty understanding these values. Outside of C++, these numbers are generally represented by 16-character hexadecimal values.

The community has build a 'hashlist'. This is a list of all the known strings in the game. Whenever you have a hash number (also known as an idstring) you can feed it to a program which will hash each of the strings in the hashlist and compare it to the value you provided. This allows you to effectively undo the hash function. Find a tool that does this (IIRC the bundle modder tool does) and keep it handy.

There is the so-called 'bundle database' file (assets/bundle_db.blb) which catalogues all of these files - for each one, it lists it's name, it's filetype, if it's available in multiple languages it lists that and so on.

The rest of the assets are split between the all_<number> packages and the hash-based packages. For example, assets/all_5.bundle and afb0290240327d2f_h.bundle respectively. These come with more files which store where the files are located in them. The exact details of this aren't important, but what is important is that a single file can be stored in multiple places among these files. This wastes space, but greatly improves loading times on mechanical hard drives.

These files are then divided up into 'packages'. A package is effectively just a list of assets. Each map in the base game calls for a list of packages, which together contains all the assets that will be used in that map. While the game is loading, it will copy all the assets from all the packages that the map requires into RAM.

This poses something of a problem for custom maps. A map author might pick an asset they would like to use, but they will then have to pick a package to load it from. Their map will load all the assets in that package into RAM, even if only a single one is actually used by the map. This isn't a problem for vanilla maps, since they can just copy all the odds-and-ends into a single map-specific package.

(Custom maps can use extracted assets which do something similar, bundling all these loose assets along with the map. This is inconvenient to set up, makes the maps much larger and is of somewhat questionable legality, so it's not ideal.)

The SuperBLT asset hook system

SuperBLT allows you to 'hook' an asset - whenever the base game tries to open an asset, SuperBLT will check if a Wren script has hooked that asset, and if so it will substitute it for the asset specified by that script.

A hook can also optionally be marked as a fallback. This means it will only be used if the game fails to load the asset by itself - this can improve loading performance if you hook files that the game has already loaded in a package, but if you're trying to change the contents of that file you cannot use this mode as sometimes your changes might not be seen.

When a script hooks an asset, it gets a DBAssetHook representing that asset. By default that hook is marked as 'disabled' - PAYDAY 2 will try (and possibly fail) to load that asset just like normal. From there you can put it into a different mode which enables it, making SuperBLT use this newly-specified source instead.

There are three sources you can set for an asset hook:

• Plain file mode - in this mode, SuperBLT will inject the contents of the specified file to be used instead of the regular file. This is the same as DB:create_entry or mod_overrides, except that it can be used from very early on in the game lifecycle, letting you override files like shaders.

• Direct bundle mode - in this mode, you specify the name and type of an asset. SuperBLT will then search for this asset in the bundle files discussed earlier, find the first copy of it and hand that back to the game. This allows you to make a single asset available without loading a whole package that contains it, addressing the custom map scenario discussed above.

If this lets you get more flexibility than the package system, why doesn't the base game use it? Loading speed on mechanical drives. Using this results in many random reads, while loading an entire package is probably going to be (depending on fragmentation) a single huge sequential read.

• Wren Loader mode - this will run a given piece of your code each time this asset is read, and lets you run all kinds of custom code for it. This is by far the most flexible option, but is inherently slower. Used in moderation it's just fine though.

Examples

Override the base-game shaders with customised ones:

import "base/native/DB_001" for DBManager

var hook = DBManager.register_asset_hook("core/temp/base", "shaders")
hook.plain_file = "mods/testing/my-custom-shaders.shaders"

Or with mod path detection included:

import "base/native/DB_001" for DBManager, DBForeignFile
import "base/native/Environment_001" for Environment

var hook = DBManager.register_asset_hook("core/temp/base", "shaders")
hook.plain_file = "%(Environment.mod_directory)/my-custom-shaders.shaders"

Make a unit file available, even when the bundle it's in isn't loaded (note you'll also have to set it up using the dynamic resource manager in Lua):

import "base/native/DB_001" for DBManager

var hook = DBManager.register_asset_hook("path/to/the/unit", "unit")
hook.fallback = true
hook.set_direct_bundle("path/to/the/unit", "unit")

Run a custom bit of code to dynamically generate the asset:

import "base/native/DB_001" for DBManager, DBForeignFile

class TestLoader {
    construct new() {}
    load_file(name, ext) {
        Logger.log("Hit the load function!")
        return DBForeignFile.from_string("Hello, World - this asset has the ID %(name).%(ext)")
    }
}

var hook = DBManager.register_asset_hook("testing123", "unit")
hook.wren_loader = TestLoader.new()

Plain File mode

This mode can be enabled by setting the plain_file attribute of the hook object. When the asset is to be loaded, the specified of the file will be used instead.

This is essentially the same as using DB:create_asset from Lua, but you can use it on assets such as shaders that are loaded before any Lua code runs.

Direct Bundle mode

This can be set with the set_direct_bundle(name, type) function of the hook object. This will directly load the file from the asset as it's used.

This allows you to use assets without loading them in from a package. If you're doing this, please ensure that you set the fallback flag, as it would otherwise ignore the user's mod_overrides mods, and also potentially increase loading times (using an asset that has already been loaded as part of a package is faster than loading it disk again).

Note that heavily using this (eg, for custom maps) could have a significant effect on loading times on mechanical drives, as the files won't be loaded in any particular order.

Wren Loader mode

In this case, you pass an object with a load_file(name, ext) method to the wren_loader property of the hook object. Whenever the target asset is then opened, the load_file method will be called and passed the path of that file, with both arguments being a plain hex-encoded idstring.

This object must then return a DBForeignFile instance, which represents the file that should be returned to the game. There are three types of DBForeignFiles:

• Plain files. Create them with DBForeignFile.of_file(path) and they behave like the file was loaded in plain-file mode.
• Asset files. Create them with DBForeignFile.of_asset(name, ext) and they are like files in direct bundle mode.
• String files. Create them with DBForeignFile.from_string(str) and the supplied string will be used as the file's contents.

You should generally try and avoid using a Wren loader if one of the other modes will suffice. Only a single thread can run Wren code at a given time, while many threads can concurrently load assets in any of the other mods. It's fine for occasional use, but please don't try loading an entire map's worth of assets in like this.

Also note that the load_file function may be called many times - if you are making an expensive computation in that function, you may want to cache it between calls (though then you'll have to consider memory usage).