Virtual File System

[blunted2night]

I have created a branch that implements a plug-able a virtual file system for asset loading. It is mostly functional except for hot-reloading, the ability to preload entire directories, and WASM are currently broken.

It works by adding a list of "asset storage providers" to the asset server and an API to add new providers to that list. A default provider that implements the current behavior is automatically added to the assert server when it is created. When a request for an asset is made, the set of providers are queried in order until one returns the file contents. The provider may return the data as a reference or a vector depending on what is convenient for the provider.

Once the asset data is available it is passed to the existing asset loader with some minor changes. The asset loader now takes the storage provider's "asset storage" enum instead of a vector of bytes. This allows for zero copy asset loading from memory mapped or embedded asset archives.

For Example

This is a storage provider that can serve files that are embedded directly into the executable:

pub struct StaticStorageProvider(&'static [(&'static str, &'static [u8])]);

#[async_trait]
impl AssetStorageProvider for StaticStorageProvider {
    fn find_asset_storage_sync<'a> (&'a self, path: &Path)
        -> std::result::Result<Option<AssetStorage<'a>>, AssetLoadError>
    {
        for (entry_path, value) in self.0 {
            if Path::new(entry_path) == path {
                return Ok(Some(AssetStorage::Static(value)));
            }
        }

        Ok(None)
    }
}

A data table for it can be specified like this:

const ASSETS : &'static [(&'static str, &'static [u8])] = &[
    ("texture.png", std::include_bytes!{"../assets/texture.png"}),
];

And configured into the app:

fn main() {

    App::build()
        ...
        .add_asset_storage_provider (StaticStorageProvider(ASSETS))
        ...
        .run();
}

Then accessed:

fn setup(
    asset_server: Res<AssetServer>,
    mut textures: ResMut<Assets<Texture>>,
) {
    let embedded = asset_server
        .load_sync(&mut textures, "texture.png")
        .unwrap();
    ...
}

There is certainly room for improvement

The names of entities should probably be improved, for example AssetStorage might better by AssetContent or event AssetBytes.

Some care was required to allow the file open function to return a reference, but it comes down to: as long a reference to a "provider list" Arc or the "resolver" Arc is held, a reference to asset data can also be held.

An asset loader should be given an opportunity to clone the providers Arc, so that it may extend the lifetime of the data reference if the desire exists.

It's not clear to me what the difference between sync and async loading is in WASM is. I see that sync calls are done via the standard library where as the async seems to use the browsers JavaScript API. It shouldn't be a problem to preserve the current behavior though.

The hot-reloading (at-least the monitoring portion) would need to be put in the storage provider, but the code now works in a global static way that I'm not sure how to transform.

The fix for folder pre-loading I guess would be for each storage provider to return a list of filenames, or take a closure that can be invoked on each file within folder that provider can see. Then asset server would filter potential duplicates present in different providers.

If anyone is interested, it is accessible from my fork under the vfs branch. Any feedback is appreciated. Thanks you for you time.

[cart]

Sounds like we had very similar ideas 😄

A couple of hours before you posted this i submitted this pr #693, which among many other changes adds an "AssetIo" abstraction.

[blunted2night]

I had a thought that something might already be in the works. It was an interesting dive into the code and on to how to approach things from a rust perspective, and if there was nothing on the horizon it could be a improvement to the current situation.