Designing a shader loading and management data structure

Last week we looked at how we can use the builder pattern to build shader programs out of individual shaders. We abstracted away all actual functionality in favour of a clean and easy to read interface.

In that post we assumed the existence of a shader manager – an object that held track of all our shaders and that we could ask for any of them given their type and a string name.

Today we will design a system fulfilling those requirements. Our goal is having this system take over as much work as possible, so that we do not have to bother with any of the details when using it.

Additionally we will look at how we can easily and automatically load shaders from code files – though this part of the system could be easily replaced by one loading them from a different source.

We will take an incremental approach to these topics, building our system step by step, according to our requirements.

The ShaderManager

The goal of designing our shader manager is to create an object that can handle an arbitrary collection of shaders for us. It serves as a database or abstract collection, that we can add and retrieve our shaders to/from.

Retrieving shaders

Last week we saw that we would like to ask our shader manager to return an arbitrary shader by its type and string name. Our method signature looked like this:

public Shader GetShader(ShaderType type, string name)

It is clear that we need to use a backing collection through which we can search based on a combination of the type and the name.

The easiest way of doing this is making use of a nested dictionary.

It does not matter much whether we index our outer dictionary by type or name, but it seems more intuitive to me to divide first by type, and lastly lookup by name.

This means we will want this field in our shader manager:

private Dictionary<ShaderType, Dictionary<string, Shader>> shaders;

To make sure we do not have to do so later, we will also initialise the field right away:

private Dictionary<ShaderType, Dictionary<string, Shader>> shaders
    = new Dictionary<ShaderType, Dictionary<string, Shader>>(3)
    {
        { ShaderType.VertexShader, new Dictionary<string, Shader>() },
        { ShaderType.FragmentShader, new Dictionary<string, Shader>() },
        /* other shader types */
    };

We then implement our method as follows.

public Shader GetShader(ShaderType type, string name)
{
    return this.shaders[type][name];
}

Note that this can throw an exception, either if an unknown type is given, or if the dictionary for the given type does not have a shader by the given name. If we want to prevent these exceptions and handle the error ourselves, we can do so by using Dictionary.TryGetValue instead of the indexer.

Retrieving with indexers

Speaking of indexers, I like to use indexers myself when creating classes that represent collections of some sort or another. Note here that indexers can take any type as parameter, and take multiple parameters as well, which makes them very flexible.

Since our shader manager is in effect a collection of shaders, I would thus do so:

public Shader this[ShaderType type, string name]
{
    get
    {
        Shader shader;
        this.shaders[type].TryGetValue(name, our shader);
        return shader;
    }
}

This implementation also shows how we can use Dictionary.TryGetValue to prevent exceptions – at least for invalid names in this case. The indexer will return the default value of Shader, which is null, if the named shader is not found. It will still throw however, if an unknown or invalid shader type is received.

Adding shaders

Adding shaders to our shader manager is extremely simple. Shaders already know their type, since this is necessary when compiling them.

Therefore, we can quickly add an Add() method like follows.

public void Add(Shader shader, string name)
{
    this.shaders[shader.Type].Add(name, shader);
}

Loading shaders from files

As we saw in my introduction to OpenGL in C# creating a shader program from code is relatively easy.

Loading it from a file is only marginally more difficult since we can use .NET’s File.ReadAllText() to quickly read in an entire source file.

Fully expanded, loading a shader from a source file thus looks like this:

var shader = new Shader(ShaderType.VertexShader, File.ReadAllText("shaders/myShader.vs"));

Using a line of code like this for every single shader that we need is neither a pretty, nor a flexible solution however. Would it not be much nicer if we could simple drop all our shader files into a folder, and have our program load them all by itself?

We will create a ShaderFileLoader class that will do exactly this.

Inferring shader types

First now that to be able to have a usable shader, we need to know its type. Since we want to load our shaders completely automatically, we have to encode this somewhere in the file itself.

We could do this by using a special line at the beginning of the shader’s source code, or by keeping the files in separate directories.

However, I like neither of those approaches. Instead, I like to use the file’s extension to signify the shader’s type: “.vs” for vertex shaders, “.fs” for fragment shaders, and so on.

Given a file name, we can use this method to quickly determine if and how to load it, even before looking at the file’s content in the first place.

Loading all the files

In fact, knowing that we want to load all files with a certain extension in a certain way, we can use another useful .NET method to enumerate all files that fit that pattern for us: Directory.EnumerateFiles() (also see the similar Directory.GetAllFiles()).

In essence we can now make methods like:

public IEnumerable<Shader> LoadAllVertexShaders(string searchPath)
{
    return Directory.EnumerateFiles(searchPath, "*.vs")
        .Select(f => new Shader(
            ShaderType.VertexShader, File.ReadAllText(f)
            ));
}

This method will find all vertex shader files in a given directory, and return an enumerable of loaded shaders.

Inferring shader names

There is a problem with this however: To be able to add this shader to our shader manager, we need a name for it. Since everything needs to happen automatically, we do not have the time to assign names manually. We again have to infer the names from the files themselves.

The most obvious solution is to use the filename itself.

To do this we will create a small helper method getFriendlyName(). This method will take our full file name and the base path we are loading shaders from, and return the file name, including sub-directories, but excluding the extension.

For example, if we are searching in “data/shaders/” and we find a shader under “data/shaders/particles/fancy.vs” we want to return “particles/fancy”.

To make our code work on any operating system, we first need to make sure to replace all “\” with “/” (or the other way around, if we prefer). We then take a substring, starting after the length of our search path, and then use yet another .NET method to remove the extension.

The combined method looks like this:

private string getFriendlyName(string prefix, string fullPath)
{
    return Path.ChangeExtension(
        fullPath.Replace(@"", "/").Substring(prefix.Length),
        null
        );
}

We are not done however. While we can now create very nice names for our shaders, we still need to keep track of them.

We can do so easily enough by creating a small container class:

class ShaderFile
{
    private readonly Shader shader;
    private readonly string friendlyName;
    public ShaderFile(Shader shader, string friendlyName)
    {
        this.shader = shader;
        this.friendlyName = friendlyName;
    }
    public Shader Shader { get { return this.shader; } }
    public string FriendlyName { get { return this.friendlyName; } }
}

Making things more general

While we plug this new functionality into our loading method, let us also make that one more generic.

As you saw, our name generating method already has fine support for nested directories. We will add a parameter to search recursively.

Additionally, we will change the method to also take the shader type and file extension as parameters, so that we can reuse it for the different kinds of shaders.

The result looks as follows.

public IEnumerable<ShaderFile> LoadAllShaders(ShaderType type, string searchPath, string extension, bool recursive)
{
    var searchOption = recursive
        ? SearchOption.AllDirectories
        : SearchOption.TopDirectoryOnly;
    return Directory.EnumerateFiles(searchPath, extension, searchOption)
        .Select(f => this.loadShader(type, searchPath, f));
}
private ShaderFile loadShader(ShaderType type, string prefixPath, string fileName)
{
    return new ShaderFile(
        new Shader(type, File.ReadAllText(fileName)),
        this.getFriendlyName(prefixPath, fileName)
        );
}

Note that I extracted the actual shader creation into its own method for ease of reading.

Loading all the shaders

Now that we are this far, loading all shaders is easy

We simply create a method that calls the one above for all the shader types we are interested in. For example:

public IEnumerable<ShaderFile> LoadAllShaders(string path, bool recursive)
{
    var shaders = new List<ShaderFile>();
    shaders.AddRange(this.LoadAllShaders(
        ShaderType.VertexShader, path, "*.vs", recursive
        ));
    shaders.AddRange(this.LoadAllShaders(
        ShaderType.FragmentShader, path, "*.fs", recursive
        ));
    /* other shader types */
    return shaders;
}

Adding loaded shaders to the manager

The last thing that remains is to add our loaded shaders to our shader manager. Once we have done so, we can use the shader program builder we created last week to easily link our shaders together into usable shader programs.

To make things easy, we will create an overload of our Add() method that directly accepts the return value from our shader file loading method – an enumerable of shader files.

public void Add(IEnumerable<ShaderFile> shaderFiles)
{
    foreach(var file in shaderFile)
    {
        this.Add(file);
    }
}
public void Add(ShaderFile shaderFile)
{
    this.Add(shaderFile.Shader, shaderFile.FriendlyName);
}

And that is all!

We can now load any number of shader files – as long as they are in the same directory with just a few lines of code:

var shaderLoader = new ShaderFileLoader();
var shaderManager = new ShaderManager();
var shaders = shaderLoader.LoadAllShaders("data/shaders/", true);
shaderManager.Add(shaders);

Or, to be more concise:

var shaderManager = new ShaderManager();
shaderManager.Add(
    new ShaderFileLoader().LoadAllShaders("data/shaders/", true)
    );

We could make it even more concise by using C#’s collection initializers, but that goes a bit too far here, and is more of a curiosity than something I would do in a production setting.

Conclusion

I hope this has been interesting and that I have been able to show how to create a system like the one above from scratch, in an iterative and agile manner, based on simple requirements.

The same approach can of course be taken for virtually any kind of system, and in that way, this post serves merely as an example for how one might go through the software design process.

If you would like to see an even more extensive version of the code above, I would invite you to check out the full version in my open source C# OpenGL library on GitHub.

That version is a lot more flexible compared to what we designed here, and it also includes one more important feature: the ability to reload changed shaders at runtime.

That has turned out to be a very useful feature while editing shaders, since one can see the effect of a change within the game or application almost instantly.

We will go into the design and implementation of that system in a future post.

For now, let me know what you think of this way of designing a data structure, and also what you think about my implementation itself.

Of course, always feel free to leave a comment below if you have any questions.

Enjoy the pixels!