Large Streaming Terrain in Unity: An Adventure

Recently, I made a fair bit of progress towards a streaming terrain solution in Unity. This kind of system powers open-world games like Grand Theft Auto, Skyrim, Fallout, and more.
I wanted to share the process behind it, and current progress thus far.

Some of my work was initially based on the GIS talk from Unite. The idea is that I'll store terrain files to be loaded in the StreamingAssets folder. So far, I've got Heightmap files (RAW) and Splatmap files (PNG). Later on I'll probably figure out how to store detail densities, but that's enough for now. When a terrain tile is needed, I load both the Heightmap and Splatmap associated with the tile (by name, based on tile X and Y) and create a new Terrain object.

Just having a terrain isn't enough. I need scenery, of course. I decided on the condition that all scenery to be saved must be a prefab in a resources folder - this allows me to simply serialize the name of the object, and later I can load the object via Resources.Load.
Objects are stored in yet another file, in a custom binary format I call the World Tile Format, and files are stored as .WTFs. The obvious pun was probably the biggest driving factor in choosing the name.
My first attempts to save objects were unsuccessful. The files always came out empty - WTF indeed. Turned out to be a simple error - forgot to close a stream. After getting that sorted out, I could save and load objects for individual terrain tiles. I also drafted up a simple editor utility to let me do this - it allows me to load terrain tiles, and save WTF files (saves the list of objects within the bounds of each terrain tile).
The actual format is pretty simple.

The header consists of 10 bytes, and looks like this:
{
    MAGIC: three ASCII chars ('WTF')
    VERSION: one byte.
    OBJ_COUNT: four bytes (uint)
    TREE_COUNT: two bytes (ushort)
}

As you can see, I also decided that my WTF file should store trees for a terrain tile.

Following the header is a big dump of each object. Each one is 296 bytes and looks like this:
{
    OBJECT_ID: 256 ASCII chars
    POSITION_X/Y/Z: four bytes per (float)
    SCALE_X/Y/Z: four bytes per (float)
    ROTATION_X/Y/Z/W: four bytes per (float)
}
I may trim down OBJECT_ID later, depending on how I feel. Have yet to see a prefab with anywhere close to 256 characters in the name, but you never know.

And finally, a dump of all trees. Each tree is just 5 bytes and looks like this:
{
    TREE_IDX: one byte
    POSITION_X: two bytes (short)
    POSITION_Y: two bytes (short)
}
Each position field is divided by short.MaxValue on load to map to the range 0-1 (I wanted to use bytes, but given a terrain tile spanning 650 meters that gives me a resolution of several meters, whereas a short gives me a resolution of smaller than a meter, but is still half the size of a full float value).

And this brings me to the current state of the thing.
Currently trees are unused (they are loaded from disk into a TreeInstance array, but not applied to the terrain - doing this would be trivial, however). My TerrainManager loads a radius of terrain and object tiles around the camera. The radius of each is different, so that I can load a big portion of terrain to avoid popping, but reduce the object load radius (since objects popping in/out is less noticeable than terrain popping in/out, especially with smaller objects). I believe Skyrim does something similar.
The last piece of the puzzle missing is detail objects like grass and bushes. I have not yet decided how I will store these, I might figure out how to store them in splatmap-style images (four detail objects neatly maps to R, G, B, and A channels which could map to a density range).
I also have a fair bit of optimizing to do. Loading terrain by itself is pretty speedy, although loading objects causes a noticeable hitch in framerate. I expected to spend time optimizing, however - it's my first try at anything remotely like this.

Still don't know what to do with this if I finish it.