Teal - A Turn Based Strategy Game

Unreal Engine Custom Crash Reporter

Well hello again (glances at last blog post release date). Teal is now available on Steam and as such we are now in the post-launch support phase. One important aspect is to be able to track crashes and debug them. The Unreal Engine 4 actually already does something like that. You have probably seen this screen at some point during development.

This crash reporter sends the some data about the crash to Epic. The debugging symbols can be included when packaging a build and you will get the call stack. So, the first thing we want is our own crash reporter to be called when something happens. Then we can send the crash data to us, to help us debug the problem. For that we need to know how the Unreal Engine does it. For that you need to download the full source code for the Unreal Engine 4, it is called the CrashReportClient. You should download the Unreal Engine 4 source anyway and build the MiniDumpDiagnostics, since we will be using it later.

Side Note: You can skip to the next paragraph if you already know how to access the Unreal Engine source. You need to be invited to the GitHub repository to get access. To do that you need to link your Epic Games account with your GitHub account. So, create a GitHub account if you don’t already have one. Now go to your Epic Games account and go to the Connected Accounts section. Scroll down and enter the name of your GitHub account. Now you should get an invite on your GitHub account and go to the Unreal Engine GitHub page.

The crash reporter is a separate program that gets called if the game crashes. The crash reporter executable is located in Engine\Binaries[BuildConfigration]\CrashReportClient.exe. You just have to replace that executable with the one for your custom crash reporter. Just make sure it has the same name as the default one “CrashReportClient”, and it will get called if something happens.

The crash reporter for Teal is a C# application, since we only Ship Teal on desktop Windows. The Unreal crash reporter ships for all desktop platforms, but the general idea is always the same. So let’s create a simple WPF application that opens a message box and closes afterwards. You should name the solution CrashReportClient to make things easier for yourself later. Otherwise, you will have to rename your executable later.

App.xml.cs

using System.Windows;

namespace CrashReportClient
{
    public partial class App : Application
    {
        protected override void OnStartup(StartupEventArgs e)
        {
            base.OnStartup(e);

            MessageBox.Show("A Crash has occurred", "Custom Crash Reporter");
            Shutdown();
        }
    }
}

Now we package a build that can crash on command. In case we just added a function to our CheatManager that will always crash. We just try to read from a null pointer and the variables are volatile so the optimizer can’t remove them.

volatile int* ptr = nullptr;
volatile int crash = *ptr;

If you don’t have a cheat manager setup you can also put the function anywhere with a UFUNCTION(exec) declaration. Then you can crash the game with a simple console command. Or put it anywhere you want, just make sure you can trigger it somehow.

So, now we package a build and replace the CrashReportClient.exe with our own. Now if we trigger a crash our message box will pop up. You will probably notice something because our crash reporter does not only start when a crash occurs but also whenever any exception occurs. This is generally not an issue in a shipping build since asserts are removed, and we generally don’t trigger exception that cause a crash. But luckily the crash reporter is started with a couple command line arguments, and these help us distinguish between a regular exception and an actual crash. We get the “-Unattended” command line argument if the crash reporter was not triggered by a crash. So, we just add a simple check for that parameter.

App.xml.cs

using System.Linq;
using System.Windows;

namespace CrashReportClient
{
    public partial class App : Application
    {
        protected override void OnStartup(StartupEventArgs e)
        {
            base.OnStartup(e);

            if( !e.Args.Contains("-Unattended"))
            {
                MessageBox.Show("A Crash has occured", "Custom Crash Reporter");
            }
            Shutdown();
        }
    }
}

We actually get a couple more interesting command line arguments but more on that later. Now we can execute custom code when our game crashes. This brings us to the second part of this post. What data does the Unreal Engine 4 collect when a crash happens and what can we do with it.

Crash Data

A crash report folder is created whenever a crash occurs. These crash reports are stored alongside the save file in the Saved/Crashes folder. The save folder is stored right next to the game executable in debug and development builds. Shipping builds store their save data at %USERPROFILE%/AppData/Local/[Game]/Saved/
(for example, for Teal this is %USERPROFILE%/AppData/Local/Teal/Saved/). The crash report folders are named after the system and the crash GUID. They contain a CrashContext.runtime-xml, a CrashReportClient.ini, potentially a .log file and finally a UE4Minidump.dmp.

The XML file contains some information about the system where the crash occurred. The more interesting file is the minidump. That is a small crash dump. While these do not contain as much information as a regular crash dump they are small and can be generated pretty fast. You can open them with Visual Studio to read them, as long as you have the executable and the symbols (the .pdb in our case). Reading them directly with Visual Studio never really worked for me but the Unreal Engine 4 comes with a handy tool that lets us extract the call stack and the loaded modules from the minidump. That is also why I told you to download the engine source code at the beginning.

Go to the MiniDumpDiagnostics project in the Unreal Engine 4 solution. Compile it and create a minidump with a crash in the meantime. Now we need to edit a config file so the tool knows where the executable and .pdb are located.
Go to Engine\Programs\MinidumpDiagnostics\Saved\Config\Windows and edit the Engine.ini in the folder where you compiled the Unreal Engine.

Engine.ini

[Engine.CrashDebugHelper]
DepotRoot=[PathToSourceCode]
PDBCachePath=[PathToCacheFolder]
PDBCacheSizeGB=250
MinDiskFreeSpaceGB=25
DaysToDeleteUnusedFilesFromPDBCache=3
PDBCache_0_Branch=++UE4+Release
PDBCache_0_ExecutablePathPattern=[PathToExecutableFolder]
PDBCache_0_SymbolPathPattern=[PathToPdbFolder]

Just create an empty folder to be used as the cache folder, and the executable and .pdb are in [Game]/Binaries/[BuildConfigration] of your packaged build. The source code is optional. Now you just have to call the programm from the command line with a couple arguments.

MinidumpDiagnostics.exe [PathToMinidump] -SyncSymbols -bUsePDBCache=1

You can also add the “-log” argument if the process fails and you want to know why. Now you either have a “Diagnostics.txt” or a “DiagnosticsFailed.txt” with the call stack and all loaded modules. And this covers what you get with a crash report. You can of course also send anything else you want like the save game, just make sure the user knows what gets sent.

The Crash Reporter

The only left is to actually implement the crash reporter. For example here is the one we use for Teal.

The only thing I will cover here is how to get the crash data. What you want to send and how is up to you. We just send the user description, the XML and the minidump to a database where we can poll them later. We can also mark the crash reports as handled once their issue has been resolved. You should also add a localization for the crash reporter for every language the game ships in. For Teal this is German and English.

But first, let’s cover the remaining command line arguments I hinted at earlier. You get the following arguments with a crash. - The path to the executable - “-Appname=UE4-[GameName]” The name of your game - “-CrashGUID=UE4CC-[Platform]-[GUID]” A GUID for the crash - “-DebugSymbols=[PathToDebugSymbols]” - “-Unattended” This is only there if this was not a full crash

We only want the crash reporter to start if we also have some crash information so we check for the “-CrashGUID” argument. With that our application entry point looks like this.

App.xml.cs

using System.Windows;

namespace CrashReportClient
{
    public partial class App : Application
    {
        protected override void OnStartup(StartupEventArgs e)
        {
            base.OnStartup(e);

            // check the command line arguments whether we have been launched from unreal after a crash
            bool shouldStart = false;
            bool onlyExceptionTriggered = false;
            foreach (var arg in e.Args)
            {
               if(arg.Contains("-CrashGUID"))
                {
                    shouldStart = true;
                }

               // check whether this was only a exception and not a real crash
               if(arg.Contains("-Unattended"))
                {
                    onlyExceptionTriggered = true;
                }
            }

            if(!shouldStart || onlyExceptionTriggered)
            {
                Shutdown();
            }
        }
    }
}

The next step is to retrieve the crash data so we can send it.

// we get crash data

// we get our userDescription from our textbox
string userDescription = CrashDescription.Text;

string logText = String.Empty;
string xmlText = String.Empty;
byte[] miniDump = new byte[0];

// we get the environment independent appdata local path
string path = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);
path += "\\[YourGameName]\\Saved\\Crashes";

// first we get directory where the crash data is stored
DirectoryInfo crashDir = new DirectoryInfo(path);
if (crashDir.Exists)
{
    // make sure our data is up to date
    crashDir.Refresh();

    // we get the directory where our current crash data is stored (this is just the newest folder in the directory)
    var dir = crashDir.GetDirectories().OrderByDescending(d => d.LastWriteTimeUtc).First();
    dir.Refresh();

    FileInfo[] fileBuffer;

    // first we get the log data
    {
        fileBuffer = dir.GetFiles("*.log");
        if (fileBuffer.Length > 0)
        {
            FileInfo logFile = fileBuffer.First();
            logText = File.ReadAllText(logFile.FullName);
        }
    }

    // now we get the xml log data
    {
        fileBuffer = dir.GetFiles("*.runtime-xml");
        if (fileBuffer.Length > 0)
        {
            FileInfo logFile = fileBuffer.First();
            xmlText = File.ReadAllText(logFile.FullName);
        }
    }

    // now we get the minidump
    {
        fileBuffer = dir.GetFiles("*.dmp");
        if (fileBuffer.Length > 0)
        {
            FileInfo dumpFile = fileBuffer.First();
            miniDump = File.ReadAllBytes(dumpFile.FullName);

            // we compress the minidump
            miniDump = Compressor.Compress(miniDump);
        }
    }
}

Our four variables now contain all the crash data we want to send. One thing to note is that we compress the minidump. A minidump is generally around 360kb in size, which is not that large but we can compress that down to roughly 90kb. The Compressor ist just a simple class which uses the .Net implementation of the deflate algorithm.

Compressor.cs

using System.IO;
using System.IO.Compression;

class Compressor
{
    // compresses the provided data with the deflate algorithm
    public static byte[] Compress(byte[] data)
    {
        MemoryStream outStream = new MemoryStream();
        using (var compressor = new DeflateStream(outStream, CompressionLevel.Optimal))
        {
            compressor.Write(data, 0, data.Length);
        }

        return outStream.ToArray();
    }

    // de-compresses the provided data with the deflate algorithm
    public static byte[] DeCompress(byte[] data)
    {
        MemoryStream inStream = new MemoryStream(data);
        MemoryStream outStream = new MemoryStream();

        using (var deCompressor = new DeflateStream(inStream, CompressionMode.Decompress))
        {
            deCompressor.CopyTo(outStream);
        }

        return outStream.ToArray();
    }
}

And with that all the crash data is ready to be sent. And that concludes the crash reporter. We send the data to a database but you can use whatever bug tracking solution you want to use.

Preparing a Shipping Build

Now the only thing left is how to handle a shipping build. For Teal we always package a shipping build with debug files.

This includes the .pdb files we need to read the minidumps. Then we prepare a build for distribution by deleting the manifest text files and all .pdb files. And finally we replace Unreal’s CrashReportClient.exe with our own. Now your build is ready for distribution and players can log crashes.

Custom Unreal Engine 4 Crash Reporter

by Martin Kernjak

Unreal Engine Custom Crash Reporter

App.xml.cs

App.xml.cs

Crash Data

Engine.ini

The Crash Reporter

App.xml.cs

Compressor.cs

Preparing a Shipping Build