All teams using minidumps for crash reporting need to make use of debug symbols. Symbols play an important role in callstack generation, as they allow a dump’s memory addresses to be associated back to the appropriate function names. Private symbol servers, and crash reporting solutions that integrate with them, eliminate the need to manually store and retrieve symbols post application build.
The following will cover an overview of what symbol servers are, why they’re useful, and how they can be created.
What’s a symbol server?
Symbol servers are servers that store debug symbols for individual users and/or crash reporting solutions to access. In this discussion, we’ll focus on http(s) web servers.
Widely used symbols (think: Windows, Mac, Electron, and more) are published on public symbol servers (Microsoft’s, Mozilla’s, and Electron’s, respectively). These symbol servers allow anyone to download symbol files instead of generating, storing and finding them independently.
Symbol servers are straightforward to build and the time initially spent is well worth its return. When configured to do so, custom http(s) symbol servers allow users and/or tools to share symbol files specific to their application (.pdb, .sym, .dSYM, .dll, .exe – you name it).
Why is a symbol server useful?
The most pronounced benefit of symbol servers is putting an end to the cumbersome process of tracking down symbol files from build machines and storing them locally (or on another machine) in order to view a meaningful callstack while debugging a crash. Symbol servers can maintain many versions of the same symbol file, and debuggers that support using them can find the appropriate symbol file efficiently. Using a properly configured symbol server, especially in combination with a crash reporting solution, allows users to view a meaningful callstack more quickly, so they can get right to prioritizing and debugging crashes.
How can I create a symbol server?
Once symbol files are generated from a build, Windows symstore can be used to store the files in the Microsoft standard symbolstore structure. An example invocation of symstore looks like:
<WindowsSDKDir>Debuggersx86symstore add /r /f <TargetDir>*.* /s <C:pathtomycustomsymstore> /t "<AppName>" /c "<Comment>" /v "<version>"
replacing the bold values.
Here’s an example generating a symbol store for a test application:
c:Program Files (x86)Windows KitsDebuggersx86>symstore add /f C:UsersmbreauxsourcereposConsoleApplication1Debug*.* /s c:pathtomycustomsymstore /t "Test" /c "Test" /v "126.96.36.199" Finding ID... 0000000004 SYMSTORE: Number of files stored = 2 SYMSTORE: Number of errors = 0 SYMSTORE: Number of files ignored = 0
More information about this tool is available in Microsoft’s documentation. It also can be helpful to know that this can be added as a post build step in Visual Studio. To do so, add the appropriate command in the project’s property pages, under Build Events | Post-Build Event | Command Line.
Once the symbols are generated and stored in appropriate structure, popular candidates for a web frontend include nginx, AWS, and Google Cloud buckets.
All Backtrace instances are integrated with public symbol servers to pull Windows OS, macOS, ATI (gpu) and Electron symbols. Enterprise customers can integrate their own private symbol servers, making symbolification completely hands off.
Private symbol servers integrated with Backtrace need to meet the following requirements:
- Follow Microsoft’s standard symbol structure (ie
- A popular option to generate the files in this structure is using Windows symstore
- The symbol server must also have a web frontend to serve up the files via http(s)
- Popular candidates for this are nginx, AWS, and Google Cloud buckets
- Your Backtrace instance must have access to the server (ie, Firewall or other network rules may need to be adjusted)
Taking advantage of this functionality frees teams using Backtrace up to spend more time addressing problems important to them. Specifics on how to integrate your custom symbol server with your Backtrace instance are laid out in our “Connecting to symbol servers” help doc.