Minidump Support with Backtrace
A minidump file can be incredibly useful to software engineers who need to debug an application crash or fault. That’s because minidumps can contain the callstack along with a section of the application’s memory at a point in time.
Most common minidump generators, like Crashpad, will contain the application’s threads and their callstacks, the stack space/region for each thread, register values, attributes (environment variables, system information, and custom metadata), and a listing of loaded modules with their metadata.
Backtrace has industry-leading minidump support to store, process, and manage .dmp files that are generated from applications running on Windows, macOS, and various versions of Linux, including Android. We also maintain our own distribution of Crashpad with binaries for Windows and source code available! Let’s dive in and take a look at Backtrace’s minidump feature set:
Backtrace supports a variety of methods to capture and submit minidumps.
- Our recommended approach is to use the Crashpad or Breakpad libraries for C / C++, Objective-C, or Rust applications to generate a minidump.
- If you use Visual Studio for Windows C++ development, check out the Backtrace’s Extension to Visual Studio to simplify the setup of Crashpad.
- Backtrace provides a distribution of Crashpad (binaries for Windows, Source Code) that address several bugs and capture more system level information than the standard versions.
- Backtrace fully supports the open source Breakpad and Crashpad libraries.
- macOS developers who write in Objective-C have methods available to them to generate and submit minidump files for analysis, including Breakpad as previously mentioned.
- Electron developers use the built-in crashReporter to generate data for easier debug of their Electron based applications.
- Game developers using the Unreal Engine can generate and submit minidump files using the Unreal Crash Reporter
- C# developers can use Backtrace’s C# Library to attach minidumps to exception reports when required.
Once generated, a minidump file can be submitted to Backtrace over HTTP(s). Most minidump generators have built-in mechanisms to upload minidumps to a specified HTTP(s) endpoint, like Crashpad’s url parameter, Electron’s crashReporter, and our Unreal plugin.
System level metadata, such as process uptime and memory usage, or custom metadata, such as graphic card driver, application mode, or version information, can be associated with the minidump. This instance-level metadata provides contextual data points for investigating root cause.
Backtrace automatically aggregates every attribute across all instances of a crash, giving you impact analysis and aggregate root-cause investigation out-of-the-box.
Backtrace also accepts generic file attachments, such as log or configuration files, to be associated to a crash. This additional contextual information can be invaluable during debugging as the engineers are looking to see what happened in the recent past, and what additional configurations might be set.
One of the main drawbacks of minidumps is that they require processing (i.e. symbolication), before they are human-readable. Symbolication requires debug symbols which are necessary to extract the full callstack (function names, line numbers), which ultimately feeds into things like deduplication.
Most platforms require developers to upload their symbol files in advance if they want to create symbolicated crashes and extract the callstack but Backtrace gives its users flexibility:
- Automatic retrieval of third-party symbols for Windows, MacOS, Electron, and more.
- Integration of your private symbol store, including S3 backed stores, into our automatic symbol retrieval system.
- A turn-key symbol management system to help engineers identify and apply missing symbols, even after the minidumps are uploaded to Backtrace.
- Native support for.pdb .sym, dSYM and DWARF, and an internal proprietary format to improve storage utilization and performance.
- More accurate callstack unwinding, including improved support for inline functions. Check out the Improved Windows Debug Accuracy section of the April 2019 Product Update for more details.
- Heuristic algorithms to group similar crashes beyond simple callstack based grouping.
These processing capabilities have saved our clients valuable hours, allowing them to dedicate more time to triaging and solving the issue.
Use the Backtrace Web Debugger to view contextual metadata associated and symbolicated callstacks for all application threads.
Minidump Support and Management
Rather than simply deleting the minidump after initial processing, Backtrace offers numerous controls for how many dump files to retain, for how long, and what you can do with them.
- By default, minidump files are retained in order to be available for reprocessing when needed.
- Data retention policies ensure minidumps and/or metadata is removed as required by your data hygiene policies.
- For minidumps that are retained, engineers have the ability to download dump files for further analysis in IDEs or debuggers such as Visual Studio or GDB. This allows them to be able to see crash details alongside source code, or dig deeper into variable information.
- Data scrubbers allow administrators to remove sensitive information from minidump files before they are written to disk to further protect privacy and limit exposure.
- Minidump sampling allows you to save all relevant metadata from your minidumps while not bearing the cost of storing every minidump file. Minidump sampling is performed such that minidumps are no longer required after the callstack has been symbolicated and deduplication performed. Sampling can be configured with multiple keys (such as app version, data center, or operating system), ensuring a flexible approach that enables you to store dumps based on app version or other specified key.
Backtrace has been working with customers to improve the debugging process for minidump crash reports, and the aforementioned features are a direct result of those engagements. We learn from software engineering teams who are building and supporting native applications and embedded devices using C/C++, Electron, and Objective-C on Windows, macOS, Linux and more. Recent improvements have focused on better system statistic extraction on Linux (including virtual memory details and cpu thread count), improved deduplication for Windows, MacOS, QT, Electron, Chromium Embedded Framework (CEF), LUA, wxWidgets, and the addition of new symbols servers and whitelist controls to simplify symbolication. We’re confident that teams looking for industry leading minidump support need to be using Backtrace.
As we look ahead, Backtrace intends to continue to improve deduplication and symbolication, offer native symbol support for additional formats, and provide more binary builds of Crashpad and Breakpad crash reporting libraries for commonly supported environments. Please reach out to us if you have questions, or just want to chat about how we can help you reduce the time to detection and resolution of the most difficult to diagnose crashes and faults.
Introduction to Symbolic Debuggers
Join us for a complimentary 30-minute webinar with Samy Al Bahra, CTO & Co-Founder Backtrace on Wednesday, May 22, 2019 at 2:30 pm, and learn about Symbolic Debuggers, how they work, when they don’t, and potential mitigations.