Many native application crashes can be attributed to poor memory management, memory leaks, or race conditions. These errors often remaining undetected for long periods, before causing occasional, difficult-to-diagnose crashes. Tools like Valgrind Memcheck, AddressSanitizer (ASan), ThreadSanitizer (TSan), and Dr. Memory are used early in the development process usually before production testing to identify these issues.
These tools are immensely valuable! But as anyone who’s used them knows, they produce large, semi-but-not-really structured reports that make it difficult to know if you’re dealing with a benign issue or a serious problem, and to get everyone needed to debug an issue on the same page.
We’re happy to introduce the newest Backtrace feature, the Memory Error Tracker, to make these reports as useful as crash dumps. Backtrace now automates collection of memory error reports, parses the data into structured items that you can query across or visualize, and deduplicates and classifies the errors so you can easily figure out which errors are most important to work on.
We built this feature for ourselves first. But now that it’s part of the product, we hope it will help your engineering team build a repeatable process around detection and resolution of memory errors.
Let’s take a look and see what teams can do once they start using Backtrace to collect and action on their Valgrind, ASan, TSan, and Dr. Memory reports.
Ingest, parse and index reports
Memory error reports are usually created as a result of Continuous Integration/Deployment pipelines that invoke memory analysis tools like Valgrind Memcheck, Address Sanitizer, Thread Sanitizer and Dr. Memory, or by individual engineers running their own tests on their own branches. Backtrace Memory Error Tracker natively ingests, parses, and indexes these reports.
Learn about how we test and generate reports at Backtrace in our engineering blog post here!
Backtrace will assign an ID to each report as well as to individual errors inside, so that you can query both within or across reports and see the error history of individual tests.
Reports can be submitted directly to Backtrace via HTTP. Using curl, you can further attach additional attributes, such as appBuildNumber for identifying which build or version a report was run against.
$ curl -s --data-binary @asan.log.28706 '<submissionURL>?application=asan&appBuildNumber=0.0.28'
Make the error data more useful
Once parsed and indexed, engineers can perform triage reports and decide how best to resolve issues. Backtrace has developed a large number of features to help engineering teams prioritize and handle crashes and exceptions, and these same features can be used to build a repeatable process for handling memory and thread errors spotted by Valgrind, ASan, TSan, and Dr. Memory.
- Merge issues with the same root cause – Backtrace’s database allows you to merge multiple errors that have they same root cause into a single issue for further tracking and resolution.
- Triage by classification, application name, or other dimensions such as leaked bytes – Backtrace’s automated classification system can tag possible leaks, direct leaks, race conditions, etc so that you can more easily focus. You can also use extensive query and filtering operators allow you to search callstacks or other attributes using contains, exact match, regex, inverse-regex, and more,
- Take automated actions such as open tickets or notify appropriate team members – Backtrace allows you to define rules, or use ad-hoc actions, to open tickets in issue trackers like Jira or communication channels like Slack.
- Exclude benign issues in favor of real leaks – Rather than manually editing suppression lists or black lists, Backtrace offers a Mute feature to declaratively specify errors that you want to ignore.
- Visualize errors related to certain subsystems within the code – Backtrace makes novel use of the flame graph visualization to make it easy to identify errors seen in certain code paths and call stacks.
“I integrated our test suite, which runs under ASAN, in just a few minutes. Our team was able to identify 4 important memory errors we had previously missed and was able to quickly narrow down on high impact memory leaks based on subsystem. Now, we have a repeatable process and have memory error tooling integrated directly into our CI framework so we get proactive alerts on anything that might have slipped through the cracks.” Samy Bahra, CoFounder/CTO, Backtrace
Save money by identifying and fixing memory issues before they cause crashes
A study from the IBM Systems Sciences Institute showed that it is up to 100x more expensive to fix a software defect after the software has been released. This is one of the key reasons we’ve been working to bring the power of Backtrace to the kinds of issues you run into earlier in the development process. Integrating with tools like Valgrind, ASan, and TSan, helps us identify defects early in development, before software gets into the hands of customers. This is crucial to developing a “shift-left” strategy, where testing and issue identification are automated and pushed as early into the Software Development Lifecycle as possible.
Customers use Backtrace to build dashboards to better understand the right issues to work, the ones with the most impact on users. Backtrace’s saved views enable you to:
- Determine how different leaks contribute to waste – An indexed leaked.bytes field, and aggregations such as sum, mean, and range, allow engineering leads to understand how many bytes are actually being leaked by different call sites, so they can narrow down their focus areas.
- Understand which commits or versions introduced a leak – By tracking commit SHAs and versions associated with each submitted report, Backtrace makes it easy to see which commit or version an error was first seen in.
- Combine issues from various tools in a single view – Backtrace allows you to combine the results from reports of various tools into a single view. This allows engineering teams to truly utilize all the best tools for identifying memory and thread errors and make appropriate fixes from one single UI.
The Backtrace Memory Error Tracker is available today! Learn more by signing up here, or begin submitting reports to your existing Backtrace instance to triage and fix the most important errors before they turn into production crashes.