November has been an exciting month at Backtrace. After a significant release in October (PII Data Scrubbers, File Attachments, No Hassle Symbol Management, and more), we recently announced a beta of our native support for the Unreal Engine 4 (UE4), Beyond UE4, we spent most of November focusing on the web debugger, performance and optimizations:
- Performance – Backtrace is now more than 5x faster for page loading when dealing with large number of crashes (200k+) for lists and histogram visualizations.
- Retention Policies for Dump Files – Automate the removal of dump files for security or space saving purposes, and retain attributes and metadata to maintain history and impact analysis.
- Web Debugger – More granular Identification of Missing Symbol and Guessed Call Frame are now available.
- The performance and consistency around concurrent symbol uploads has been improved.
Let’s take a peak behind the curtain to see what’s in store.
5x Performance Boost for Large Amounts of Data
Backtrace customers are starting to collect, process and analyze crashes and dump files from more and more customer devices. We’re beginning to see multiple customers scaling up to 3 million crashes per month and more. Responding to customer demands, our engineering team has begun to design some new infrastructure to allow Backtrace to meet their needs.
In November, we enabled two new core features to support the larger numbers of crashes under management.
Server Side Sorts and Limits
One of the primary capabilities we recently added was server side support for sorts and limits. Previously, the client would download all the data it needed and then handle sorts and limits on its own. This works fine until there is too much data to download and process in the client tier. Now, the client and server work more closely together to provide higher performance in response to queries like “Give me data for the top 10 crashes over the last month”
Bounded Histogram Operator
One of the key differentiators of the Backtrace platform is its ability to perform aggregation and grouping based on various dimensions of crash data. This allows developers and engineering managers to explore a particular issue more deeply and better understand its impact, posing questions like:
- What is the distribution of users being impacted by each unique crash?
- What data centers have seen a significant amount of downtime over the past two weeks?
Aggregations, however, can yield large data sets. For example, a histogram on device serial numbers for a popular consumer application yields over 200,000 unique values. Performing a distribution analysis on this data set can become a bottleneck if the system is not designed properly. Backtrace addresses this situation with a new bounded histogram aggregation operator. The new operator allows for the backend to limit the output to the first 64 bins or other custom supplied limit.
Retention Policy for Dump Files
Backtrace provides a default retention policy for 90 days for our hosted customers. This means that dump files and their associated metadata are available for analysis in the system for 90 days. For our enterprise customers, Backtrace offers the ability to customize retention schedules. System administrators can configure a maximum age to retain crash objects for. Some will retain for a year or more.
Recently, we had a few enterprise customers request the ability to set retention schedules for dump files, but retain metadata and callstacks for further analysis. This is typical for organizations with mass deployed applications (100,000+ production versions) and large number of crashes (500,000+ a month). These companies usually have corporate policies around retention of crash dumps or want to save on storage costs.
- Corporate Policies – Many organizations implement PII Data Scrubbing to remove personally identifiable information from their systems. Some organizations have additional policies to remove any files submitted by customers after a few days. Rather than lose the valuable metadata surrounding the crash, it is preferable to retain attributes to allow for long term trend and impact analysis.
- Storage cost savings – Dump files can be large (ranging from a few hundred KB to many MBs and GBs). With Backtrace’s sophisticated de-duplication, there are efficiencies gained when individual dump files are removed while the callstack and other attributes are retained.
Web Debugger Improvements
It is now easier to identify guessed frames and missing symbols as they relate to certain portions of the callstack. Check out the tooltip in the Web Debugger below to see what it looks like. If you need a refresher on what our Web Debugger is used for, check out our docs here.
Not only did we improve our Web Debugger, but the addition of the new performance features greatly reduces the amount data of serialized and transferred to the client, enabling a much snappier experience. And retention policies for dump files enables a whole new level of control for system administrators. Taken together, these two features provide a great foundation to build upon, as we continue to optimize for larger data sets and incoming streams.