THQ Nordic Reduced MTTD and MTTR by more than 50% with Backtrace I/O

Backtrace I/O helps high velocity gaming studios automate their debugging process to fix bugs faster and improve end user experience.

Challenge:

  • MTTD (Mean Time To Discovery) took days and not all crashes were being reported by end-users.
  • MTTR (Mean Time To Resolution) had to be reduced because developers were spending more than 50% of their time debugging.
  • Crash management process was manual, which meant application stability was not keeping up with the development of new features.
  • Needed 100% crash visibility across studios and platforms like PC, Unreal Engine 4, and Console.
  • Support for both 32-bit and 64-bit C++ applications.

Results:

  • Automated crash aggregation, symbolification and deduplication, which reduced MTTD by more than 90%.
  • Leverage Query Builder to prioritize “Top 10 bugs” and the Web Debugger to investigate and resolve bugs faster, which reduced MTTR by more than 50%.
  • 100% visibility across platforms and studios, which ensures no crashes fall through the cracks.
  • Have a single repository for crash data across all studios and platforms.
  • Have complete isolation between studios, which includes both external and internal developers.

“Unfortunately every software has crashes and every developer should be able to fix these. With Backtrace, we get everything as it happens and in a way where we can filter through the data nicely.”

Reinhard Pollice, Executive Producer at THQ Nordic

THQ Nordic is an international computer and video game publisher with external and internal development studios in Europe and the US. They publish games on PC, XBox One, and PS4, and use multiple engines, including UE4 and Unity.

Reinhard is an Executive Producer at THQ Nordic and he’s been working there for almost 8 years. He oversees over 10 AAA titles, and also leads both business and product development.

“With Backtrace, we have 100% visibility into all of our crashes and have all the data we need to quickly identify the root cause. This has doubled our developer velocity and significantly improved our stability and TTM.”

When platform provided crash reporting tools stops scaling

Isolated Data: THQ Nordic was using crash reporting tools that platforms like Steam and Playstation provided. Through this, they realized the value in gathering crash reports directly from consumers and internal test runs. This however kept the crash data for a game isolated between the respective platforms and it’s crash reporting tools, which impacts MTTD, and limits the ability to prioritize, which impacts MTTR.

Limited Visibility: Playstation has a way to upload symbol data during their submission process. Steam had a manual way of uploading necessary data but that only worked for 32-bit applications and not for 64-bit applications. Without getting 100% crash visibility, it was possible to miss out on a critical crash that could be a bigger issue down the road as a game scales.

Manual Debugging: THQ Nordic was manually gathering crash reports from QA testers, and based on reproduction rate, they set the priority to fix an issue. Although this was valuable, THQ Nordic were expanding, and as more projects started spinning up at a higher velocity, these vanilla tools could not meet the requirements of their new projects. For THQ Nordic, the most important factor is that a crash reporting solution works with their current application standards, which is mostly 64 bit applications, and that it has integrated solutions for Unreal Engine 4, which is now vastly used with their projects.

“We looked for a crash reporting solution that supports modern C++ developed applications like our games and UE4, and soon realized that Backtrace I/O is the best pick. Backtrace gives us unlimited scale in terms of platforms, users, and analytics.”

Get immediate crash telemetry across all studios while maintaining data isolation between individual studios

THQ Nordic - Feature

As a major game company, THQ Nordic has a complex development, distribution and support process. There are multiple studios involved with any given project, and all of them have both internal and external developers working on them. To maintain confidentiality and protect data, it’s crucial that only developers working on the respective projects have access to necessary data.

Therefore Backtrace was implemented in a multi-tenant environment which was necessary to make sure that each team has the freedom it needs to use Backtrace for their own projects and without interfering with any others.

Developers can work on their own projects, and only authorized personnel can access data across studios within the UI. This means data across studios can also be correlated to identify trends and prioritize resources.

Improving developer efficiency and end-user experience

Slack_THQ

MTTD: Before Backtrace, THQ Nordic relied on users to report crashes which is fine during the early stage of a game, but as development progressed and the # of users scaled, this became problematic for a number of reasons. Once a crash occurred, users had to send the crash report to THQ for analysis and these users were not inclined to take time out of their game-playing to do this. This was problematic because developers had to spend time trying to collect these crashes and it often took days for a user to submit the crash. This also meant THQ Nordic was missing out on bugs that could resurface as a bigger issue as the game scaled in features and users. This not only meant a high MTTD, but it also meant game stability and TTM would be impacted.

Now with Backtrace, THQ Nordic is able to get immediate visibility into all crashes from dev, QA and production, and get immediate notifications of a crash. Many studios have custom integrations with chat applications to receive immediate notifications and automated triaging with their ticketing service.

 

“Backtrace cut our MTTD by more than 90% by automating our crash aggregation. Now our developers can focus on building new features rather than chasing down bugs.”

 

MTTR: Attaching debugging symbols and reducing noise by grouping similar crashes together are critical to reducing manual work. Majority of the debugging work starts once a crash is received, and this entails attaching debug symbols manually and then doing manual analysis work to prioritize the crashes to debug. THQ Nordic was able to reduce the manual work significantly because Backtrace automated symbolification, deduplication and analysis.

Once a crash is received, Backtrace automatically attaches debug symbols to the binary, which ensures the information is human readable (call stacks etc). Once this is done Backtrace groups similar crashes together into a unique fingerprint. Currently, THQ Nordic only has to look at less than 10% of their total crashes every month. As part of the debugging process, developers were able to leverage classifiers (invalid-read, invalid-write, out-of-memory etc.), which is automatically assigned by Backtrace, and the web debugger to get to the root cause faster.

 

Web Debugger populating call stack, attributes, register values etc.

By combining these automation capabilities and the Web Debugger, THQ Nordic was able to reduce MTTR by more than 50% while also fixing more bugs.

“Now in Backtrace we can see exactly how often a certain issue occurs and we can directly get all the information to fix it.”

Impact Analysis: Part of what studios need to do is measure the impact of crashes across versions and several other meaningful attributes. With Backtrace, THQ is able to leverage Query Builder to run deeper analysis on their crash data and ask meaningful questions like “What are my top 10 crashes?” “What versions were impacted by a crash?” “Which crash had the highest user impact?” etc.

 

THQ Nordic is able to use this tool to quickly prioritize the top 10 crashes they want to debug. They’re also able to run queries across multiple dimensions to measure the impact of crashes across version, platform, users etc. By going beyond top 10 crashes, THQ Nordic is able to identify trends in their crash data and measure the stability of their games over time.

Next Steps

If you’re interested in learning more please head over to THQ Nordic to check out the current and upcoming titles. And if you’re interested in learning more about Backtrace I/O please reach out to us or take the product for a spin with a free 30 day trial.

By | 2018-06-13T17:39:52+00:00 May 31st, 2018|Backtrace, Customers|