How THQ Nordic Cut Crash Detection and Resolution by 50%

About THQ Nordic

THQ Nordic is an international computer and video game publisher with external and internal development studios in Europe and the US. They publish games on PC, XBox One, and PS4, and use multiple engines, including UE4 and Unity.

Challenges:

  • Crash detection took days and required players to do something
  • Developers were spending >50% of their time debugging crashes
  • Crash prioritization and debugging was mostly a manual process
  • Did not have cross-platform visibility across studios and platformsPC, Unreal Engine 4, and Console.

Results with Backtrace:

  • Automated crash capture, reducing detection time by >90%
  • Automated symbolication, deduplication and, analytics enable accurate bug prioritization across studios and platforms
  • Unified crash-related context and files in one interface, reducing resolution time by >50%
  • Achieved 100% visibility across platforms and studios

“Unfortunately every software has crashes and every developer should be able to fix these. With Backtrace, we get everything as it happens and in a way where we can filter through the data nicely.”

Reinhard Pollice, Executive Producer at THQ Nordic

THQ Nordic is an international computer and video game publisher with external and internal development studios in Europe and the US. They publish games on PC, XBox One, and PS4, and use multiple engines, including UE4 and Unity.

“With Backtrace, we have 100% visibility into all of our crashes and have all the data we need to quickly identify the root cause. This has doubled our developer velocity and significantly improved our stability and TTM.”

When platform tools aren’t enough

Prior to Backtrace, THQ Nordic relied on the crash reporting tools provided by platforms like Steam and Playstation. As the company grew, they ran into some.. challenges:

Isolated Data: Crash data for each game was isolated between the respective platforms and it’s crash reporting tools, which slowed down the entire capture, prioritize, and resolve cycle.

Limited Visibility: Playstation provided a way to automate uploading symbol data during their submission process. Steam had a manual process, but it only worked for 32-bit applications and not for 64-bit applications. Without getting 100% crash visibility, the team regularly missed critical crashes that could turn into bigger issues as a games scaled.

Manual Debugging: THQ Nordic was manually gathering crash reports and prioritizing what to work on based on reproduction rate. This became unsustainable very quickly after the company started to scale game development to multiple titles across multiple platforms, including Steam, Playstation, and Unreal Engine 4, at once.

“We looked for a crash reporting solution that supports modern C++ developed applications like our games and UE4, and soon realized that Backtrace I/O is the best pick. Backtrace gives us unlimited scale in terms of platforms, users, and analytics.”

Immediate crash telemetry reported across all studios while maintaining data isolation

THQ Nordic - FeatureAs a major game company, THQ Nordic has a complex development, distribution and support process. There are multiple studios involved with any given project, and all of them have both internal and external developers working on them. To maintain confidentiality and protect data, it’s crucial that only developers working on the respective projects have access to necessary data.

To ensure this, Backtrace was provided a multi-tenant environment so each team could use the service for their own projects without interfering with any others. Only authorized staff can access cross-studio data. This way, trends can be spotted across the whole organization while still providing every team with it’s own presonalized experience.

Improving developer efficiency and player experience

Slack_THQ

Reducing detection time: Before Backtrace, THQ Nordic relied on users to report crashes. This was fine until development progressed and the # of users scaled. Once a crash occurred, users had to send the crash report to THQ for analysis and these users were not inclined to take time out of their game-playing to do this. Developers had to spend time trying to collect these crashes and it often took days for a user to submit crashes. THQ Nordic was missing out on bugs that would resurface as bigger issues as games grew.

With Backtrace, they are able to get immediate visibility into all crashes from dev, QA, orproduction. Different teams use Backtrace’s integrations with messaging apps to receive immediate notifications and integrations with ticketing services to automate the triage process.

“Backtrace cut our MTTD by more than 90% by automating our crash aggregation. Now our developers can focus on building new features rather than chasing down bugs.”

 

Reducing resolution time: The majority of the debugging work starts once a crash is received, and this entails attaching debug symbols manually and then doing manual analysis work to prioritize the crashes to debug. THQ Nordic dramatically reduced this facet of manual work with Backtrace’s automated symbolification, deduplication, and analysis capabilities.


Once a crash is received, Backtrace automatically attaches debug symbols to the binary, which ensures the information is human readable (call stacks etc). Backtrace then groups similar crashes together into a unique fingerprint. Because of these features, THQ Nordic developers only have to look at <10% of their total crashes every month. Backtrace’s automatic classifiers (invalid-read, invalid-write, out-of-memory etc.) further reduce debugging time by pushing likely causal factors to the surface.

 

Web Debugger populating call stack, attributes, register values etc.

By combining these automation capabilities and the Web Debugger which collects all the context and files developers need in one place, THQ Nordic was able to reduce crash resolution time by more than 50%.

“Now in Backtrace we can see exactly how often a certain issue occurs and we can directly get all the information to fix it.”

Impact Analysis: It’s important to diagnose the impact of crashes across versions and other meaningful attributes. Using Backtrace’s Analtyis Engine, THQ Nordic runs queries against their crash data and easily gets answers to questions like: What are my top 10 crashes?, What versions were impacted by a crash?, and Which crash had the highest user impact?

 

THQ Nordic can now quickly prioritize the top 10 crashes they want to debug. They’re also able get a handle on the impact of crashes across version, platform, users, etc. By going beyond the top 10 crashes, they are able to identify trends and measure the stability of games over time.

Backtrace can deliver the same benefits to your team as it does for THQ Nordic. See for yourself today with a free 30 day trial!

By | 2019-02-21T17:37:18+00:00 May 31st, 2018|Backtrace, Customers|