How to analyse Test Results and “deflake” your Flaky Tests?

In one of our previous posts, we spoke about the causes of flaky tests, why such unpredictable behaviour can happen and how our test results are affected in consequence. Let us sum up the main points:

  • Non-deterministic/Undefined behaviours
  • Broken Test Data
  • Order of your tests
  • Extensive Tests
  • AJAX calls
  • Flaky 3rd party integration
  • Environment issues
  • Timing

Let me introduce you to two Jenkins plugins, which can help in analysing and reducing the flakiness of your tests:

  • Flaky Test Handler
  • Test Results Analyzer

Flaky Test Handler plugin

Unfortunately, this plugin is not maintained actively, but it can still support you in handling your flaky tests. You can:

  • rerun your whole failed build,
  • re-execute the failed tests only
  • aggregate statistics of tests over Git revisions as well

To install this plugin to your Jenkins, navigate to Jenkins -> Plugin Manager page and install the plugin from there. The only prerequisite is, that you need to use JUnit, since it generates the test report from the JUnit’s XML file. The recent version is 1.0.4 (https://plugins.jenkins.io/flaky-test-handler).

Flaky Test Handler
Flaky Test Handler

To run the plugin, just open your failed Jenkins job and click on the “Deflake this build” item on the left side.

Flaky Test Handler: Options
Flaky Test Handler: Options

As you can see from the screenshot, this plugin doesn’t come with hundreds of features, all it can provide is the re-execution of the build.

One of the main features of this plugin is that you can rerun the Failed tests N times (~rerunFailingTestsCount). If in any of those N times the build passes, the execution will stop and the appropriate tests will be marked as “Flake”.

Deflake Build
Deflake Build

Test Results Analyzer plugin

Let me introduce you to Test Results Analyzer plugin, which is way more powerful than the Flaky Test Handler one.

When the build fails, and you need to investigate the cause of the failure and find the exact test which caused the build marking red, this plugin comes very usefully. Since it’s very time consuming and cumbersome to investigate the failures, this plugin solves your issue by showing the build result history of test-classes and test packages in a tabular tree format.

Similar to the previous installation steps, this plugin can be found on the Jenkins’s Plugin Manager page, the recent version is 0.3.4.

Settings

Once the installation is successful, you can configure the plugin on Jenkins => Configuration page:

Test Results Analyzer: Settings
Test Results Analyzer: Settings

One of the significant advantages of this plugin is that you don’t need to configure your tests or your test reports, there’s no need to install other dependencies; it uses Junit’s (or TestNG’s) default XML test report. Just run your tests in Jenkins, enable the “Publish JUnit results” for your Jenkins job and from the left menu choose the Test Results Analyzer item to analyse your results:

Test Results Analyzer: Menu
Test Results Analyzer: Menu

The main page

Test Results Analyzer: Main page
Test Results Analyzer: Main page

As you can see from the screenshot above, the page is very informative; it gives us lot of insights to understand our Test Results. Test Results Analyzer has three sections:

  • Options
  • Table with build numbers and tests
  • Graphs (Line, Pie, Bar)

Options

Test Results Analyzer: Options
Test Results Analyzer: Options
  • Number of Builds: If you don’t need to check the results for all the builds, you can specify the number of builds here, so the test analyzer will analyse only the last X builds.
  • Display run time for each test: If you’re not interested in the execution time/ runtime, you can disable this option.
  • The plugin offers three kinds of charts by default: Line, Bar and Pie, and all of the graphs are displayed by default. Obviously, you don’t need to use all of them, feel free to select the ones, which are useful in your investigation and fits your needs.
  • The last option is “Chart Data Type,” where you can set whether you want to display the Test Results (Passed / Failed) or the Test Runtimes on the graphs.
  • Although it’s not stressed enough, you can export the test results in CSV format as well. This function will create a CSV file, with the test package names, test class, test name, and with all the specified builds + run times. This extra functionality comes very handily when you want to analyse the test results in another tool, or create more complex graphs in Excel or Tableau.
  • If you have hundreds or thousands of tests, and you want to check the results for one specific test, just type in the name of the test into the Search field, and the magic happens.

Test Results in table format

Test Results Analyzer: Test Results
Test Results Analyzer: Test Results

This section is the most important one since here you can see the test results for all your builds. Also, if the appropriate test passed last time, but it failed now, you can see a warning triangle in the “New Failures” column next to your test.

The graphs display test results for all the tests by default. If you don’t want to see all the results, just select the tests in which you’re interested (“Chart” column).

Graphs

We all know, that “one picture can say more than a thousand words”. The graphical report gives us a clear picture of the execution, failed builds, failed tests. The plugin gives you an option to export the displayed graphs to PNG or SVG as well. These images come very usefully when you need to include your test results and charts in your Test Report.

Conclusion

If you want the quickest “solution” then the first plugin can be enough for you. Sometimes the re-execution really helps, especially if your test is flaky because of some timeout issues.

However, if you need to analyse test results in detail and want a definitive solution, go for the second plugin.

If you’re using Jenkins and Jira at work and would like to get notified when the above plugins were not enough to solve your flaky test problem, check out one of our past posts: Failing Builds? Integrate Jenkins & Jira for a sunny day

Have you already used these plugins? What is your experience with them? Have you found better plugins for Test Analysis? Don’t be afraid and share it with us 😉

Adrian Benko

Quality Assurance and Testing Advocate with 10+ years of experience, passionate about quality and education.

Leave a Comment