I have been looking for some test management jira add-ons lately. I’ve noticed there is a lot of materials about Zephyr or Xray. However, I couldn’t find any comparison between TestFLO and Xray. Therefore, I’ve decided to discuss it in this article.
#jira | #testflo | #xray | #testmanagement | #softwaretesting
How did Jira add-on research start?
I’ve made a comparison in a few steps described below.
- First of all, I installed a trial version of Jira locally. I had an Atlassian account created before so I skipped this step.
- Next, I looked for add-ons in Atlassian Marketplace and installed them.
The most important aspects, that I’ve chosen to compare both solutions, can be found below.
Spoiler – That fight will be 8-round distance. It is the last chance to grab popcorn or some drink. Please read and enjoy…
Round 1. Issue types
There are six new issue types delivered as part of Xray, as you can see in the table above. However, there are only three in TestFLO add-on. Additional issue types work in a very similar way in both test management tools. Thanks to a bigger number of them, Xray ensures more options for test management. For instance, the user is able to create Pre-condition, which might be re-used in many different Tests. It’s not possible to achieve that in TestFLO as such issue type is not available.
Test Set is used to group Tests in some logical way like regression tests, security or those related to a specific component or feature. Test Sets are also used to speed up Test Execution or Test Plan creation.
There is only Test Plan issue available in TestFLO. It’s used for test executing purpose.
Round 2. Jira version support
The solution delivered by Deviniti supports Jira in version 7.0.0+, however Xray supports Jira in version 5.2+. Let’s keep in mind that the available features might depend on jira and test management tools version.
It was not part of the exercise to check functionality availability in different versions of the tool.
Round 3. Dashboard gadgets
The number of dashboard gadgets is a huge advantage of Xray, as there are eight built-in gadgets and none in the second tool. In case of TestFLO, it is possible to use jira built-in gadgets such as Pie Chart, Filter Results in order to present desirable statistics. Again, Xray is a much more powerful tool. It provides us with more possibilities to present different kind of metrics, which are very important in test management.
Round 4. Built-in reports
Built-in reports delivered by both tools are listed in the table above. This time again Xray contains more reports than TestFLO, however the difference is insignificant.
Round 5. Documentation
The quality of documentation should be considered very important during any toll research and should always be part of it. Clear and detailed instructions might be very helpful in case of tool installation and configuration. It would also be of use in troubleshooting.
💥First KO – Xray wins the round
In my opinion, Xray knocks TestFLO out in this respect. People from Xpand IT have created high quality documentation with a detailed description of all the components.
The main points from the documentations of both tools can be found below or you can check it by yourself.
- introduction with getting started and tests structure approach
- settings – global and project configuration
- working with steps
- about Xray
- what’s new – description of changes in different versions
- installation and configuration
- getting started with project organization, requirements and defects definition as well as test process overview
- user guide with a description of all issue type, reports and dashboard gadgets
- integrations and API
- roadmap of the product
- and much more
Round 6. Automated Tests support
Test automation support is included in Xray add-on, however the installation of additional paid TestFLO automation for Test Execution plugin is required in case of TestFLO. Below a quick overview of both solutions can be found. Bear in mind that it was prepared by looking at the documentation only.
- trigger execution of JUnit or Selenium tests on Bamboo or Jenkins automatically, by changing the status of Test Plan (or another type that is currently the parent of your Test Cases)
- integration with CI (Bamboo, Jenkins)
- many testing frameworks are supported, natively (e.g., Cucumber) or indirectly through the processing of execution reports
- most common workflow while dealing with automated tests is to write the automated tests code, execute them and then import the results and the test themselves to Xray
- generic Tests may be used as an abstraction of automated tests, regardless of the testing framework and the technologies/platforms used (most popular: JUnit, TestNG, NUnit, Robot framework)
- integration with CI (Bamboo, Jenkins, Team City)
- Xray also provides specific integrations for Cucumber, Behave, Robot, Xamarin, and other frameworks
- mobile tests in the cloud using Calabash and Xamarin Test Cloud are also supported
- very good and detailed documentation
💥Second KO – Xray wins the round
Xray provides many more options for different frameworks usage, CI integration and I think it might be easier to start thanks to great documentation.
Round 7. Pricing Cloud version
The table above shows differences in pricing for cloud version of the tools. As you can see, costs are the same when the number of users is not higher than 100. Please note that in this version TestFLO doesn’t support test automation. It means that TestFLO might be more expensive when test automation support is taken into consideration.
Xray is cheaper when the number of users increases. It is better visible when we look at the average price per user. I’m not sure why it has happened. Please shed more light on this aspect if you have more information.
Round 8. Pricing Server version
As you may have noticed in the table above, TestFLO is cheaper in case of server version. It is cheaper even if additional costs of automation plugin are included. However the difference is not significant in this case.
And the winner is… Xray
Taking into account all the points presented above, there is only one winner. In my opinion, Xray has won almost all of the rounds (more issue types, more built-in reports, more dashboard gadgets, better documentation and test automation support). It is a bit more expensive (server version only), however it ensures more functionalities and flexibility.
The result of the exercise may not be applicable in all possible cases. It is strongly recommended to do such research and make a comparison on your own in order to consider all your needs.
The exercise was performed based on installed add-ons and documentation delivered by Devinity and Xpand IT, which can be found via Atlassian Marketplace.