Successful Automation of All Mobile Test Cases in Three Months for the World’s Leading Business News Organization — The Financial Times
The Financial Times, one of the world’s leading business news organizations, is recognized internationally for its authority, integrity, and accuracy. Mobile is an important channel for the FT, driving significant FT.com traffic and a larger percentage of digital subscriptions. FT education products serve many of the world’s top 50 business schools.
The drivers for automation
A redesign of the FT mobile web application prompted the quality assurance (QA) team to rethink the way that software testing was being delivered in the organization. The key driver behind the review was that every new software build and release would take increasingly longer to test end to end. The sheer number and complexity of test scenarios meant that it could take a team of five manual testers an entire day to go through a complete test suite, before a new release was given the green light.
There were additional internal demands from the business, such as reducing time to market for new products, increasing end user satisfaction, and creating a closer-knit relationship between QA and development departments.
Furthermore, the proliferation of new devices and operating systems meant that a manual testing approach was becoming increasing less scalable, thereby encouraging the team to explore the option of a test automation tool to take the load off their functional and regression testing.
Jay Sethi, Head of QA, Financial Times, explains:
“The rapid growth of new technologies, devices and operating systems has a significant effect on our QA processes and poses a real challenge to meeting release deadlines. The device-agnostic nature of Eggplant allows our team to use one automation script to test the functionality of multiple devices at once, irrespective of manufacturer, operating system, or software version. Crucially, this allowed us to get our manual test cases automated much sooner than expected, lifting the burden from our team.”
The challenges and key requirements
The FT searched for an automation tool that would not only increase the quality of their builds, but also monitor the true end user experience via the graphical user interface (GUI). Furthermore the FT’s development team follows an agile development methodology, meaning that feedback from QA was required quickly in order to keep up with development bursts.
Before starting an evaluation of the various test tools on the market, the FT outlined several key requirements that would form the selection criteria. From a technical viewpoint the ability to test HTML5 content and native web apps for iOS were two critical factors. Ease of use was also very important to reduce the learning curve for manual test engineers and increase deployment speed. The ability to perform data driven testing on real devices was also a key requirement.
Chris Bower, QA Manager, Financial Times, adds: “Our requirements quickly narrowed the list of options to just a couple of tools. The decision on Eggplant was taken because it ticked all of the boxes technically and allowed our engineers the freedom to test every part of our mobile web app. From installation of the app via a native mobile browser, to checking that the FT’s news feeds were displaying correctly and the cache on the device could be emptied. The ease of use and Eggplant’s local support were also instrumental factors.”
Automation tools that would allow testing on real devices inside the FT’s lab were preferred to hosted solutions that offer access to devices in the cloud. The reasons were very clear — the team found that hosted solutions often prevented access to the full features of the device and that devices were often slow and unreliable due to overuse.
Antony Edwards, CTO, Eggplant, explains the rationale behind the implementation plan: “Test automation projects have a tendency to be slow in their execution. Day-to-day manual testing efforts and other priorities frequently get in the way and make people lose confidence in test automation. With a dedicated human resource, a strict plan, and Eggplant’s technical support, the FT team was able to automate all of its core test cases and achieve an ROI in just three months.”
The hit-and-miss nature of test automation projects encouraged the FT to draw up a tight implementation plan. The primary objective of this was to achieve 90 percent test case automation for the FT’s mobile app within three months. Eggplant’s gentle learning curve meant that one full time engineer was enough resource to achieve the target.
The FT found the creation and maintenance of automation scripts with Eggplant to be very straightforward.
“We thought the biggest challenge would be the maintenance of scripts when the look or layout of our app would get changed slightly,” adds Jose Corral, Test Engineer. “But as soon as we started working with Eggplant, we realised that the tool mitigates this effort and our scripts didn’t change. From then I knew it would be really easy, and it was.”
During implementation, the QA team received support and guidance from Eggplant’s local technical consultants. In three months they were able to successfully create all of their Eggplant scripts for a multitude of iOS and Android devices, as well as establish integration with the Jenkins continuous integration tool, using the Eggplant plug-in for Jenkins.
Jay Sethi adds:
“The main challenge of bringing in new technologies and ways of working is ensuring there is a smooth transition from the old methods. The knowledge and responsiveness of Eggplant ’s technical team meant that we were never sitting idle and could make the transition to automation swiftly and efficiently.”
Results and benefits
The standout results of deploying Eggplant were the improved speed and confidence with which the FT can now release all future updates to their mobile web app. Core test cases that would take six to eight hours to complete manually on the iPad were reduced to minutes. The Eggplant script from the iPad is then reused to simultaneously perform the same test on the iPad 2 and iPad 3 devices, saving days of work and accelerating the flow of feedback to the QA team.
The robustness of Eggplant scripts also allows them to run for long periods of time. The team uses these to monitor the mobile web app for 24 hours or more, setting Eggplant to pick up any changes in the performance or the GUI. There is an alert if something in the app changes and the team can evaluate the causes and provide quick feedback to development, enabling a quick fix.
Equally important is the advantage of increased test coverage. With Eggplant, the FT could test all flavors of iOS and Android, ensuring that the behavior of the app is the same across all. Chris Bower also has plans for the future:
“Now that we have the critical test cases automated with Eggplant, we can start creating tests for more unusual scenarios on a wider range of devices; that would simply not be possible if we were still testing manually. In addition to the time savings and improved coverage, using Eggplant has helped instill a strict routine to our testing, which can often get lost with manual testing. We schedule our scripts to run for hours, constantly taking snapshots of the application so that we have a real-time view of performance, just as a user does.”