Met Office, UK National Meteorological Service

Met Office automates testing to increase confidence in mission-critical weather warning system.

ian-froome-362138.jpg

Profile

The Met Office is the United Kingdom’s national meteorological service and is responsible for making predictions across all timescales from weather forecasts to climate change. The Met Office provides a National Severe Weather Warning Service (NSWWS), which warns the public and emergency responders of severe or hazardous weather that has the potential to cause danger to life or widespread disruption.

Key Metrics

  • Reduced risk: 4x increase in regression frequency to deliver improved confidence in mission-critical system.
  • Saved time and resources: Reduced a half day of manual testing to 10 minutes.

NSWWS is a complex service that interacts with a lot of Met Office's other systems and products. The organization’s IT department wanted to conduct more frequent regression testing to reduce the risk of potential problems occurring as a result of new releases or changes to other systems that connect into it. Ian Gentry, test lead at Met Office, explains.

“NSWWS is a service which provides warnings to the public and emergency responders of severe or hazardous weather that has the potential to cause danger to life or widespread disruption. Introducing issues to NSWWS in software updates could therefore be disastrous because it could ultimately impact lives. There is also the potential reputational damage that would be associated with any system errors, so we are constantly seeking to improve confidence in the system and lower corporate risk wherever possible. Regular testing is a huge part of this.”

Testing NSWWS

NSWWS has a complex implementation with many different outputs and processes, beginning with a warning being issued by meteorologists in the Met Office’s Operations Centre, that includes details about the type of warning (snow, wind, ice, etc.) and the areas affected. This is processed through a range of internal systems and data networks before being displayed in a range of different end points, including the Met Office website, mobile website, and mobile apps. Confirming that all information issued by the forecaster displays as it should on all channels is vital to the successful operation of NSWWS.

Previously, every four to six weeks, when a change or update was made to NSWWS or a significant change to a connecting system, the testing team would take around a half day to manually run through a set of regression tests to ensure that the system would still display the alerts correctly following the forthcoming change. “We were able to use Eggplant documentation to solve any challenges and did not need to raise many support requests," says Gentry. "Our Eggplant account team has been heavily invested in our success and provided ongoing strategic guidance throughout the deployment.”

Automation to transform confidence

The Met Office decided to adopt a test automation solution that would allow it to automate the existing regression tests it was conducting manually, and ultimately to increase the overall amount of regression testing to reduce potential risk and raise confidence.

The IT department selected Eggplant Functional, Eggplant’s functional test automation solution, which uses a patented, image-based approach to UI testing that allows it to interact with any device by looking at the screen, in the same way a user does. Essentially, the organization is using Eggplant to verify that the visualization of warnings on its public website appear exactly as they should.

The Met Office’s IT team wanted to be able to generate warning scenarios in its internal Warnings Manager tool and then check that these visualised correctly on multiple end points. They created a tool to recreate the output of Warnings Manager, that injected the XML output directly into the system that processes the warnings. Eggplant Functional was used to test the output on the website and to verify that the test scenario was visualised correctly, exactly as the public or emergency services would see it.

“We wanted a tool that was simple to maintain, could support our goal of testing from generation to visualization, but also that used a non-invasive approach. We found that Eggplant, with its user-centric, image-recognition-based approach to test automation was a great fit,” adds Gentry.

“With a range of warning types as well as multiple regions to cover, there are many small but important details, for example a warning for the Northeast may partly extend into the Northwest, which means that the warning should display in both regions. It is important that our testing picks up on any inaccuracies, however small, that may result from changes or updates to the system. We had to allocate resources to do those regression tests, often taking people off other tasks to do so. But we also wanted to be able to run tests more often — even for those releases and changes to other systems that we didn’t believe would affect NSWWS in any way,” he says.

Results

Since implementing Eggplant, the Met Office has been able to to slash the time previously spent on regression testing from a half day to a mere 10 minutes, allowing the IT team to be significantly more productive. As a result, the test team has been able to dramatically increase the overall volume and frequency of regression testing from once every four to six weeks, to once or twice a week. Every time there is a change to any system that touches NSWWS, the team can quickly run tests.

"Automation tooling is well suited to our strategy of moving from waterfall to agile. It allows us to achieve our aim of risk mitigation in a cost-effective manner, whilst having an audit trail that is vital to any newly implemented process," emphasizes Aidan Green, head of solutions delivery at Met Office.

“Using Eggplant, we can now provide assurance in a matter of minutes that there will be no impact when making changes to NSWWS," concludes Gentry. "It has transformed our approach to testing. Through automation, our testing is now quicker and more frequent, allowing us to confidently release changes secure in the knowledge that our critical services are protected."

With Eggplant, we can now provide assurance in a matter of minutes that there will be no impact when making changes to NSWWS. It has transformed our approach to testing.
— Ian Gentry, test lead, Met Office
Case Studyjeff schlauch