Leading Defense Systems Integrator

Eggplant - Case Study - Defence Contractor

This defense prime has been a trusted and reliable partner to the global defense sector. They are a cutting-edge defense systems integrator, providing technology and in-service support to defense organizations and public security markets in North America and abroad. It coordinates multiyear projects of strategic national significance, scope and scale, delivering cost-effective, mission-critical systems.

They are a customer-centric organization, which recognizes that reliability in adverse environments is a key requirement. That is why it is critical to ensure that all of its technical solutions represent the best combination of leading edge technology and robustness. Their technical leadership determines the right balance of strategy, innovation, collaboration and reuse for all areas of operation.

Challenges

Testing plays an important role in the software development lifecycle of any business. Intuitive, bug-ridden software can cause loss of sales, market share and reputation. But when your customers are in the defense sector, untested software could mean a loss of confidence in vital systems, and ultimately perhaps even loss of life.

At this defense prime, the structure of testing teams is wholly dependent on the scale, scope, and duration of each project. Teams range from as small as two-to-three testers to as large as several hundred individuals. Smaller projects might be research and development programs that still need to undergo thorough testing. Larger programs demanding extensive teams might include the development of an entire maritime helicopter program comprising very complex systems that require continual test and refinement for the rest of their lifecycle.

A systems engineer at this defense prime explained, “Regardless of scale, testing requirements are always very well-defined by our customer. They come to us with a set of system requirements and we have to meet those. But the challenges of accuracy and delivery are great. Take air and naval for example — we have to adhere to a specific software development standard called D0178B that defines high-level processes and the levels of rigor needed in controlling the development and testing of the software as well as the associated documentation and traceability of test back throughout the course of software development.”

They also stipulate their own strict internal requirements. Because of the nature of their customers, it is held to a different level and standard of reliability in the software it produces, and so it is required to go through several different iterations of tests under very strenuous conditions. At every stage they conduct a lot more formalized and rigorous testing then most other industries. For them, a project deadline tends to be in the order of years as opposed to months or weeks.

Solutions

A system engineer at this defense prime explained that his first step was to research possible solutions online. “We wanted to start developing automated testing, so we started looking at a number of different black box tools on the internet and found that Eggplant was by far the most mature out of several dozen competing products.”

Prior to investigating automated testing solutions, this defense prime relied on a manual test team that developed test cases and test processes themselves and actually tested them with customer witnesses. After taking a closer look at a handful of alternative automated solutions, with failings ranging from a lack of remote controls to inappropriate levels of operator interaction, they decided to opt for its first choice, Eggplant.

“One of the most important considerations was the ability to be able to test a system non-intrusively, without actually affecting that system’s configuration,” explains the defense test engineer. “Eggplant really allowed us to leverage that and sell that functionality to our customer. One of their biggest concerns is that when we’re testing, we do not affect the actual operational environment of their software. A lot of automated test tools out there need to be embedded with the developed deployed software, whereas Eggplant is extremely smart in removing itself from the test environment.”

Other major benefits of Eggplant cited by this defense prime included maturity, stability, ease-of-use, flexibility of configuration, adaptability to a variety of testing environments, and finally, the ability to look at image comparison for Operator Machine Interface (OMI) testing.

Results

So what benefits have this defense prime experienced since working with Eggplant? Although it is still very much early days in terms of tangible implementation of automated process, they now have the infrastructure in place and have shown a number of people at very high levels within the company that test automation is the way to go.

The defense system engineer quotes, “Test automation needs to be intrinsic in terms of cost savings and ability to deploy. One of the reasons we hadn’t implemented test automation up until this point was that there was naturally a lot of resistance towards a method that had never been demonstrated as a cost-effective way of testing systems that our customers would buy into. In the last year, our customers have begun to understand the value of automated testing, so we are now able to execute tests and define the processes around automation specifically using Eggplant.”

Eggplant is causing a shift in thinking, as well as a shift at this top defense contractor, with ability to execute projects and execute software development into the future. It’s a gradual change for an organization of this size that employs 80,000 people worldwide, but Eggplant adoption is set to continue to grow over the coming years. So far, the defense prime has firmly established the need for automated testing and shown it can be performed at a level that their customers find agreeable, can get involved with and can be very happy with when a higher quality product is the result.

Future roadmap

The test team's next initiative is for the generalized automation of software integration as well as system testing. They are primarily a systems integrator that conducts a degree of software development, and so it’s now the systems integration side that is a focus area for the defense engineers. “Systems integration is the bread and butter of what we do, so automating that is our end goal. Eggplant is absolutely fundamental and at the core of being able to do that testing because of its ability to verify software and systems functionality at a black box level. In terms of the grander goals and a grander scheme of things, we’re also looking to predicate a change of culture within the company and a change of culture within the defense industry by showing that we can develop these automated tools to a very high level of rigor and robustness in order to process a much more cost-effective product to our customer.”

With shrinking defense budgets, contractors need be able to react by adopting more-efficient but equally effective processes if they are to remain competitive within the industry. This defense prime have decided to bid more cost-effectively for defense program projects so that its customer base can continue to grow while always maintaining a cost-effectiveness to tax payers.

So how does this defense prime measure the success at this early stage of adoption? The engineer suggests, “Our metrics are always in terms of hours saved. As our testing is wholly manual, any hours that our test engineers don’t need to be in the lab executing tests, benefit the company. This is especially true when it comes to automating lower-level testing. To get those guys out of the lab to focus their time on developing higher-level, scenario-based testing and more customer-focused testing, instead of this lower-level software functionality robustness testing, is invaluable to the company. Our metrics are based per test cycle. Each of those tests equates to a certain number of hours so by automating say 1,000 tests we might save 2,000 hours for example. But whether it’s yearly, annually, or another period would depend entirely on the project, the tests and the customers.”

Conclusion

The success of the testing team is measured through metrics that are calculated through the recorded hours of test engineers. Any lab hours saved by automation are of great financial benefit to this defense prime, and so is the ability to enable those valuable engineers to focus their time on developing higher-level, more scenario-based, customer-focused testing instead of lower-level software functionality robustness testing.

So does the system engineer feel that the defense prime ultimately made the right decision to talk to Eggplant and to deploy Eggplant? “Absolutely,” he says. “We have a great relationship with Eggplant, the level of service has been excellent. Anytime we have questions we get instant answers, and when we have enquiries about new functionality or new capabilities that we’d like to see, we can have very open conversations with Eggplant's technical and sales teams about when these things might be implemented, how we will be using them, and what capabilities are best both for us. For us, Eggplant is the right tool, provided by a company with the right attitude.”