Automatic testing for medical device software

The pain of software version release

As everyone knows, the release process for medical device software is stressful and time-consuming. Medical software verification and validation (V&V) requires verification that software meets predefined specifications and validation that it fulfills the intended purpose in the view of the customer. Practically speaking, this is achieved by performing functional and nonfunctional testing. Regulatory compliance and documentation requirements demand a huge manual effort from development teams, including multiple iterations of development and testing. All of this leads to long release cycles, overspending of resources for software developers, software testing, project management, and technical writing, as well as frequent delays in release timelines.

Why are faster release timelines required?

Fast release timelines mean:

●      Cost reduction and better resource utilization

●      Competitive business advantage

●      Handling security threats on-time

●      Satisfied customers

Our solution

Back in 2015, we thought about hiring a staff dedicated to manual testing. However, we ended up applying a completely different approach. Instead, we decided to spend our money to hire top-talent software engineers and implement development methodology that emphasizes testing driven development (TDD). Moreover, code that can be easily tested is evidence of good software engineering. Indeed, this was another motivation to promote the quality assurance state of mind.

We decided to invest in:

●      Massive automatic testing

●      automatic report generation

 

However, not every test can be fully automated with reasonable effort. There are GUI and HW tests that are challenging to automate. Fortunately, with our new approach, the time required to run manual test

The test-driven approach is deeply integrated in our company's DNA. Quality assurance plays as an integral role during the planning of each feature, even before we start thinking about how to implement that feature. Questions like, “How will this be tested?” and “How do we know it works?” are fundamental and are being asked of all our deliverables, from big features to small development tasks. During code reviews, we ensure that everything is tested properly before a feature is integrated into the main development branch in our source control.

The system is designed with a very thin UI layer in mind, for easier testing and mocking. The back-end consists of a number of small and independent services (micro service architecture), each one having relatively small product responsibility. The lack of coupling between services allows for easy testing.

We use a continuous integration (CI) system that consists of a building process and executing of automatic tests.

We have the following types of tests in place to ensure the high quality of our product:

●      Unit tests - these are tests on the level of a single object.

●      Integration tests - these tests integrate several objects together.

●      System tests - these tests are executed in the context of the entire system; essentially, they represent the biggest group of integration tests.

●      GUI tests - these tests are executed to verify the UI’s behavior, such as responding to user input, state changes of the system, or controlling navigation between screens, etc.

●      Accuracy tests - these tests are actually benchmarks that execute algorithms on the big data sets and assert some performance metrics.

●      Requirement verification tests - can be any of the above, but are explicitly marked as verification tests with a specific requirement, allowing automatic traceability generation.

●      Static analysis - verifies the correctness of the dynamically typed languages and ensures that the code adheres to industry standards.

●      Exploratory testing - “free style” manual testing that is usually done around critical system features and/or around newly-added or updated features.

The tests are categorized as fast (under 1-2 seconds), slow (all other tests), and run-by-demand tests. All fast tests are executed on each code commit, whereas all slow tests are executed nightly.

How is the product released?

The following are the steps of product release:

  1. All relevant automatic tests are executed, creating a file in JSON format with results of the execution
  2. All relevant manual tests are executed and their results are recorded electronically
  3. Traceability matrix is generated
  4. Software Test Document (STD) and filled Software Test Report (STR) documents are generated automatically from the manual test steps and automatic tests.
  5. All updated documents are filed and signed in the Device Master Record (DMR).
  6. Software release artifacts are stored.

Summary

Today, we have thousands of fully automatic tests and only 13 manual tests. We constantly improve our testing coverage and decrease the effort required for manual testing to ensure the quality of our future product releases.

There are cases where 100% automated testing can be achieved, reducing the number of manual tests to zero or a negligible number.

Using the approach described above, our full software release flow is executed in less than one day. This allows us to address our business goals much faster, provides the flexibility needed for business decision-making, and significantly increases the quality of our delivered products.