The pain of software version release
As everyone knows, the release process for medical device software is stressful and time-consuming. Medical software verification and validation (V&V) requires verification that
Why are faster release timelines required?
Fast release timelines mean:
● Cost reduction and better resource utilization
● Competitive business advantage
● Handling security threats on-time
● Satisfied customers
Back in 2015, we thought about hiring a staff dedicated to manual testing. However, we ended up applying a completely different approach. Instead, we decided to spend our money to hire top-talent software engineers and implement
We decided to invest in:
● Massive automatic testing
● automatic report generation
However, not every test can be fully automated with reasonable effort. There are GUI and HW tests that are challenging to automate. Fortunately, with our new approach, the time required to run manual test
The test-driven approach is deeply integrated
The system is designed with a very thin UI layer in mind, for easier testing and mocking. The back-end consists of a number of small and independent services (micro service architecture), each one having relatively small product responsibility. The lack of coupling between services allows for easy testing.
We use a continuous integration (CI) system that consists of a building process and executing of automatic tests.
We have the following types of tests in place to ensure the high quality of our product:
● Unit tests - these are tests
● Integration tests - these tests integrate several objects together.
● System tests - these tests are executed in the context of the entire system; essentially, they represent the biggest group of integration tests.
● GUI tests - these tests are executed to verify the UI’s behavior, such as responding to user input, state changes of the system, or controlling navigation between screens, etc.
● Accuracy tests - these tests are actually benchmarks that execute algorithms on the big data sets and assert some performance metrics.
● Requirement verification tests - can be any of the above, but are explicitly marked as verification tests with a specific requirement, allowing automatic traceability generation.
● Static analysis - verifies the correctness of the dynamically typed languages and ensures that the code adheres to industry standards.
● Exploratory testing - “free style” manual testing that is usually done around critical system features and/or around newly-added or updated features.
The tests are categorized as fast (under 1-2 seconds), slow (all other tests), and run-by-demand tests. All fast tests are executed on each code commit, whereas all slow tests are executed nightly.
How is the product released?
The following are the steps of product release:
- All relevant automatic tests are executed, creating a file in JSON format with results of the execution
- All relevant manual tests are executed and their results are recorded electronically
- Traceability matrix is generated
- Software Test Document (STD) and filled Software Test Report (STR) documents are generated automatically from the manual test steps and automatic tests.
- All updated documents are filed and signed in the Device Master Record (DMR).
- Software release artifacts are stored.
Today, we have thousands of fully automatic tests and only 13 manual tests. We constantly improve our testing coverage and decrease the effort required for manual testing to ensure the quality of our future product releases.
There are cases where 100% automated testing can be achieved, reducing the number of manual tests to zero or a negligible number.
Using the approach described above, our full software release flow is executed in less than one day. This allows us to address our business goals much faster, provides the flexibility needed for business decision-making, and significantly increases the quality of our delivered products.