Agile Development in a Medical Device Company

Pieter Adriaan Rottier, Victor Rodrigues

Cochlear Limited
[email protected]

Abstract To support this “mapping” process Cochlear™

This article discuss the experience of the software provides the audiologist with a software application
development group working in Cochlear™ with that interacts with the sound processor, audiologist and
introducing Scrum as an Agile methodology. We the patient to obtain the best mapping of sound unto
introduce the unique challenges we faced due to the the patient’s nervous system. Cochlear™ also
nature of our product and the medical device industry. provides the audiologist and patient with several other
These are long validation cycles, strict regulatory applications including an application to diagnose the
standards and a high level of dependence on physical condition of the implanted part of the Nucleus®
devices among others. We then show how we managed system.
to overcome these by adapting Scrum and mapping it The Audiological Software group of the Research
unto our current process in such a way as to continue and Development (R&D) division is responsible for
to satisfy the strict standard and regulations. We show the development and maintenance of these software
how we automated the testing of physical devices to applications.
overcome our dependence on manual testing. We
present the tools we used for managing requirement, 2. History/Environment
validation, reporting and performance metrics. Finally
we briefly look at the implications of the proposed All medical devices have to adhere to certain
IEC62304 standard on software development with standards to ensure patient safety and wellbeing.
Scrum. Cochlear™ implants are no exception. As our implant
systems deliver electric current directly to the nervous
system of a human being they have to adhere to the
1. Introduction highest level of the medical device standards. The
software directly controls levels of stimulation and
Cochlear™ is a medical device company that receives the same level of regulatory scrutiny as the
delivers innovative hearing solutions for severely to rest of the system. Some of these standards are shown
profoundly deaf people. Cochlear’s current product in Table 1.
range is the Nucleus® Freedom™ cochlear implant
system. This system consists of three components: Table 1: Regulatory standards applicable to
• a thin, flexible electrode array that fits inside Cochlear software
the cochlea EN ISO 14971:2007 Medical devices – Application of risk 2007 Required
management to medical devices (ISO
• an implanted receiver/stimulator that transmits 14971:2007) part 1 and 2
electric stimuli to the nerves of the cochlea via EN60601-1-4:1996 Medical electrical equipment – Part 1-4: 1996 Required
(IEC60601-1-4:1996), General requirements for safety – A1: 1999
the electrode array Amendment IEC Collateral standard: Programmable
• an externally worn sound processor that 60601-1-
4:1996/A1:1999
electrical medical systems

receives sound from the environment, encodes IEC 60950-1 Information technology equipment - 2005 Guidance
it and transmits it via an RF link to the Safety - Part 1: General requirements
ANSI/AAMI/IEC 62304: Medical device software - Software life 2006 Guidance
receiver/stimulator 2006 cycle :processes
The cochlear implant system interfaces directly with ANSI/AAMI Medical Device Software – Software Life 2001 Guidance
SW68:2001 Cycle Processes
the human nervous system. Therefore the electrical
impulses have to be customised to the nervous system
To ensure compliance to these standards and
of the individual. This “customisation” is referred to
therefore that our implants are safe and effective
as “mapping” and is done by audiologists that
Cochlear’s development procedures and quality system
specialises in cochlear implants. This is where the
are audited on a regular basis. Cochlear has developed
software enters the picture.
its own product development process, shown in Figure
1. This process is based on the V-model. The V- Scrum (the process developed by Ken Schwaber and
model is often used in regulated environments and is a Jeff Sutherland) was going to be the best fit for our
type of waterfall process. enviroment. It just made sense!
The process shown in Figure 1 is the one that was At that point the software development group was
used for software development, which is why the phase working on the third version of the “mapping”
dealing with the manufacturing process development is software and just starting a new project to improve
crossed out. Cochlear’s diagnostic software capability.
Scrum was introduced to both projects at the same
Project Definition/ Launch Product Definition/MAT System Development / Design/Verification Process Development/
Procedure Procedure Integration Procedure Procedure Verification
I40162AE E11429AE N94194AE I40165AE N94196AE

Project Definition Product Definition System Development Design Process Development

time but its success with the first project was slower
Start
Generate PMS,
PMP, Business
Needs Analysis
and Concept System Development
Preliminary and
Detailed Designs
Develop
due to the amount of legacy code in the project. The
second project was able to implement the process
Case Development Processes

T1 T1a T2

Pilot

much more easily and our experience in this project

Processes

T5 T4A T4 T3
Sustaining Unit Test & Process
Engineering

will form the basis of this paper.

Launch Market Design Verification, Build
System Integration,
Acceptance Verifications Sub-systems
V&V
Test

Systems Integration and

Launch MAT Design Verification Process Verification
Verification

Figure 1:Cochlear’s Design Control Process for 4. Challenges to the introduction of agile
software
Organisational challenges
A detailed description of this process is beyond the
scope of this paper. Suffice to say that it consists of Perhaps an important aspect of why Scrum was
formal definition, design, implementation, verification successfully adopted at Cochlear, at least within the
and validation phases with a sign-off between each of software development group, was that it was driven
the phases called a Tollgate. from senior management. Admittedly a lot of lobbying
went on and it was not immediately accepted as it was
3. The Introduction of Scrum largely seen as a process with short term goals and did
not cover the longer term business objectives and
In 2001 the software development group began timing needs. It was really an educational process
adopting some Agile development practices, supplemented by high visibility on the execution.
implementing a small number of changes at a time in Senior management and the software teams
an attempt to increase the chances of success within a encouraged people from around the organisation
waterfall organisation. Practices such as automated encouraged to attend Daily Scrum and Review
nightly builds, peer reviews, unit testing and others meetings. The business started seeing benefits in a very
were instrumental in the improvement of product short time and soon Scrum became a word that was
quality. The group also introduced the refinements ubiquitous within the R&D division.
shown in Figure 2 to the DCP process in an effort to
make the design and verification phases less Process challenges
monolithic and more iterative.
T1A T3 The medical devices industry is regulated by a
Feature 1 Specify Design Develop Test
number of regulatory bodies of which the FDA is
Feature 2 perhaps the best known example. For these bodies it is
Feature 3 extremely important to be able to see that the system
Feature 4
you are developing is safe as well as effective for its
Feature 5
Feature 6
intended purpose. As a company Cochlear™ has a
very good reputation based on its quality system and
Feature N its track record. Cochlear’s quality system is based on
Figure 2: Software design, implementation and Good Automated Manufacturing Practices (GAMP)
verification guidelines, the same as commonly used by
pharmaceutical companies. These guidelines require a
large amount of documentation and extensive
The benefits of these changes were obvious and
component, system, product and process verification
were welcomed by the organisation. Software
and validation.
products were far more robust than before but there
Other issues that faced introduction of Agile
was still something missing. “Feature creep” was still
practices at Cochlear were:
quite prominent and schedule slippages were
commonplace. After some further research into Agile • the disparity between using an Agile
development practices, the group decided in 2005 that methodology in the software department versus
 using a waterfall/iterative process throughout
the rest of the organisation Product Definition/Product Validation
• the identification of the recipient safety aspects
of a product and the mitigation of such risks in In this phase the first conflict between the classical
a structured and traceable way Scrum approach and our highly regulated environment
• managing the evolution of requirements in a became evident.
traceable way Traditionally we provided the validation team with
• the long validation cycles required to validate a Use Case document. The validation team used this
the marketing claims and ultimately the user document as the basis of their validation protocol. The
requirements Use Case document contained all the functional
• the extensive manual verification testing requirements. This document was supplemented by
required to verify that the electrical stimulation the Software Requirement Specification (SRS) which
of the implant under software application detailed all the non-functional requirements. After
control is within safe limits validation the reports generated by the validation team
• device drivers with a large amount of legacy along with the documents described above were
code that was neither unit tested nor conducive provided to the regulatory bodies as evidence of the
to any other type of automated testing. product meeting its safety and efficacy requirements.
We could not replace this process with disposable
5. Using Scrum for the CS19 project requirements, as recommended by Scrum. The
alternative had to show the same traceability as the
In this section I will present how Scrum was original approach while allowing the evolution of
introduced for the CS19 project. I assume that the individual requirements.
reader is familiar with the Scrum process. For a good We decided to adapt the User Stories concept used
introduction to Scrum we recommend “Agile by Extreme Programming (XP) as described by Mike
Development with Scrum” by Ken Schwaber [4] Cohn [1]. These User Stories needed to be stored
I will show how we mapped Scrum to the DCP and electronically in a fashion that allowed easy access to
I will point out the challenges still remaining. In the individual stories but also provided the ability to
next section I will introduce the tools that we used and generate reports showing each User Story or Constraint
explain how we adapted these to fit our purposes. and how they related to a validation activity.
As a start we only identified enough User Stories
for the first two development iterations.
Project definition/closeout We realised that we would not only have to evolve
our product requirements but also our process as we
In the first phase of the project we followed the
went along.
procedure mandated by the DCP. We developed the
business case and a project management plan where we
described the development process we would be Design, Implementation and Verification
following. The quality system allows for deviations
from the development process described in the DCP During the first few iterations we aimed to deliver a
provided the deviations is defined and justified in the slice through the entire system that could be used to
project management plan. Additionally it had to be start investigating the feasibility of the concept on
approved by the Quality Division. To obtain approval which the product was to be based. Having done just
we had to show that we would still adhere to the levels enough architectural design to identify the major
of traceability, verification, risk mitigation and system components, development started.
validation required by our quality system and the Almost immediately another big challenge became
regulatory standards. apparent; since the system needed to interface with
Scrum was justified as an appropriate process for external hardware, most of the safety testing had to be
the project based on the high level of uncertainty done manually. Because we only consider a User
associated with the capabilities of the measurement Story completed once it was implemented and tested, it
technology that we were using in this product. proved to be impossible to complete a User Story
The main issue in this phase was setting the within a Sprint (four weeks at this point). Initially we
timeline as all our estimations and experience were tried to circumvent this issue by allowing the testing of
based on a different development process. The a User Story to happen during the subsequent Sprint
timeline ended up being very far off the actual project while the developers worked on something else. As
duration; one thing that we are still struggling with. most of those experienced in Agile will know, this
proved to be a bad idea. Stories that were supposed to document and mitigate the risks by adding a section to
be done kept creeping back by way of bugs into the the tests to exposes the risk, in the form of a failing test
backlog and this quickly started to seriously degrade and then ensures that the risk is contained by getting
our velocity. the test to pass. Where this could not be done the risk
A difficult decision had to be made at this point; do was again added as a constraint to the product backlog.
we spend a significant amount of time automating the
testing or do we keep on going the way we are? We 6. The tools we used
decided to automate a part of the manual testing, the
data analysis, but not the rest. A project which started Requirements Management
a few months after the CS19 project subsequently
managed to successfully automate the entire manual We started off using CaliberRM® from Borland
testing process, a significant but ultimately worthwhile (http://www.borland.com/us/products/caliber/rm.html).
effort. This proved to be too hard for the developers to
We also decided to introduce the developers to Test navigate as it was very different from the tools they
Driven Development (TDD) using the Framework for were used to, viz. Attlasian Jira and Confluence
Integrated Tests (FIT) developed by Ward (http://www.atlassian.com/). Thus we transferred all
Cunningham [2] for the acceptance tests and NUnit for the requirements over to Confluence, making use of
the component/object verification. At some point some advanced macros to allow the generation of
during the process the hardware testing load started reports. Each User Story had its own Confluence page,
dropping and we shortened the iterations to two weeks. and consisted of three sections:
Finally Scrum really started showing its strengths. • the summary, containing the estimate, priority,
For tracking issues we used the standard procedure status and description of the User Story
of adding these to the product backlog along with the • the discussion, containing the minutes of any
new features. We used a fairly formal change control discussions around that User Story as well as
process for prioritising issues. The advantage to the any mock-ups, models and designs associated
formal process was that it ensured that we spend the with the User Story
time and effort required to understand every issue and
• the tests, containing both the manual tests and
its implications, as well as allowing us to spot trends
the automated tests associated with the User
that might point to underlying problems early.
Story.
The clinical validation still had to be done manually
Using advanced macros the relevant sections can be
but through iteration and extensive bench testing we
extracted for reporting, regulatory and dissemination
managed to ensure that the validation only provided
purposes.
confirmation that all the requirements had been met. It
Developers use the Jira issue tracking system to
was possible to decouple the clinical validation to a
manage and track the tasks associated with every User
large extent from the development by increasing the
Story, these tasks are linked on the one side to the User
amount of bench validation.
Story and on the other side to the source code through
our version management system, Perforce
Risk Management (http://www.perforce.com/).
Risk management is important in any industry and
Acceptance Testing
especially so in medical devices. Initially we tried to
manage risks according to the DCP. The DCP requires
We started off using the FIT framework for
a hazards analysis to be created during the product
acceptance testing but it proved to be difficult to
definition stage and then updated at major milestones.
integrate with our other systems and we were hesitant
This approach was not very successful as we were
to start using another completely separate tool. After
trying to determine risks on features that were not yet
some searching we found the Greenpepper framework
well defined. As an alternative we split the process
by Pyxis Technologies
into two stages. We identified high level system risks
(http://www.greenpeppersoftware.com/en/), which
during the initial phase of the project and added these
while using fixtures very similar to FIT, allows
as constraints to our product backlog. Any other risks
seamless integration with Confluence.
that became apparent during development we also
Our process evolved to the point that developers
added to the backlog. Then we identified, for each
together with the verification engineer would write the
User Story, the relevant risks at the point where we
acceptance tests in Confluence. Developers would
moved the User Story from the product backlog to the
then write the unit tests to express their design intent
Sprint backlog. For these User Stories we tended to
after which the code was written to first pass the unit first time, as we only allow enough story points to do it
tests and then to pass the acceptance tests. In this way once.
we managed to get rid of manual testing in almost all I like the suggestion by Alistair Cockburn that we
areas of code. allow the user to see the software twice and request
changes before considering a User Story as done. For
Project monitoring and control more information see [3].

Initially we used just a burndown chart, updated by Benefits of Agile

hand for project monitoring. This allowed some
visibility of the project status but often it would not get So what do we consider the biggest benefits of
updated, either because the developers have not using Agile in our environment? By using test driven
updated their effort from the previous day or because development along with the iterative development of
someone forgot to update the chart itself. features we have been able to fairly consistently
The entire monitoring and control area is one we produce high quality code. While we did not find the
still struggle with (for more see the section on development process to be any more efficient than the
challenges). We have however found Greenhopper, process we were used to we think this is due to the
also by Pyxis Technologies large amount of supporting frameworks we had to
(http://www.greenpeppersoftware.com/en/), very construct to allow it to operate in our environment. As
useful for automating our burndown charts. we move forward we fully expect to be reusing a lot of
what we have developed and therefore increased
7. Lessons learned efficiencies.
As we are almost always faced with a high degree
System Architecture Design of uncertainty in our projects we appreciate the
increased flexibility afforded by using Scrum.
Although we did some up front design of the system
architecture we did not do nearly enough in terms of
consciously evolving and revisiting the architecture at
regular intervals. This led to code being added without 8. Remaining challenges
sufficient thought being given to how it would
influence the big picture. The result was seemingly Better estimation
small code changes being nearly impossible due to
architectural issues which would then required major As with a lot of others that we have spoken to over
refactoring and redesign to resolve and often lead to a the years, estimation is one of the biggest challenges
Sprint goal not being met. we face. We seem to be continuously underestimating
the amount of effort involved in delivering a feature.
Agile without TDD
Further automation
Considering our environment it was an enormous
mistake to think that we could be Agile without being Although we have managed to automate a lot of the
test driven. We often spent as much, or even more, verification and reporting the team still require about
time on the tests as we did on the actual code, two weeks to of release activities before any release to
developing frameworks and helper classes, figuring out the field. A lot of the tasks during this time, like the
how to manage the hardware interaction and refining manual data capture, installation verification and visual
the requirements. Our experience has shown that verification of graphs could still be automated along
without those tests being in place before we start on the with all the reports required as evidence of these
code we have no way of reaching a finished state on verification and validation activities.
any User Story.
Project monitoring and control
Forgetting about the iteration time
We can track our progress, we can see when we
One of the common mistakes people make, and one start to fall behind schedule, but when do we actually
we fall prey to many times as well, is forgetting to cancel the Sprint? How do we prevent the same thing
allow time for the iteration of User Stories. We from happening again? At which point to we start
subconsciously still require our users to get it right the cutting tasks from the Sprint? These are some of the
questions that we still haven’t been able to answer to teams. At a system level Scrum has had a significant
our satisfaction, we have the metrics but we are not impact on the way we develop products at Cochlear.
convinced that we are measuring the right thing. And
if we are measuring the right things we are not sure Bibliography
that we understand what to do with those
measurements. [1] M. Cohn, Agile Estimating and Planning, Prentice Hall
PTR, New Jersey, 2005.
9. The way forward [2] W. Cunningham, “Framework for Integrated Test,”
Framework for Integrated Test, October 12, 2007;
As we have mentioned before the industry we are in is http://fit.c2.com/.
highly regulated and software used in devices is [3] A. Cockburn, “Three cards for user rights,” Alistair
Cockburn, 2007;
certainly no exception. In fact regulators are becoming http://alistair.cockburn.us/index.php/Three_cards_for_user_ri
far more vigilant with regards to the scrutiny of ghts.
software in a medical device. Software development [4] K. Schwaber, M. Beedle, Agile Software Development
standards have existed for some time but recently a with SCRUM, Prentice Hall, New Jersey, 2001.
new comprehensive standard emerged featuring a
comprehensive guideline as to how to go about
developing software that form part or is itself a medical
device. The standard, IEC 62304, acknowledges that
software can be developed through three types of
development methodologies, viz. Waterfall,
Incremental and Evolutionary. It does not prescribe
any of these methods but does insist that all process
outputs be maintained in a consistent state. This
requirement is quite easy to achieve in a Test Driven
environment. Furthermore many attributes of the
standard can be mapped quite elegantly to several
Agile processes. The standard has a significant focus
on safety and prescribes a continuous assessment of
risk to the patient at a system level. This is where the
challenge lies as this requires the involvement of multi-
disciplinary teams which may include non-Agile
groups. Although we have just started this process we
are confident that the Agile practices we have
employed are compatible with the standard.

After 2 years of using Scrum within the software

group at Cochlear we can honestly say that we are still
learning. One can argue that it’s what being Agile is all
about! There is no doubt that the software we are
producing is of a much higher quality and certainly
contains features that have delighted our customer far
more than ever before. But customers have short-term
memories and the bar needs to be raised constantly.
Being Agile allows us to do this. The software group’s
total staff complement has grown by about 15% over
the last 5 years and we are developing 4 times as many
products as before.
The more compelling story, however, is that other
non-software groups have started using the Scrum
process and this has had a phenomenal impact on both
the level of interaction from the teams that comprise
those groups as well as the level of visibility of the
various products that are being developed by those