This article has been accepted for inclusion in a future issue of this journal.

Content is final as presented, with the exception of pagination.

IEEE SYSTEMS JOURNAL 1

AUTILE Framework: An AUTOSAR Driven

Agile Development Methodology to Reduce
Automotive Software Defects
Ahmed M. Khan and Timothy D. Blackburn
.

Abstract—This article introduces AUTOSAR Driven Agile De are adopting newer approaches to designing software [1]–[4],
velopment (AUTILE) Framework as a new methodology to de with a focus on standardization to achieve better scalability of
velop automotive software with an aim to reduce the number of the software for different variants of the electronic control
defects and their severity. Automotive software, developed in a units (ECUs) [9]–[20]. Several new frameworks have also
traditional/legacy way, currently must be rewritten from scratch
been intro duced to fundamentally change software
to support new features or modifications in associated hardware.
Disruptive automotive megatrends: connectivity, electrification, development method ology. These frameworks enable the
and autonomous driving continue to demand new features in a developers to integrate modifications, even in late design
complex software-base. As a result, greater quality issues are stages, without compromising the quality and safety features
faced when more functionality is added to traditional, of the product [21]–[33].
nonstandardized automotive software. Several researchers have In recent years, the integration of software into automotive
proposed standard ization of automotive software to improve the in dustry has seen exponential growth [5], [6], [8], [34]. The
software quality. However, as yet, the industry lacks a de facto
current generation of cars is equipped with more software
standard. At the heart of the AUTILE Framework, we
recommend following automotive open system architecture modules than some highly sophisticated machines such as,
standard as the basis to design modular and open software Boeing airplanes and US Air Force fighter jets [5]–[7].With
architecture. To further realize the benefits of modular software increased dependence on software-controlled components,
architecture designed in the first phase, AUTILE chooses and there is a greater risk of equipment malfunction. Many
integrates Agile methodologies as its software devel opment researchers have claimed a linear relationship between the
approach. Widespread adoption of Agile methodologies for number of lines of code and the average number of software
automotive software development has not occurred and this
defects [6], [34]. Having an automobile with millions of lines
research highlights the enablers and barriers in this domain. The
proposed framework was applied practically to develop automo of code implies a significant number of defects. Even with
tive software projects, demonstrating that defect reduction can be strict quality assurance measures, up to 15% of these defects
accomplished using the steps outlined in this new methodology. can go undetected in manufactured units [6], [34]. A steady
rise in automotive recalls has been observed over the years
Index Terms—Automotive open system architecture
[35], [36] and among these recalls, 15% have been due to
(AUTOSAR), agile systems engineering (ASE), automotive
software development, software quality, software defect software defects [35]. As software content increases in
reduction, defect severity index. automobiles, software portability, scalability, and
transferability become vitally important [9], [16], [19].
Utilization of a sta ble software base for more than one
I. INTRODUCTION automotive variant, with little or no modifications, reduces
research and development costs. However, traditional software
T HE automotive industry is rapidly evolving through development practices in the automotive industry experience
poor portability, i.e., the code has to be rewritten from scratch
global from any modifications to the associated hardware [9], [10],
[16], [19]. Scholars and professionals have argued that
trends of electrification, connectivity, and automation
standardization of automotive software can improve the
[1]– [4]. The integration of software in automobiles is
reusability and several attempts have been made in this regard
becoming a key differentiator in providing value-added, as
[9]–[11], [13], [14], [17], [20]. One successful example of
well as safety-critical, features [5]–[8]. A typical automotive
such an attempt is the establishment of the Automotive Open
development cycle ranges from four to eight years during
System Architecture (AUTOSAR), with an aim to provide
which several technologies considered state-of-the-art during
modular and open software architectures [9], [10], [16], [19],
the design phase may become obsolete at the time of
[37].
production for the consumer market. Therefore, in order to
stay competitive, automotive companies The automotive industry has also been characterized by
strin gent demands on quality and documentation for all
software requirements. This makes adoption of development
Manuscript received February 13, 2020; revised May 13, 2020; accepted
May 17, 2020. This work was supported by Siemens Digital Industries models like V or Waterfall the preferred choices [25], [30],
Software Division. (Corresponding author: Ahmed M. Khan.) [38]– [40]. The automotive industry is constantly disrupted by
Ahmed M. Khan is with the Mentor, A Siemens Business, Wilsonville, OR trends like automation, connectivity, and electrification, for
97070 USA (e-mail: [email protected]). which
Timothy D. Blackburn is with the Department of Engineering Management,
 2 IEEE SYSTEMS JOURNAL
the Waterfall Model a preferred choice. In the automotive in
dustry, the process begins by gathering associated
requirements are anticipated at the later stages of the de sign requirements from customers (OEMs, Tier-1s). Tier-2s focuses
phase [1]–[4], [41]. For this purpose, adoption of a lightweight on the devel opment process internally afterward and create a
process for software development, reducing delays due to stable software design and architecture. Software
requirement elicitation to validation and production without development, testing, and integration phases are executed
compromising quality, becomes a key requirement. An afterward. During these phases, customers are again involved,
Agile-based software development approach can provide these and the software is deployed only if it meets predefined
benefits [25], [30], [38]–[40], [42]–[46]. Although auto motive acceptance criteria. Waterfall-based software development
manufacturers are starting to follow the Agile frame work, the minimizes risk and uncertainty, especially if the requirements
adoption process is very slow [21], [22], [31], [32], [47], [48]. do not change after the initial definition phase and the relevant
Standardization of automotive software and adoption of skillset is available for the development [30], [38]. Its other
Agile methods may allow automotive manufacturers to advantages include efficient resource and financial planning,
effectively deal with emerging trends in the industry. “scope creep” prevention, accuracy of design, and extensive
Standardization and early adoption can translate the documentation [30], [38], [39].
challenges faced by many automo tive companies into Strict traceability between definition and test phases of the
competitive advantages by improving soft ware quality aspects project can also be achieved by deploying a V-model-based de
like scalability, transferability, reusability, portability, and velopment process [30], [49]. All the requirements are
maintainability over the product life-cycle [9], [10], [16], [17], gathered at the start of the process with an assumption that the
[19]. It may lead to better software integra tion, resulting in required skillset is readily available [25], [38], [39]. From the
improved efficiency [15], [16]. It will also provide a faster requirement set, software architecture is designed with
path to innovation with better designs [15], [16], [19]. individual software components. After the completion of the
Problem Statement: size and complexity of automotive soft design phase, imple mentation is carried out, followed by
ware are increasing and as a result, modern cars could face individual component testing and integration. With the upward
hundreds of thousands of software defects [6], [34]. bent V, there is an increased focus with 1:1 traceability
Research Statement: An AUTOSAR-driven Agile develop between the design and the test phases. Its other advantages
ment methodology will reduce automotive software defects. include efficient resource and financial planning, “scope creep”
Taking this era of automotive disruption into account, two prevention, accuracy of design, extensive documentation, and
traits have been considered in this research to minimize au detailed evaluation before delivery [30], [38], [39].
tomotive software defects and their severity: 1) a standard ized Traditional automotive software development methods use a
AUTOSAR-based architecture providing better scalability, highly structured and organized approach [39]. The choice of
maintainability, and transferability [9], [10], [15], [16], [19], software development methodology depends on the characteris
and 2) an Agile development approach which is currently not tics of the project, the team, and the organization [29].
being widely used in the automotive software industry [21], However, there is an underlying assumption which is not
[22], [31], [32], [47], [48]. necessarily true in this era of disruption, i.e., “the software
The scope of this research is limited to automotive software requirements will not change after initial elicitation” [25].
projects (ASPs) executed by Tier-2 software suppliers. This Deployment of software into complex, interdependent, and
research does not consider ASPs executed by other automotive connected systems will require modifications at advanced
companies or projects in other industries. stages. Thus, modern automotive-grade software development
needs to cater to con tinuously changing requirements. As a
II. LITERATURE REVIEW result, traditional devel opment approaches do not produce
optimal results [44], [45], face a greater number of defects
A. Automotive Industry and the Ongoing Disruption [25], and may result in conflicts between customers and
McKinsey & Company determined that the conjunction of suppliers at the time of delivery.
automotive trends is disrupting the auto industry [3]. The
report suggests that diffusion of advanced technology will lead C. Agile Methods and Their Suitability for
to more autonomous and electrified vehicles in the coming Automotive Industry
years. Another study identifies that over 1700 start-ups with a
focus on mega-trends of connectivity, autonomous operation, Agile Manifesto has led to the creation of various Agile
and electrification are entering the marketplace and thus techniques including Scrum: [50], [51], eXtreme Program
changing the rules of doing business [41]. This disruption will ming [50], Lean Development [52], Kanban [52], and Feature
affect all aspects of the supply-chain and to remain relevant in Driven Development [50], [52]. Although Agile has a strong
the future, all stakeholders will have to innovate [1]. emphasis on flexibility, the process is not always easy to fol
low [53]. To ease the adaption effort, researchers have pro
posed tailoring Agile methods to organizational requirements,
B. Traditional Automotive Software Development and
environment, and culture [26]–[28], [54], [55]. Automotive
Their Disadvantages
soft ware industry is characterized by stringent demands on
Automotive software development is characterized by quality with all the requirements defined up front, resulting in
upfront definition of requirements and strict quality assurance, minimal use of Agile development. A survey conducted over
making 15–20 leading automotive companies reports that the scope of
Agile
 Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

KHAN AND BLACKBURN: AUTILE FRAMEWORK: AN AUTILE METHODOLOGY TO REDUCE AUTOMOTIVE SOFTWARE DEFECTS 3

implementation for a given company amounts to only 5.6%

of the overall operations [23]. However, the disruptive trends
of connectivity, electrification, and automation illustrate new
use-cases and demand requirement modifications to later design
stages, making Agile a preferred choice for the software develop
ment [30], [40], [56]. Several researchers, in collaborations with
major car manufacturers, have undertook case studies to identify
challenges that companies might face, and the opportunities
they might explore when adopting an Agile approach. It was
found that most challenges resulted from management change
processes [47] and a stable requirement-set surrounded by rigid
processes [48]. With organized efforts, Agile adoption in the releases for its various specifications are in process. Several
automotive domain has improved significantly. The first “Agile researchers have attempted to gauge the tradeoffs on how
for Automotive” conference was held in May 2019 in Detroit much
[57]. In this conference, the theme of most of the discussions Fig. 1. AUTILE Framework—Initialization.
was to share the lessons learned when Agile was adopted for
selected pilot projects at participating organizations [21], [31].
Scrum is an Agile development framework, which is
of the evolution needs to be followed with a new standard
suitable in situations where the requirements are not fully
release after initial deployment [59].
defined at the start of a project [51], [54], [56]. It relies on
regular feedback and incremental modifications to cater to
changing requirements [40], [51], [52]. The Scrum cycle starts III. METHODOLOGY
with the collection of requirements and associated This section describes the overall methodology and formal
prioritization and planning to create a Sprint Backlog. A series izes the concepts in the form of a framework, which
of Sprints is planned over prioritized items to get shippable automotive software suppliers can use to improve the software
increments, and during the Sprint cycle, a daily Scrum is quality by reducing the number of defects and the DSI. This
conducted to ensure that the team stays on the agreed-upon research was conducted in six phases and several methods
track. At the end of a Sprint, a Retrospective is performed. were utilized to accomplish its goals:
This cycle continues till either the desired product is delivered, 1) identification of practices and trends in automotive soft
or the allocated time has passed or funds are depleted [51], ware development;
[52], [56]. Automotive software industry projects are large 2) hypotheses for AUTOSAR- and Agile-based projects for
scale, globally distributed, iteration-driven, and have stringent number of defects and their severity levels;
requirements on quality. Keeping in view the suitability of 3) hypotheses testing over traditional/legacy and
Scrum for these dynamics [40], [51], [56], [58], along with AUTOSAR- and Agile-based projects;
requirements and processes of the automotive software 4) Development of AUTILE framework;
industry, this research establishes Scrum as the standard 5) hypotheses for AUTILE-based projects for number of
framework to introduce agility into the development process. defects and their severity levels;
6) hypotheses testing over AUTILE projects;
7) conclusions and recommendations for future work.
D. Automotive Software Standardization and AUTOSAR
A. AUTILE Framework
There have been several attempts at standardization in the
automotive industry, which may inherently change the The AUTILE framework provides a holistic view of
working methodology in which the software is developed [11], automotive-grade software. It is intended to be used as a guide
[14], [20], [33]. One such standardization attempt, which that draws focus on early design and architecture at an all
gained significant acceptance in the auto industry and the inclusive level by utilizing AUTOSAR-based standardized and
associated global deploy ment and proliferation, is the creation modular software architecture (37). Furthermore,
of the AUTOSAR standard [9], [10], [12], [18]. AUTOSAR modifications are anticipated in software requirements at later
takes a modular approach where separate components can be design stages due to progressive industry [1]–[4]. This fact is
integrated into a complete working product, and adopts an taken into account at an early design phase by proposing an
open software policy with publicly available design and source Agile-based Scrum development approach, which makes
code [16], [19]. By doing so, software quality aspects, such as AUTILE quite flex ible to evolve. The framework is also kept
scalability, reusability, and transferability to different vehicle customizable to fit into varying organizational strategies and
and platform variants become the core development cultures. Moreover, Agile principles are tailored to suit
philosophy [9], [15], [18]. Furthermore, it enables better automotive software requirements, processes, and challenges.
maintainability metrics throughout the life-cycle of the product AUTILE framework is divided into three main phases:
[18], [19]. AUTOSAR is an evolving standard and new initialization (see Fig. 1), planning (see Fig. 2), and execution
(see Fig. 3). collected requirements at this point are at a very high level.
1) Initialization: The first phase of the AUTILE framework Two sources are
collects the available set of requirements up front. The

Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE SYSTEMS JOURNAL

is called an “ECU Extract.” An AUTOSAR-based ECU extract

in the AUTOSAR-specific ARXML format is utilized by Tier-2
Architects, PO and development team leads to gain visibility of
the overall design and realize associated software requirements.
For rigorous quality, traceability, and audit necessities, the
requirements gathered during this phase are documented via a
requirement management system by the PO.
2) Planning: AUTILE integrates Scrum as an Agile ap
proach. Its planning phase is divided into three activities: Product
Backlog, Product Backlog Refinement and Sprint Planning, and
Sprint Backlog.
Product Backlog: during this phase, high-level software re
quirements are translated into Epics or User Stories. The PO,
with the help of a Product Manager and System Architects,
Fig. 2. AUTILE Framework—Planning. formulates the “System Extract” of a car. From the system
extract, information about the signals and messages that
belong to a single ECU could be taken out by Tier-1s. The
extracted information, detailing the signals for a specific ECU
within a car,
translate high-level software requirements into newly required
functions, feature requests, requirements, enhancements or
bug fixes. The Product Backlog is the super-set of all the
require ments which are needed to be supported. It is an
embryonic artifact—where requirements can be modified or
revised at any time by the PO, based on changing business or
market circumstances. Items in the Product Backlog can have
attributes like “high-level estimation” or “order.”
Product Backlog Refinement: the Product Backlog is refined
at regular intervals, usually at a weekly or biweekly frequency.
The PO is in charge of the backlog refinement process and
relevant stakeholders, e.g., technical leaders from the devel
opment teams that are involved. As a result of the refinement
Fig. 3. AUTILE Framework—Execution. sessions, higher priority items from the Product Backlog are
brainstormed, detailed, prioritized, and roughly estimated.
This refinement process continues until higher priority items
utilized in this regard: 1) For efficiency and design purposes, are analyzed and understood so that a specified item could fit
all kinds of automotive software require specific hardware into one sprint with a clear definition of completion. The
modules or other software components to be in place on a outcome of these sessions is a list of items, over which a
vehicle. These rudiments are known as automotive system specific sprint could be planned.
requirements. Categorized as absolute and optional, these Sprint Planning: it is the last step in the planning phase.
requirements act as guidelines and define the structure of the During Sprint Planning, the PO collaborates with the Scrum
software. Software Architects and Product Managers from team members to target Product Backlog items. The
Tier-2s (software suppli ers) get engaged with Architects of discussion is moderated by the Scrum Master to ensure
Tier-1s (hardware integra tors) and OEMs (car manufacturers) effectivity and settlement of plans. In this process, certain
to define detailed design specifications at a system or platform features or fixes—at the highest priority level and thus, with
level. Software guidelines are extracted from these system the highest certainty to be delivered—could be committed as a
requirements by the product owner (PO). Development team product roadmap item. We found that a quarterly frequency
leads are also engaged by PO to map these requirements onto worked best when AUTILE was deployed at a Tier-2.
specific software modules. 2) Be fore execution of software However, the framework allows flexibility at organizational
development, the electrical/electronic (E/E) design of the levels and provides freedom to POs to define the frequency
automobile is complete and a system-level proposal is which works well in their specific scenarios.
prepared by OEM Architects. This proposal defines the signals 3) Execution: The AUTILE Execution phase begins after a
and messages communicated across the ECUs and it Sprint is planned. During this phase, software is designed and
 written. It has the following set of activities. enhancements, also evolves at this stage. AUTOSAR
Software Design and Architecture: AUTILE emphasizes a requirements (SRS) and Basic Software Module (BSW)
well-defined software design and architecture before imple specifications are taken as a baseline to define the overall
mentation. To achieve this goal, the Scrum Team also in software architecture. Individual BSW configuration
cludes a Software Architect [58]. Moreover, software design parameters and standardized interfaces at the
and architecture for existing items, in case of fixes or

Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

KHAN AND BLACKBURN: AUTILE FRAMEWORK: AN AUTILE METHODOLOGY TO REDUCE AUTOMOTIVE SOFTWARE DEFECTS 5
of the product is prepared. If the contract allows shipment of
the source code, .ARXML, .c and .h files are supplied to the
application programming interface level are defined with a customer. In other cases, formats like .elf or .s19 are provided.
goal to achieve a structured and modular software architecture.
Since AUTOSAR is an evolving standard, extensions, and
deviations to its specifications are expected. This aspect is B. Research Hypotheses
considered up front during the design phase. Similarly, RH1: ASPs developed with some application of AUTOSAR
vendor-specific ECU configuration parameters are planned tend to have fewer defects.
with AUTOSAR-based configuration parameters to attain the RH2: ASPs developed with some application of AUTOSAR
required functionality. For consistency and quality purposes, tend to have a lower DSI.
these ECU configuration parameters are extracted from the RH3: ASPs developed with some application of Agile develop
initially supplied ECU extract and taken into account in the ment tend to have fewer defects.
AUTOSAR-specific .ARXML format. AUTOSAR provides a RH4: ASPs developed with some application of Agile develop
modular and open software archi tecture and thus Tier-2s are ment tend to have a lower DSI.
afforded extensions and deviations to that. Dependencies RH5: ASPs developed with an AUTILE-based approach tend
between BSW modules are considered dur ing the design to have fewer defects.
phase to ensure that the developed software meets specified RH6: ASPs developed with an AUTILE-based approach tend
requirements. BSW module dependencies are docu mented in to have a lower DSI.
the AUTOSAR-specific .ARXML format. Moreover, an SWC
Service Layer is generated so application-level service needs C. Variables
can also be identified during the software development
process. Predictor/independent (Y) variables include: application of
Tasks and Low-Level Software Requirements: Low-level AUTOSAR to ASPs and application of Agile to ASPs. Simi
soft ware requirements and individual tasks are created after larly, response/dependent (X) variables are: the total number
the software has been designed. BSW module definitions in of defects, number of average defects per project, and DSI.
the AUTOSAR-specific .ARXML format are extracted from Defect Severity Index (DSI): The automotive software indus
the initially supplied ECU extract. At this point, the try classifies defects into several categories [21], [22], [47],
development team has a clear understanding of the [60] for prioritization and scheduling purposes. While the
requirements and individ ual tasks, such as creation of static nomencla ture could be used interchangeably (e.g., terms like
source code (in .c format) or generator code (in .jar format) “critical” or “blocker” are used for the highest priority defects
that can be scheduled. ECU configurations are also generated and “minor” or ‘’low” for the lowest priority ones), the
as well as application software components. Unit tests underlying classification philosophy remains consistent. Thus,
accompany the developed software-base to satisfy demanding we decided to adopt the terms used by the Tier-2
requirements of quality assurance. More over, to achieve manufacturers, which deployed the AUTILE framework and
compliance with automotive safety standards, attention is paid supplied data for this research. Accord ing to their Quality
to requirement traceability as well. It ensures that a System, defects are classified as follows: Critical: if the
requirement from high level, broken into lower levels, is not artifact is on the customer’s critical path, the defect makes it
only implemented but also fully tested. In the end, artifacts unusable and there is no acceptable workaround. Significant: if
and documentation are created to prepare for release. the artifact will get on customer’s critical path in a short
Daily Scrum: During Sprint execution, a daily stand-up timeframe, the defect makes it unusable and there is no
Scrum is planned with a focus on three questions: 1) What did acceptable workaround. Medium: if the defect impairs use of
we do yesterday?. 2) What is the plan for today?. 3) What the artifact (not necessarily making it unusable), there is no
dependen cies, risks, and roadblocks do we see that may acceptable workaround. Minor: if a workaround exists but the
obstruct the plan? This way, the development plans are tracked defect causes an inconvenience. The DSI is used as a metric to
on a daily basis. Moreover, it is ensured that commitments are determine the level of severity of defects. DSI is a weighted
met and potential hindrances are tackled. average, ranged 1 to 4, which articulates the severity of defects
Sprint Review, Retrospective, and Shippable Increments: At reported for a specific project. With Tier-2 manufacturer we
the end of the Sprint cycle, Sprint Reviews and Retrospectives worked with, it was rounded to nine decimal places and
are planned. The intent of these activities is to learn from past defined as
mistakes so planning can be improved in future projects. DSI = (Number of Critical Defects ∗ 4 + Number of Signif
Missing items, if any, are added back into the Product Back icant Defects ∗ 3 + Number of Medium Defects ∗ 2 + Number
log for reprioritization and scheduling. A shippable increment of Minor Defects ∗ 1) / Total Number of Defects. (1)
 assigned to each ASP. These ASPs were developed to deliver
IV. RESULTS different types of automo tive software products and services
to leading Tier-1s (hardware integrators) and car
The source of the dataset is a large, highly mature, and
manufacturers in the automotive industry. These products and
leading Tier-2 (software supplier) in the automotive industry.
services include automotive-grade commu nication, operating
Project status reports were acquired from ASPs delivered by
system, microcontroller drivers, and security components. The
the Tier-2. During the development phase, the duration of the
acquired project reports contained fields like
ASPs was kept from six months to one year so they fit into
release cycle of the company. Two to six developers were

Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

6 IEEE SYSTEMS JOURNAL

with alpha value of 0.05 to compare the means of two groups
and determine if the samples are statistically different from
TABLE I each other. Moreover, average defects per project were also
SUMMARY OF RESULTS FOR ALL HYPOTHESES
analyzed with a 2-sample Poisson rate. The results are
summarized in Table I.

A. Total Defects With AUTOSAR: Reduction Realized

P-value was observed as 0.028 with a 2-sample T-test. The
test confirmed that the mean of total defects in ASPs with
AUTOSAR is statistically lower than the mean of total defects
in ASPs developed in a traditional/legacy method (without
AUTOSAR).
B. DSI With AUTOSAR: Reduction Realized
P-value was observed as 0.000 with a 2-sample T-test. The
test confirmed that the mean of DSI in ASPs with AUTOSAR
is statistically lower than the mean of DSI in ASPs developed
in a traditional/legacy manner (without AUTOSAR).

C. Total Defects With Agile: Reduction Realized

P-value was observed as 0.009 with a 2-sample T-test. The
issue type, severity, issue status, priority, resolution, affected test confirmed that the mean of total defects in ASPs with
version(s), components, labels, epic name, AUTOSAR Agile is statistically lower than the mean of total defects in
version, assignee, reporter, time tracking, issue links and ASPs developed in a traditional/legacy manner (without
creation, and closure dates. For this research, issue severity Agile).
(for DSI) and total number of defects for ASPs were taken into
account. The ASPs were of four types: ASPs delivered without D. DSI With Agile: Reduction Realized
an application of AUTOSAR or Agile were considered
traditional/legacy ASPs, ASPs with some application of P-value was observed as 0.000 with a 2-sample T-test. The
AUTOSAR, ASPs with some application of Agile, and ASPs test confirmed that the mean of the DSI in ASPs with Agile is
with an application of AUTILE framework statistically lower than the mean of the DSI in ASPs
For analysis purposes on these independent samples of developed in a traditional/legacy manner (without Agile).
projects, five datasets were formulated such that 1) First
Dataset (DS-1): projects with and without an appli cation of E. Total Defects With AUTILE: Reduction Realized
AUTOSAR, to analyze RH1 and RH2. 2) Second Dataset
(DS-2): projects with and without an application of Agile, to P-value was observed as 0.002 with a 2-sample T-test. The
analyze RH3 and RH4. test confirmed that the mean of total defects in ASPs with
3) Third Dataset (DS-3): legacy projects and the projects AUTILE is statistically lower than the mean of total defects in
with an application of AUTILE framework, to analyze ASPs developed in a traditional/legacy method.
RH5 and RH6.
4) Fourth Dataset (DS-4): AUTOSAR projects and the F. DSI With AUTILE: Reduction Realized
projects with an application of AUTILE framework. 5) Fifth
Dataset (DS-5): Agile projects and the projects with an P-value was observed as 0.000 with a 2-sample T-test. The
application of AUTILE framework. test confirmed that the mean of DSI in ASPs with AUTILE is
Normality confirmation for both variables (total number of statistically lower than the mean of DSI in ASPs developed in
defects and DSI) in all the datasets (DS 1-5) was achieved by a traditional/legacy method.
observing a P-value of greater than 0.05 in Probability Plots.
After that, the data were analyzed with a two-sample T-test G. Comparison of AUTILE With AUTOSAR and Agile ASPs
 ASPs is lower than the mean of total defects in Agile ASPs,
This section is not linked to any research hypothesis.
the finding could not be statistically confirmed with the
AUTILE ASPs were compared with AUTOSAR and Agile
acquired sample. Thus, effectiveness of the proposed
ASPs to de termine efficacy of the proposed framework. In
framework could not be established against Agile ASPs for
case of DS-4, the P-value was observed as 0.039 for total
one variable (total defects). To gain further insights, it is
defects and 0.048 for DSI with a 2-sample T-test. The tests
proposed to increase the sample size of projects. However, this
confirmed that both, the mean of total defects and DSI, in
activity is beyond the scope of this research. In case of DSI in
ASPs with AUTILE are statistically lower than their
DS-5, the P-value was observed as 0.013 with a 2-sample
counter-parts in ASPs developed with an AUTOSAR method.
T-test. The test confirmed that the mean of DSI in AUTILE
In case of total defects in DS-5, the P-value was observed as
ASPs is statistically lower than the mean of DSI in Agile
0.190 with a 2-sample T-test. It is not lower than the alpha
ASPs.
value of 0.05. Although the mean of total defects in AUTILE

Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

KHAN AND BLACKBURN: AUTILE FRAMEWORK: AN AUTILE METHODOLOGY TO REDUCE AUTOMOTIVE SOFTWARE DEFECTS 7
industry. The AUTILE framework provides insights on how to
realize the benefits of software standardization and use Agile
H. Average Defects Per Project: Reduction Realized methodologies for software development in an automotive
Average defects per project were analyzed with a 2-sample company. It can also be used by companies outside the
Poisson rate test. The Poisson rate was observed as 307.818, automotive industry by applying the learning from this
195.414, 169.938, and 128.9 for traditional/legacy, research to their processes in order to improve the software
AUTOSAR, Agile, and AUTILE projects, respectively. It was quality.
observed that the Poisson rate was higher when the framework The findings highlighted by this research open avenues for
was not applied to the ASPs. To gauge statistical significance, other researchers to extend the AUTILE framework for auto
DS 1-3 were tested with a 2-sample Poisson rate test. motive companies by addressing any deficiencies. The
Statistically significant improvement was observed across all research focuses on improving software quality in the
datasets with a P-value of 0.000. The tests confirmed that the automotive in dustry. The research can be extended to other
average number of defects in ASPs with AUTOSAR, Agile, industries, like aerospace, which have stringent quality
and AUTILE is statistically lower than the average number of requirements, or the ones like medical device manufacturing,
defects in ASPs developed in a traditional/legacy method where great attention is paid to processes. Application to the
(without AUTOSAR, Agile, or AUTILE). The results are retail industry, which is
summarized in Table I. currently being disrupted by technology, could also be
explored. Moreover, the research could be replicated among
different product types and technologies or among different
V. CONCLUSIONS, CONTRIBUTIONS, AND
groups and countries to investigate efficiency of the framework
RECOMMENDATIONS for various types of software or within different cultures and
This research is culminated with the following conclusions: geographical settings.
1) ASPs developed with some application of AUTOSAR
face fewer numbers of defects as compared to those without
REFERENCES
it. 2) ASPs developed with some application of AUTOSAR
experience a lower DSI as compared to those without it. 3) [1] “Five trends transforming the automotive industry,” 2017. [Online]. Avail
ASPs developed with some application of Agile face fewer able: https://www.pwc.com/gx/en/industries/automotive/assets/pwc
five-trends-transforming-the-automotive-industry.pdf. Accessed: Jun. 18,
defects as compared to those without it. 2019.
4) ASPs developed with some application of Agile experi [2] P. Pavlinek, “The impact of the 2008–2009 Crisis on the automotive
ence a lower DSI as compared to those without it. 5) ASPs industry: Global trends and firm-level effects in central Europe,” in
developed with AUTILE framework experience fewer Proc. Eur. Urban Regional Studies, 2015, pp. 20–40.
[3] Automotive Revolution & Perspective Towards 2030. Gurugram, India:
defects as compared to those without it. McKinsey & Company. 2016. [Online]. Available: https://
6) ASPs developed with AUTILE framework experience a www.mckinsey.com/industries/automotive-and-assembly/our-insights/
lower DSI as compared to those without it. disruptive-trends-that-will-transform-the-auto-industry. Accessed: Jun.
This research contributes to the body of theoretical knowl 17, 2019.
[4] P. Gao, H. Kaas, D. Mohr, and D. Wee, Disruptive Trends That Will
edge by studying the complexity of deploying standardization
Transform the Auto Industry. Gurugram, India: McKinsey & Company,
and Agile Systems Engineering (ASE) into the automotive Jan. 2016.
software industry and by proposing an automotive software [5] O. Burkacky, J. Deichmann, G. Doll, and C. Knochenhauer, Rethinking
development framework. In a broader perspective, the research Car Software and Electronics Architecture. Gurugram, India: McKinsey
also contributes to the understanding of the complexities of & Company, Feb. 2018.
[6] Z. Fox, How Do We Trust the Software in a Driverless Car? NJ, USA:
standardization and ASE for commercial product development Forbes, Jun. 5, 2018.
environments into environments with established processes. [7] R. N. Charette, “This car runs on code,” IEEE Spectrum, Feb. 2009. [On
The research contributes in the practical application with line]. Available: https://spectrum.ieee.org/transportation/systems/this
deployment of the AUTILE framework to develop ASPs at a car-runs-on-code
[8] A. Haghighatkhah, M. Oivo, A. Banijamali, and P. Kuvaja, “Improving
leading Tier 2 automotive software supplier. This the state of automotive software engineering,” IEEE Softw., vol. 34, no.
implementation activity provides an outline of the procedures 5, pp. 82–86, 2017.
with their respective effects and advantages in the automotive [9] C. Briciu, I. Filip, and F. Heininger, “A new trend in automotive
 software: AUTOSAR concept,” in Proc. IEEE 8th Int. Symp. Appl. [17] D. Kant, M. Buhlmann, and M. Kalhammer, “Being innovative by fol
Comput. Int. Inform., 2013, pp. 251–256. lowing standards—Evolving standards in the automotive industry for the
[10] B. Boss, “Architectural aspects of software sharing and standardization: development of safety related vehicle software,” in Proc. SAE World
AUTOSAR for automotive domain,” in Proc. IEEE 2nd Int. Workshop Congr., 2006, pp. 127–136.
Softw. Eng. Embedded Syst., Jun. 9, 2012, pp. 9–15. [18] S. Brewerton, R. Schneider, and F. Grosshauser, “Practical use of AU
[11] “GENIVI Alliance.” Accessed on: Jan. 12, 2019. TOSAR in safety critical automotive systems,” SAE Int. J. Passenger
[12] K. Sung and T. Han, “Development process for AUTOSAR-based embed Cars - Electron. Elect. Syst., vol. 2, no. 1, pp. 249–57, 2009.
ded system,” Int. J. Control Autom., vol. 6, no. 4, pp. 29–38, 2013. [13] G. [19] M. Jensen and T. Mascolo, “AUTOSAR as a key enabler for
Lami, F. Fabbrini, and M. Fusani, “Is automotive SPICE suitable to assess collaborative product development,” SAE Int. J. Passenger
product lines-based software process?” in Proc. IEEE 2nd Eastern Eur. Cars—Electron. Elect. Syst., vol. 3, no. 2, pp. 193–202, 2010.
Regional Conf. Eng. Comput. Based Syst., Sep. 2011, pp. 157–158. [14] T. [20] D. Ahrens, A. Frey, A. Pfeiffer, and T. Bertram, “Designing reusable and
Schumann, “Standardization in the automotive industry,” Electron. World, vol. scalable software architectures for automotive embedded systems in
118, pp. 20–22, Nov. 2012. driver assistance,” in Proc. SAE Int. World Congr. Exhib., Apr. 2010.
[15] S. Mirheidari, A. Fallahi, D. Zhang, and K. Kuppam, “AUTOSAR model [Online]. Available: https://www.sae.org/publications/technical-papers/
based software component integration of supplier software,” SAE Int. J. content/2010-01-0942/
Commercial Vehicles, vol. 8, no. 2, pp. 544–548, 2015. [21] D. West and N. Thurlow, “Automotive agility—Coming back to its roots:
[16] D. Reinhardt, D. Kaule, and M. Kucera, “Achieving a scalable E/E The story of agility, Scrum and Toyota,” in Proc. IQPC, Automotive iQ
architecture using AUTOSAR and virtualization,” SAE Int. J. Passenger Agile Automotive Conf., May 2019. [Online]. Available: https://www.
Cars Electron. Elect. Syst., vol. 6, no. 2, pp. 489–497, 2013. automotive-iq.com/events-agileforautomotive/

Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE SYSTEMS JOURNAL

[38] K. K. Baseer, A. R. M. Reddy, and C. S. Bindu, “A systematic survey on
waterfall vs. agile vs. lean process paradigms,” I-Manager’s J. Softw.
Eng., vol. 9, no. 3, pp. 34–59, 2015.
[22] E. Knauss, Agile Development in Automotive Software Development:
[39] M. S. Palmquist, M. A. Lapham, S. Miller, T. Chick, and I. Ozkaya,
Challenges and Opportunities. New York, NY, USA: Springer, 2014, pp.
“Parallel worlds: Agile and waterfall differences and similarities [disser
33–47.
tation],” Ph.D. Dissertation, Carnegie-Mellon Univ., Pittsburgh, PA,
[23] K. Hörmann, Agile in Automotive. Version 3.1 ed. Kornwestheim: KU
USA, 2013.
GLER MAAG CIE GmbH, 2017.
[40] A. Sasankar and V. Chavan, “SWOT analysis of software development
[24] J. Löchner, M. Dietrich, and J. Crepin, “Agile development with AU
process models,” Int. J. Comput. Sci. Issues, vol. 8, no. 5-3, pp. 390–399,
TOSAR,” ATZextra Worldwide, vol. 18, no. 9, pp. 84–85, 2013. [25] K. Chari,
2011.
and M. Agrawal, “Impact of incorrect and new requirements on waterfall
[41] “Start-ups disrupting the global automotive and mobility industry
software project outcomes,” Empirical Softw. Eng., vol. 23, no. 1, pp.
2016–2017,” 2017. [Online]. Available: https://store.frost.com/start
165–185, 2018.
ups-disrupting-the-global-automotive-and-mobility-industry-2016-2017.
[26] L. Mahadevan, W. J. Ketinger, and T. O. Meservy, “Running on hybrid:
html, Accessed: Jun. 17, 2017.
Control changes when introducing an agile methodology in a traditional
[42] R. Dove and R. LaBarge, “Fundamentals of agile systems engineering—
waterfall system development environment,” Commun. Assoc. Inf. Syst.,
Part 2,” in Proc. INCOSE Int. Symp., 2014, pp. 876–892.
vol. 36, pp. 77–103, 2015.
[43] “Manifesto for Agile Software Development,” 2001. [Online]. Available:
[27] C. Okoli and K. Carillo, “The best of adaptive and predictive methodolo
https://agilemanifesto.org/, Accessed: Jun. 18, 2019.
gies: Open source software development, a balance between agility and
[44] S. Ashmore and M. Wedlake, “Developing the product your customer
discipline,” Int. J. Inf. Technol. Manage., vol. 11, no. 1/2, pp. 153–166,
really wants: The value of an agile partnership,” Inf. Resour. Manage. J.,
2012.
vol. 29, no. 3, pp. 1–11, 2016.
[28] V. Prakash, N. Senthil Anand, and R. Bhavani, “Agile-fall process flow
[45] M. Ganis, E. M. Maximilien, and T. Rivera, “A brief report on working
model - a right candidate for implementation in software development
smarter with Agile software development,” IBM J. Res. Develop., vol.
and testing processes for software organizations,” Int. J. Comput. Sci.
54, no. 4, pp. 1–10, 2010.
Issues, vol. 9, no. 3, pp. 457–461, 2012.
[46] R. Dove and R. LaBarge, “Fundamentals of agile systems engineering—
[29] L. R. Vijayasarathy and C W. Butler, “Choice of software development
Part 1,” Proc. Int. Symp. Council Syst. Eng. Int. Symp., pp. 859–875,
methodologies: Do organizational, project, and team characteristics mat
2014. [Online]. Available:
ter?” IEEE Softw., vol. 33, no. 5, pp. 86–94, Sep./Oct. 2016.
https://www.researchgate.net/publication/29046
[30] A. I. Khan, R. J. Qurashi, and U. A. Khan, “A comprehensive study of
2513_841_Fundamentals_of_Agile_Systems_Engineering_-_Part_1
commonly practiced heavy and light weight software methodologies,”
[47] B. Katumba and E. Knauss, “Agile development in automotive software
Int. J. Comput. Sci. Issues, vol. 8, no. 1–4, pp. 441–450, 2011.
development: Challenges and opportunities,” in Product-Focused
[31] K. Khong and K. Bently, “Developing agile best practices: Integrating
Software Process Improvement. London, U.K.: Springer Verlag, 2014.
waterfall and agile processes to meet deliveries,” in Proc. IQPC,
[48] P. Hohl, J. Munch, K. Schneider, and M. Stupperich, Forces that Prevent
Automot. iQ Agile Automot. Conf., May 2019. [Online]. Available:
Agile Adoption in the Automotive Domain. Cham, Switzerland: Springer,
https://www. automotive-iq.com/events-agileforautomotive/
2016, pp. 468–476.
[32] D. Rush, “Integrating modern agile engineer processes to bring agility to
[49] S. Maro, “Improving software traceability in the development of au
onboard and embedded product development,” in Proc. IQPC, Automot.
tomotive embedded systems: A research abstract,” in Proc. 22nd Int.
iQ Agile Automot. Conf., May 2019. [Online]. Available: https://www.
Conf. Requirements Eng., Found. Softw. Qual., 2016. [Online].
automotive-iq.com/events-agileforautomotive/
Available:
[33] S. Tran, J. Cullyer, E. Hines, and K. Marks, “On the development of a
https://cefos.gu.se/english/research/publication/?publicationId=237248
high quality software design methodology for automotive applications,”
[50] T. Dyba and T. Dingsoyr, “Empirical studies of agile software de
in Proc. IEE Colloq. ‘Safety Critical Softw. Vehicle Traffic Control, Feb.
velopment: A systematic review,” Inf. Softw. Technol., vol. 50, no. 9, pp.
1990. [Online]. Available: https://ieeexplore.ieee.org/document/189810
833–859, 2008.
[34] N. Perlroth, Why Car Companies Are Hiring Computer Security Experts.
[51] V. T. Faniran, A. Badru, and N. Ajayi, “Adopting scrum as an agile
New, NY, USA: The New York Times. Jun. 2017.
approach in distributed software development: A review of literature,” in
[35] “Software Now to Blame for 15 Percent of Car Recalls,” 2016. [Online].
Proc. 1st Int. Conf. Next Gener. Comput. Appl., Jul. 2017, pp. 36–40.
Available: https://www.popsci.com/software-rising-cause-car recalls/.
[52] T J. Mueller, “Factors influencing the decision to employ agile methods
Accessed: Mar. 13, 2019.
on federal information technology projects,” Ph.D. Dissertation, Robert
[36] “Car recalls rise as industry becomes more high-tech.” [On line].
Morris Univ., Pittsburgh, PA, USA, 2014.
Available: https://www.ft.com/content/15e283a0-59eb-11e8-b8b2-
[53] S. Kiv, S. Heng, M. Kolp, and Y. Wautelet, “Agile manifesto and prac
d6ceb45fa9d0. Accessed: May 23, 2019.
tices selection for tailoring software development: A systematic
[37] “AUTOSAR - The Standardized Software Framework for Intelligent Mo
literature review,” in Proc. 19th Int. Conf. Product-Focused Softw.
bility.” [Online]. Available: https://www.autosar.org/. Accessed: Jul. 23,
Process Improve ment, Nov. 2018, pp. 12–30.
2019.
[54] E. Cetin and P. Onay Durdu, “Blended Scrum model for software de the D.Eng. degree in engineering management from
velopment organizations,” J. Softw., Evol. Process, vol. 31, no. 2, 2019, the George Washington University, Washington, DC,
Art. no. e2147. USA, in 2020.
[55] A. S. Campanelli and F. S. Parreiras, “Agile methods tailoring—A sys He personally developed and led teams to develop
tematic literature review,” J. Syst. Softw., vol. 110, pp. 85–100, 2015. [56] S. several high volume, high quality automotive, and
Paul and K. John Singh, “Be agile: Project development with scrum consumer electronic products. He is currently a Se
framework,” J. Theor. Appl. Inf. Technol., vol. 40, no. 1, pp. 105–112, 2012. nior Engineering Manager at Mentor - A Siemens Business, where he
[57] “Agile for automotive.” Mar. 22, 2019. [Online]. Available: https://www. developed and assisted the development of multiple automotive-grade
automotive-iq.com/events-agileforautomotive. solutions.
[58] M. Galster, S. Angelov, M. Meesters, and P. Diebold, “A multiple case
study on the architect’s role in scrum. Product-focused software process Timothy D. Blackburn received the bachelor’s de
improvement,” in Proc. 17th Int. Conf., Nov. 22–24, 2016, pp. 432–447. gree (summa cum laude) from the William States Lee
[59] D. Durisic, M. Staron, and M. Tichy, “Identifying optimal sets of stan College of Engineering, Charlotte, NC, USA, in 1985,
dardized architectural features,” in Proc. 11th Int. ACM SIGSOFT Conf. the MBA degree from the Kenan-Flagler School of
Qual. Softw. Architect., May 2015, pp. 103–112. Business, Chapel Hill, NC, USA, in 1991, and the
[60] P. Hohl,M. Stupperich, J.Munch, and K. Schneider, “An assessment Ph.D. degree in systems engineering from George
model to foster the adoption of agile software product lines in the Washington University, Washington, DC, USA, in
automotive domain,” in Proc. IEEE Int. Conf. Eng., Technol. Innov., Jun. 2012.
2018, pp. 1–9. He is currently a Professorial Lecturer with the De
partment of Engineering Management and Systems
Ahmed M. Khan received the B.E degree in elec Engineering, GWU, and is the Director of Operational
tronics engineering from the National University of Excellence for Upjohn, a division of Pfizer, Richmond, VA, USA. Dr.
Sciences and Technology, Pakistan, in 2007, the M.S. Blackburn is a Licensed Professional Engineer. He holds a Master Black Belt
degree in electrical engineering from Michigan State in Six Sigma.
University, East Lansing, MI, USA, in 2010, and

Authorized licensed use limited to: University of Vermont Libraries. Downloaded on July 26,2020 at 21:12:45 UTC from IEEE Xplore.
Restrictions apply.