Business Analysis Benchmark - 2009

THE PATH TO SUCCESS

By: Keith Ellis

BusinessAnalysisBenchmark ThePathtoSuccess: ExecutiveSummary

The Business Analysis Benchmark is a large scale survey effort by IAG Consulting designed to assess the link between an organizations maturity in requirements definition and management and project outcome. This years theme is The Path to Success; the study presents detailed findings on the impact of business requirements maturity and analyzes the strategies and tactics needed to implement enhanced requirements maturity. IAGs Requirements Maturity Model (RMM) is a means to benchmark an organizations effectiveness in requirements definition and management by looking at maturity in six underlying capabilities. Like similar standards-based models, it classifies companies based on observed, tangible competency in each capability to make an objective assessment of overall maturity. Using this approach, IAG found: 1. Requirements maturity improvement is highly correlated with improvement in development effectiveness. 2. Requirements maturity cannot be changed through continuous focus on only one underlying capability. 3. High requirements maturity companies can be found amongst the followers of many different approaches to development such as Agile, Iterative, Plan Driven (Waterfall), and Prototyping/Visualization centric methods. The above findings validate the Requirements Maturity Model as a mechanism for identifying the impact of poor requirements practices on companies, quantifying the performance change expected for a particular organizations situation, and, diagnosing the changes needed should a company choose to pursue a path of improvement. This report identifies both the strategy and tactics of enhancing requirements definition and management maturity. The statistics presented in the Business Analysis Benchmark not only debunk a number of commonly held beliefs about development effectiveness, it shows that the average organization wastes a large proportion of their IT development budget due to poor requirements maturity. To be clear, 75% of organizations surveyed waste over one in three dollars spent in IT development and implementation annually as a result of to poor requirements maturity. These findings detail key issues and actions needed to recapture this wastage. Key findings of the Business Analysis Benchmark include: 1) Requirements maturity has a strong positive correlation to EVERY major measure of development efficiency assessed. On time performance, on budget performance, on function performance, overrun magnitudes for each of the above, and project success rates all improve as requirements maturity increases. On average, performance virtually doubled on each of these metrics as organizations progressed from using an ad-hoc

Page 1 of 33

IAG Consulting, 2009

www.iag.biz

approach for requirements definition and management to having institutionalized and consistent competency in all capability areas: a. Average on time performance of technology This survey is a testament to the projects increased by 161%. need for investing in your b. Time overruns on projects reduced by 87%. requirements process to deliver value to your stakeholders. c. Average on budget performance for technology projects improved by just over 95%. Scott Hebner, Vice President Marketing and d. Budget overruns reduced by just under 75%. Strategy, IBM Rational Software e. Percentage of projects that deliver the functionality needed by the business rose by just over 75%. f. Average functionality missed dropped by approximately 78%. 2) A total of 74.1 per cent of survey respondents were classified as immature Level 1 or Level 2 organizations (where the highest maturity Level is 5). These organizations waste 39% and 34% respectively of their development budget due to poor requirements definition and management maturity. This wastage due to poor requirements maturity will increase to over 50% of IT spending on development and a significant proportion of the maintenance budget in certain circumstances. 3) Poor requirements definition and management maturity undermines organizational competitiveness. Organizations with poor requirements maturity expend far more time, budget, and management effort to achieve the same results as organizations with high maturity. For example, organizations with low requirements maturity achieve the business objectives of a project initiative a mere 54% of the time while taking 35% more time to achieve this poorer result. This impact may be so significant over time that it shifts fundamental financial performance metrics such as Return on Assets. IAG found Level 4 companies, on average, outperform the ROA of their peer group competitors by 10%. 4) While this report discusses and busted a number of helpful to me, not only to understand the findings relating commonly held beliefs about requirements and to my current situation, but also development efficiency, two issues garnered significant what CEO and CIOs are interested in. attention and support from the reports external review Carol Deutschlander, panel: Home Hardware Stores Limited CIOs cannot simply attempt to hire great analysts and expect the problem of poor requirements to go away: In fact, lower skilled people in a high requirements maturity company significantly outperform highly skilled people in a low requirements maturity company. Agile, Waterfall, Iterative, Prototyping/Visualization have immaterial performance differences for any given level of requirements definition and management maturity. There is a raging debate amongst development methodologists each espousing one method over another. This study finds that changing development methods - in the absence of also improving requirements competence in the areas of process, techniques, staff, technology, organization and deliverables - only nominally IAG Consulting, 2009 www.iag.biz
This [report] was extremely

Page 2 of 33

improved or reduced overall success rates on projects. IAG has had excellent results with all these approaches, and, the findings of the Business Analysis Benchmark do not endorse any one method over another. The key issue for readers: the overall level of requirements maturity has a MUCH greater effect on project outcome than the development method selected. The Business Analysis Benchmark describes the issues and impacts of each level of the organization, and the role each plays in moving a company forward along the path of maturity. This report has a preface that describes the survey, maturity model, and basic facts surrounding the impact of requirements maturity on project outcome. The remainder of the report is organized along the lines of readership group discussing the key findings as they relate to: 1. The CEO: how does requirements maturity impact overall organizational competitiveness? 2. The CIO: how does IT Leadership approach the major issues in making requirements definition and management change? 3. The Project Management & Analyst Leadership: what is the effectiveness of various paths of change, and what are the required activities to bring improvement? In addition to this content, IAG has also asked a series of external reviewers to comment on survey findings. These invaluable insights are captured in the green call-out boxes throughout the report.
I've worked carefully through the Benchmark Study. It's terrific stuff -some of the conclusions are almost iconoclastic, and yet they make tremendous sense once you analyze them. And the RMM is an excellent tool -- of course it does and should heavily parallel CMM / CMMI, but it also provides tremendous value added as you've applied/customized it to Business Analysis practice. Senior Requirements Specialist Major Property & Casualty Insurance Company

Page 3 of 33

IAG Consulting, 2009

www.iag.biz

TheBusinessAnalysisBenchmark2009: ThePathtoSuccess
Contents
Business Analysis Benchmark The Path to Success: Executive Summary The Survey The Requirements Maturity Model
Why is the Requirements Maturity Model so Critical? How is the Requirements Maturity Model Structured? Measuring Maturity The Business Analysis Benchmark 2009 Survey Focus

1 5 6
6 6 7 7

The Facts: The Impact of Requirements Maturity on Project Outcome

Poor Requirements Definition and Management Wastes 34% of the Average Organizations IT Budget Composition of Project Portfolio Increases Wastage at Larger Companies There is a Multiplier Effect of Requirements Maturity on Large Projects Concluding Thoughts: the Impact of Requirements Maturity on Project Outcome Requirements Maturity and its Impact on Organizational Competitiveness
What is the impact of poor objectives efficiency?

9
10 11 12 12 13
14

The Path to Success: The CEO Perspective on Business Requirements

Increasing the Cost of Large Scale Change Closing Thoughts for the CEO The CEO Path to Success

13
15 15 16

The Path to Success CIO Perspective on Business Requirements Maturity

CIO Issues in Addressing Requirements Maturity
Issue #1 Requirements Maturity is More Important to Outcome than the Development Methodology Selected Issue #2 Myth-Busting: Simply Hire Good Analysts and the Requirements Problem Goes Away Issue #3 Requirements Definition and Management takes a Significant Commitment of Time and Effort Issue #4 Fixing Requirements Definition and Management Maturity is not Free Issue #5 Everyone Recognizes Requirements are Important But Do They Take Action? Issue #6 There is a High Probability Your Company is a Level 2 or Lower Maturity

17
17
17 18 19 20 21 22

The Path to Success Project Managers and Analyst Leadership

The Essential Baseline Targeted Approach to Improvement The Maturity Multiplier Effect Companies Tend to Over-Emphasize Documentation Standards The Impact of Technology Implementation Scorecarding What is it, and why it may be a valuable tool Barriers and Accelerators Focusing on the Right Areas at the Right Time Optimizing the Path of Success
At Level 1: Making Sure Requirements are Performed At Level 2: Having a Process that is Defined, and Look Toward Getting This Implemented At Level 3: Have a Process that is Implemented, and Look Toward How to Best Institutionalize this Process At Level 4: Have a Process that is Institutionalized, and Look for Opportunities to Optimize and Continuously Improve this Process.

Conclusions for the CIO: Getting on the Path to Success

24
24 25 26 26 26 27 28 29
29 29

22

30

Contacting the Author About The Survey Design About the Author About IAG Consulting Page 4 of 33

Conclusions for Project Managers and Analyst Leadership the Path to Success

32 32 33 33 www.iag.biz

30

30

IAG Consulting, 2009

TheSurvey
Last year, over 22 million business and IT professionals across 80 countries and in 10 languages benefited from the statistics generated from the Business Analysis Benchmark. This year's survey theme - The Path to Success - identifies a roadmap for maturing requirements definition and management practices. This study is about getting repeatable success on strategic IT projects. In Q2 of 2009, just under 550 companies chose to participate in the Business Analysis Benchmark Path to Success survey, leading to 437 qualifying responses. This survey was designed by IAG Consulting, a world leader in requirements definition and management, with the intent to assess the link between an organizations maturity in requirements definition and management and project outcomes. For more details on the survey questions, please see the section About The Survey Design at the end of this report. The Business Analysis Benchmark statistics only include respondents that met the following three criteria: 1) The company spends over $1 million annually in application development (net of hardware) or software implementation. 2) The individual must have experience with business requirements and project management where net new functionality is added to the business. 3) The company must have run at least four projects in excess of $250,000 in the last 12 months. These criteria ensured that only experienced professionals with knowledge of requirements definition and management issues would be included in survey results. The results are weighted toward medium and large sized commercial companies, in North America. The sampling is summarized below:
Position Executive:HeadofIT,CIO,Headof Development,LineofBusinessExecutive HeadofPMOorProjectManager HeadofBusinessAnalysisCompetency CenterorBA Other %Respondents

12.2% 27.1% 52.5% 8.3% 100.0%

%Respondents 3.9% 17.7% 9.9% 8.7% 8.0% 6.0% 1.1% 1.4% 14.4% 5.0% 9.2% 6.7% 2.1% 6.0% 100.0%

NumberofEmployeesinCompany 199 100to499 500to2,499 2,500to4,999 5,000to9,999 10,000+

%Respondents 6.2% 14.3% 20.3% 11.5% 8.5% 39.2% 100.0%

Industry Energy,Resources&Utilities FinancialServices Insurance Government&SocialServices Healthcare&Pharmaceutical Manufacturing Media&IndustryAnalysts Military&Defense Professional&ITServices Retail,Transportation&Distribution Software Telecommunications Education Other

Region UnitedStates Canada Western&EasternEurope India/Pakistan Asia/Pacific Africa(mainlySouthAfrica) MiddleEast Central/SouthAmerica

%Respondents 233 116 26 24 22 6 5 5 437

Page 5 of 33

IAG Consulting, 2009

www.iag.biz

TheRequirementsMaturityModel
The concept of capability maturity has been around since Deming first started the quality movement in the 1950s. Since then, hundreds of maturity models have been developed with surprisingly few focused specifically on business requirements. IAGs Requirements Maturity Model (RMM) figures prominently into the Business Analysis Benchmark 2009 Structure.

Why is the Requirements Maturity Model so Critical?

Executives today are rightfully cynical about general statements of ROI and return. Every investment contemplated somehow magically seems to hit the requisite hurdle rates, while few investments realize the benefits anticipated. We need to change this return fuzziness and be more precise in evaluating exactly where benefits from improvement are delivered and why. The Requirements Maturity Model achieves the goal of improving return assurance and is used throughout the Business Analysis Benchmark. Whenever IAG refers to Requirements Maturity or Requirements Definition and Management Maturity in this report, it is referring to its level of maturity with respect to the IAG Requirements Maturity Model.

How is the Requirements Maturity Model Structured?

IAGs Requirements Maturity Model has two dimensions: 1) Maturity Level: IAGs is a staged maturity model similar to those used by several industry standards bodies. An organization progresses up the ladder (to the right) as goals are achieved and thresholds surpassed. Each level of maturity shifts the emphasis to different requirement practice characteristics. Each level builds a foundation for succeeding levels. 2) Capability Area: Six capability areas are assessed and combined to determine the maturity level of an organizations requirements management practice. These six are the fundamental building blocks for requirements definition and management and include: a. Process: Definition, usage, and management of requirement procedures. b. Practices & Techniques: Definition of how analysts will perform work, the efficiency and effectiveness of these activities. c. Technology: Provision, usage, and integration of software tools in the Page 6 of 33 IAG Consulting, 2009 www.iag.biz

context of the requirements practice. d. Staff Competency: Level of knowledge, skills, and ability of the workforce. e. Deliverables: Definition, production, and usage of work products as output from the requirement process. f. Organization: Organizational model and services delivered to stakeholders, the provision of resources and resource management in the delivery of these services, and the framework of process and tool governance.

Measuring Maturity
Overall Maturity is a composite of performance across the six capability areas. IAG weights each area slightly differently based on our experience in what drives effective long term performance and consistency in requirements execution. Hence, each respondent has maturity determined within each of the six capability areas, in addition to a single aggregate maturity score. Maturity is very step-like in nature. There is no Level 2.3 for example. An organization progresses up the ladder as tangible goals are achieved and thresholds are surpassed. From time-to-time, IAG may present more granular sub-step data to readers to assist in visualizing trends; however, our intent is to show the progression of skills that happen at Level 1, at Level 2, at Level 3 etc.
Maturity Levels Capabilities
Process

Level 1

Level 2

Level 3

Level Level 4 5

Staff Competency

Technology

Practices and Techniques

Organization Process Deliverables/ Results

The Business Analysis Benchmark 2009 Survey Focus

Any single research effort cannot cover the entire waterfront without making the survey impossibly long. IAGs main concentration in 2009 was to carefully examine maturity Levels 1 through 4. IAG is not presenting data at the extremes of Levels 0 or 5. Be aware, organizations with no requirements processes or standards in place whatsoever will substantially underperform the Level 1 results depicted in this survey. The characteristics of organizations for each level and capability area are listed on the following page.

Page 7 of 33

IAG Consulting, 2009

www.iag.biz

Process

Practices & Techniques

Deliverables

Technology

Organization

Staff Competency

0 Incomplete

No defined process

No defined practices & techniques

Requirement documents are not a standard deliverable

Technology is not considered or used in capturing business needs

Organization does not see need to support business analysis practice

No knowledge of fundamental concepts and skills for requirements delivery

1 Performed

Informal/ inconsistent definition and execution

Informal & Inconsistent use

Informal, nonstandardized documentation

Standard generic desktop tools. No planned approach to tool usage

Limited Support. Undefined Roles. Little Training

Have a fundamental knowledge of most core concepts & skills

2 Defined

Defined activities performed on many projects

Best practices used. Guidelines defined but not mandated

Templates, guidelines, standards defined. Not mandated. Used on most projects Use of standard deliverables mandatory. Quality standards well defined. Used on majority of projects

Requirement automation software tools provided. Inconsistent usage

Organization has delegated support of business analysis best practices and staff development to an individual or team

Good knowledge of all concepts and key BA skills

3 Implemented

Standards implemented and followed on majority of projects. Detailed task definition

Standards implemented used on majority of projects

Requirements management software implemented and mandatory

A formal Requirements/BA organization established

Good demonstrated ability in required competencies

4 Institutionalized

Universally applied. Integrated with PMLC/SDLC

Measurements of compliance, and effectiveness are applied regularly

Meets expectations on virtually all projects. Integrated with PM/SDLC. Fully automated

Required management software integrated with project management and application life cycle management software

Metrics are applied against the center of support for Business Analysis. Organization measures effectiveness of training

Solid knowledge, skills, and ability. consistently applied

5 Optimizing

Process definition is updated based on metrics, changes to corporate strategy, and industry innovations

Defined practices & techniques updated based on metrics, changes to corporate strategy, and industry innovations

Updates performed to incorporate industry best practice innovations and updates in enterprise strategy

Enhancements implemented. Usage continuously improved

Requirements/BA organization is continually improved. External audits applied to validate measurements

Advanced Level of Proficiency

Page 8 of 33 IAG Consulting, 2009 www.iag.biz

TheFacts:TheImpactofRequirements MaturityonProjectOutcome
Requirements maturity may be more important than any other single factor in the determination of overall development effectiveness. The graph below illustrates that low requirements maturity organizations underperform high Performance changes to on time, on budget, on Function, requirements maturity organizations on On Objective Development, BY MATURITY LEVEL EVERY measure of development efficiency. Not only are high 100% requirements maturity organizations 90% noticeably better at servicing the needs 80% % of Projects Delivered on A of the business, they perform nearly TIME 70% % of Projects Delivered on B twice as well on every measure of Budget 60% C C % of Projects Delivering All In % of all development productivity: Required Functionality 50%
projects

On time delivery On budget delivery Percentage of projects delivering the required functionality Percentage of projects deemed successful

40% 30% 20% 10% 0%

D B A

% of Projects Deemed Successful

Level 1

Level 2

Level 3

Level 4

Requirements Discovery & Management Maturity of Organization N=437 Source: Business Analysis Benchmark, 2009

A total of 74.1% of the companies surveyed in The Business Analysis Benchmark 2009, were found to have a low level of requirements maturity1. To underscore the magnitude of impact that requirements maturity has on the performance of the IT organization take the example of a company that successfully transitions from Level 1 to Level 4 maturity in requirements definition and management. The results of the Business Analysis Benchmark showed the average organization: Improved on time performance of technology projects "Even after working on understanding requirements best increased by 161%. practices for the past year and a Reduced time overruns on projects by 87%. half, I am still amazed at just how much of an impact that good Improved average on budget performance for requirements play in the software technology projects by just over 95%. development life cycle. Your performance indicators on page Reduced budget overruns by just under 75%. nine just blew me away." Improved the per cent of projects that deliver 100% of the functionality needed by the business by just over Rajesh Nakhwa Vice President 75%. M&T Bank Reduced average functionality missed by approximately 78%.

Maturity Level of 1 or 2 where maximum maturity is 5 IAG Consulting, 2009 www.iag.biz

Page 9 of 33

The notion that the business can produce requirements on its own in the absence of mature requirements definition and management competency is At one company I worked for we absurd; it is akin to a CFO attempting to prepare financial went from begging projects to reports in the absence of knowing Generally Accepted take on analysts to not having Accounting Principles. The capacity for requirements enough of them to meet demand. planning, elicitation, analysis, documentation and Real people doing real projects management must lie somewhere within the organization. saw the difference it can make. This capacity may exist in a wide variety of structures No amount of improved testing including centralized center of excellence, a distributed capability, improved analyst function, with developers (in the case of agile development techniques, or even practices), or in the hands of user interface experts or improved project management, can make up for poor prototyping experts (in the case of visualization/prototyping requirements and analysis centric methods) so long as it is effective. In the absence of capability. this analysis capacity, IT development will be about half as Brenda Kerton productive as an IT organization with high levels of Past Chair, Body of Knowledge requirements definition and management maturity.

Poor Requirements Definition and Management Wastes 34% of the Average Organizations IT Budget

Committee, Member Certification Committee, International Institute of Business Analysis (IIBA)

The Business Analysis Benchmark found Average Cost of a $250,000 organizations with Level 1 maturity, Project by Maturity Level intending to build a $250,000 application, will spend, on average, $356K $370,000 $356,000; absorbing much of this cost $343K $350,000 as unplanned overrun (about $72,000), $330,000 but also absorbing extremely high $310,000 $309K Additional Cost to Achieve additional maintenance to produce 100% of Target Functionality $290,000 functionality missed in the first release. Average Cost of Initial Project $274K $270,000 $261K Because low maturity organizations miss $250,000 $257K so much functionality in the initial deployment of their applications, and are relatively inefficient at defining stakeholder needs, post-deployment Maturity Level maintenance costs add an additional N=437 13.7% on top of already excessive Source: Business Analysis Benchmark, 2009 overruns. Conservatively, for an organization with 50% of its application development and maintenance budget allocated to maintenance, poor requirements maturity consumes one in every seven dollars available for maintenance. The total wastage of combined development and maintenance budget is illustrated to the right.

Page 10 of 33

IAG Consulting, 2009

www.iag.biz

IAG believes if an IT organization: is continuously late in delivery, is continuously well over budget, continues to deliver only one third of projects successfully; and, consumes unnecessarily large % of Development Budget Consumed amounts of its resources in by Poor Requirements Practices maintenance rather than 50% delivering substantially new 46.1% 45% functionality to the business, a crisis of confidence in the leadership of this IT organization will eventually occur. Average Level 1 and Level 2 organizations were found to waste 39% and 34%, respectively, of their development budget. This wastage is a result of poor requirements practices which generate high costs of initial development and high downstream maintenance costs.
40% 35% 30% 25% 20% 15% 10% 5% 0%
38.9% 34.8% 33.6%

20.9% 20.4%

7.8% 6.8% 1.2% 1.9%

% of Development budget consumed by Poor Quality Requirements Practices % of Time Wasted by Poor Quality Requirements

0.0% 0.0%

Maturity Level
N=437 Source: Business Analysis Benchmark, 2009

Composition of Project Portfolio Increases Wastage at Larger Companies

No matter what the project Wastage by Portfolio Mix size and project portfolio mix, Average of Level 1 and Level 2 Companies Versus Level 3 Companies requirements maturity will 60% improve performance. 50% However, wastage levels dramatically rise for low 40% requirements maturity 30% companies when there is Wastage: Level 1 & 2 Companies 20% great variability in the sizes of Wastage: Level 3 Companies 10% projects in the portfolio. At the extreme ranges of the data 0% 25%-50% of 50%-70% of Over 70% of 0% of Project Less than illustrated to the right, Projects in Projects In projects in over $5 Million <25% of projects in Portfolio over Portfolio over Portfolio over companies have fairly $5M $5M $5M portfolio over homogeneous portfolios; $5 Million either there are no very large Source: Business Analysis Benchmark, 2009 projects, or the vast majority of projects are very large. At these extremes, companies of similar requirements maturity have almost half the wastage of companies with a mixed portfolio of project sizes. The converse is the critical finding of this research; companies with portfolios of significantly varying project sizes experience nearly twice the wastage due to requirements than organizations with relatively homogeneous portfolios of projects. IAG concludes it is far more difficult for companies with widely varied portfolios to deliver on budget and on function in the absence of requirements maturity. The above data does not bode well for the vast majority of Fortune 500 class companies. These companies are very likely to have mixed project portfolios and experience wastage levels Page 11 of 33 IAG Consulting, 2009 www.iag.biz

between 40% and 50% due to requirements immaturity far higher than projected by looking at simple averages. However, the data also indicates: 1. There are other ways to boost IT productivity, like trying to make the project portfolio more homogeneous, rather than focusing on requirements. However, these tactics appear to be less effective in cumulative effect than the improvement made by enhancing requirements definition & management maturity. 2. Companies at a maturity level in excess of 3.0, exhibit very little difference in performance regardless of the composition of the portfolio. As requirements maturity increases, the ability of a company to effectively deal with project portfolio volatility also increases.

There is a Multiplier Effect of Requirements Maturity on Large Projects

The data above paints a bleak picture for organizations attempting to implement a single project of significantly different magnitude than those in their current portfolio. If a low requirements maturity company, dealing mainly in projects under $1 million, were to undertake a single very large project budgeted at $5 million, it is likely to spend between $7.5 and $8.9 million on that project before the full required functionality of the business is delivered. This means an organization could be spending up to 63% more for this project than a company with high requirements maturity. Given the extreme time delays and probability that the primary objectives of development will also be missed, it is unlikely that a company with low requirements maturity could undertake a large scale project successfully. This finding replicates the results of the study undertaken by IAG in 2008.

Concluding Thoughts: the Impact of Requirements Maturity on Project Outcome

It is an understatement to say that poor requirements definition and management maturity costs the industry billion of dollars annually. Not only is the loss expressed in raw dollars, there is a multiplier effect of this cost in other areas of the business both inside and outside the technology function. If left unchecked, it is likely that poor requirements maturity will lead to a crisis of confidence in IT, as well as undermine an organizations ability to define and implement projects of significant magnitude.
The IAG Business Analysis Benchmark study confirms that most companies waste over 30% of their IT budgets on requirements related issues. The solution is a combination of people, process and technology that empowers companies to improve the maturity of their application definition function." Mitch Bishop Chief Marketing Officer iRise

Page 12 of 33

IAG Consulting, 2009

www.iag.biz

ThePathtoSuccess: TheCEOPerspectiveonBusiness Requirements

Suggesting to a CEO that continuous improvement in business requirements definition and management should be on their top 5 priority list seems ridiculous. A typical CEO would think of business requirements as a tactical activity performed by mid-level members of the organization. Further, their impression is likely: while requirements are important, the activity is somewhat technical in nature rather than strategic. How wrong. Lets put Business Requirements into the context of the CEO and separate the tactical role from the strategic executive role. As the sources of impact and benefit of requirements maturity improvement are uncovered, the direct impact of requirements maturity on the CEO becomes apparent. For many CEOs, the idea of wasting 34 of every dollar spent on IT development may be sufficient to incite interest in participating in improving requirements definition and management and eliminating this waste. However, this is still a fairly tactical discussion and it is almost entirely within the technology domain. The strategic discussion for CEOs centers on issues that are mainly within the business domain (as illustrated to the right): 1. The efficiency of an organization in setting and achieving business objectives where these touch on technology deployment. 2. The impact of requirements quality on the cost of large scale change initiatives.

Sources of Benefit
Business Objectives Efficiency

Business Domain
Business Analyst Productivity Major Change Efficiency

Technology Domain

IT Development Efficiency

If time and cost issues associated with poor requirements were entirely contained within the technology function, business requirements would not be an issue for the CEO. However, the two domains of benefit described above are inherently cross-functional in nature, sit within the business domain, and directly impact the CEOs ability to achieve organizational improvement.

Requirements Maturity and its Impact on Organizational Competitiveness

Technology is an enabler that pervades organizations and creates a differentiated customer experience, differentiated cost base, differentiated channel structure, and so on. It makes sense that when there is inefficient communication between business and technology, business owners are far less capable of achieving business objectives; hence the term objectives efficiency. By objectives efficiency, we mean organizations with poor business requirements take more management effort to and achieve POORER results than organizations with high degrees of requirements maturity. Page 13 of 33 IAG Consulting, 2009 www.iag.biz

What is the impact of poor ob bjectives effic ciency? verage, orga anizations at a low level of maturity in their On av requirements prac ctices achiev the busin ve ness objectives of a ive 54% of the tim and miss one me s technology initiati a mere 5 quarte of the stat er ted project o objectives in every proje n ect execu uted. The lo requireme ow ents maturity organizatio take y ons 35% m more time to achieve the poorer re ese esults. This m means the low re equirements maturity org ganization ex xpends lots o effort to of achieve poor resu in every p ults place that te echnology touches usiness; i.e., their efficien in achiev ncy ving objectiv is very ves the bu low. H High requirem ments matur organizations are the opposite, rity e and a very effic are cient at achie eving busine objective ess es.
[a m ere 54 of organizat 4% tions achieve bus siness objective of es development] is astounding truly depicts the importance of investing in a requirements solution that incorpo orates requirem ments best practic ces." Scott Hebner, t Vice P President Marke eting and Strategy IBM Rational S y, Software

The CEO issue is one of competitiveness. Can a comp pany remain competitiv over many years n ve if man nagement is so inefficien utilized (s ntly spending twice as much effort to ac h chieve a far lesser result)? To illustrat this point: take two hy te ypothetical organization one of Le ns, evel 4 maturit and ty of aturity. Each is initially try h ying one o Level 1 ma $250,000,000 to ach hieve 100 ob bjectives per month. Ass r sume each objective, once achieved, yields o $200,000,000 Difference in enue Reve $25,00 in increme 00 ental and su ustainable Performa ance $150,000,000 revenue annually to the busin ness and add ds $100,000,000 people able to add value to the bus siness new p $50,000,000 in the future. Ove the course of 10 years, the er e , Level 4 maturity organization o outperforms the $0 1 2 3 4 5 6 7 8 9 10 on million.2 Level 1 organizatio by $215 m
Sou urce: Business Analysis Benchm mark, 2009

Year

r ts rganizations are generally slower to innovate an nd IAG believes low requirement maturity or t higher requir rements mat turity compe etitors. This re esult will eve entually adapt to change relative to h show up in the overall financia performan al nce of the company. mpact, IAG c compared To highlight this im agement efficiency metr rics from fina ancial mana statem ments of pub blically trade survey ed respon ndents. To the right is a graph showing the Av verage Retu on Assets of surveyed urn d comp panies VERSU THEIR PEER GROUP3 fo US R or each requirement maturity le ts evel. This da ata shows Level 4 mat s turity organiz zations outpe erformed the average RO of their pe OA eers group by almost 10%. i.e., if th average ROA p 1 he for an industry is 7% on average, Level 4 %, ustry had an ROA of 17% %. players in this indu

The m magnitude of the efficie o ency gap in y year 1 (the L Level 4 is 45% more efficient than the Level % e 1) yields an expon nential separ ration in reve enue growth with only nominal assumptions. h
2

Show is the ave wn erage delta between ea ach compan nys ROA and their individual industry peer d y group ROA for any given leve of requirem p el ments maturiity. ROA me easure and p peer group v value used w trailing 12 month, bo as report was 1 oth ted by Reute ers.
3

Page 14 of 33 e

IAG C Consulting, 20 009

www.iag.biz z

While evidence is not conclusive4, the obvious trending of ROA as maturity increases leads us to believe that organizations with higher requirements maturity yield an up to 10% higher return on assets than organizations with low levels of requirements definition and management maturity.

Increasing the Cost of Large Scale Change

Lets assume for a minute that any change driven by the CEO will be relatively large scale. It is in these larger scale initiatives that requirements immaturity may wreak havoc and lead to catastrophic results. For example: Where a Level 1 requirements maturity company funds a $100 million initiative, it has a very high probability of undertaking an additional $60 million in unplanned expenditure despite best budgeting and control efforts due to poor requirements. Further, this same Level 1 maturity company will likely experience such a long implementation cycle and poor resulting functionality match, that the application installed has a high probability of being untenable. Published research from Gartner, InfoTech Research and others have all stated over 70% of application failures are traceable back to poor requirements.5 As the magnitude of these CEO-driven-changes rises and the mix of projects in the portfolio becomes more widely varied, the probability of requirements-driven failure also scales to the point where low maturity organizations become ineffective at making larger scale change. The risks caused by low requirements definition and management maturity are extreme when dealing with larger scale development for low requirements maturity organizations. This impacts CEOs in: The likelihood of being faced with unsuccessful change initiatives after substantial management effort and long delays The probability of needing to show overrun contingencies on financial statements to satisfy current reporting legislation. Being far slower to respond to technology-based competitive threats.

Closing Thoughts for the CEO

Low requirements maturity organizations are inefficient in spending on technology. These same low maturity organizations must expend far more management effort than high requirements maturity competitors in achieving a similar target outcome. Such an erosion of an organizations competitiveness is significant enough to eventually become evident in the relative financial performance of even a Fortune 500 class company.

The current market volatility masks results. Variability within each maturity level is higher than the variability between the maturity levels. IAGs study also did not attempt to address the sustainability of ROA improvement.
4

InfoTech: Flawed Requirements Trigger 70% of Project Failures, February 2006 Gartner: Communication challenges between business teams and technologists are chronic we estimate that 60%-80% of project failure can be attributed directly to poor requirements gathering, analysis, and management. (originally published by Meta Group Research)
5

Page 15 of 33

IAG Consulting, 2009

www.iag.biz

The CEO Path to Success

The theme of this years Business Analysis Benchmark study is The Path to Success. Our emphasis is on positive actions that individuals and organizations can take to improve project performance through improvements in requirements definition and management. In the case of the CEO, the Business Analysis Benchmark study lends strong evidence that the maturity of business requirements definition and management has a significant impact on organizational competitiveness and management efficiency. These facts make knowing the overall level of maturity essential for the CEO. We believe CEOs need to consider the following action steps if they are to follow a path to success: 1) The CEO must signal their support for requirements definition and management maturity, and indicate an expectation of corporate executive support for the initiative. 2) The CEO must press for competence in the process of requirements definition and management, assuring a plan for maturity improvement is in place to achieve these goals and progress is being made on this plan. 3) The CEO should insist on a requirements definition and management risk assessment and mitigation for any abnormally large IT development expenditure. Low levels of requirements definition and management maturity yield high and quantifiable risks for corporations.

Page 16 of 33

IAG Consulting, 2009

www.iag.biz

ThePathtoSuccessCIOPerspectiveon BusinessRequirementsMaturity
Gathering and managing business requirements is a process best jointly owned by business and technology, but this is not always possible. With so much performance impact at stake, CIOs often are in the position of having to step in as sole owners of the requirements process when the business is immature in producing business requirements. It then falls to the CIO to: 1) Diagnose if requirements immaturity is adversely affecting their organization; 2) Own the path to success for improving requirements practices.

CIO Issues in Addressing Requirements Maturity

CIOs work hard to improve organizational efficiency, but must also face a wide array of contradictory messages that make it more difficult to see the path to success. The findings below are designed to shed light on a series of misconceptions and highlight major issues the organization must address if it is to get on a path of sustained development success. Issue #1 Requirements Maturity is More Important to Outcome than the Development Methodology Selected Comparison: Maturity Level of Organization Some organizations have attempted to shrug off the demands of maturing requirements practices thinking that by adopting agile or visualization centric methods, requirements practices become secondary. Nothing could be further from the truth. The Business Analysis Benchmark findings show that different development methodologies have no performance difference for any given level of Requirements Maturity.

in Requirements Discovery & Management VERSUS Method of Development Utilized

100 90 80 70 Per Cent of 60 Projects that 50 are Successful 40 30 20 10 0

Use Agile Use Waterfall Use Iterative Use Prototyping or Visualization 1

4 3 2

Requirements Discovery & Management Maturity

This study finds that changing development practices - in the absence of also improving requirements capabilities in the areas of process, techniques, staff, technology, organization and deliverables - only nominally improved or reduced overall success rates on projects. Companies must take these factors into account should they decide to switch methods. The Business Analysis Benchmark does not endorse any one method over another; IAG has had excellent results with all these approaches. The key issue: the overall level of requirements maturity has a MUCH greater effect on project outcome than the development method selected.

N=437 Source: Business Analysis Benchmark, 2009

In fact, in organizations we speak with, having success with an agile approach happens only when requirements practices are embedded into the agile process. Working in agile projects does impact how requirements practices are applied but does not negate them. Agile is about rapidly and iteratively building a solution. You still have to discover what you are trying to solve! Brenda Kerton Director of Research Info-Tech Research Group

Page 17 of 33

IAG Consulting, 2009

www.iag.biz

Issue #2 Myth-Busting: Simply Hire Good Analysts and the Requirements Problem Goes Away While it is necessary to hire good business analysts, it is also necessary to properly define their role and a host of other factors. The idea that a Level 1 or Level 2 organization can simply hire competent people and the problem of requirements quality will go away is false. The Hiring Challenge: Despite their best intentions, Level 1 and Level 2 organizations tend to only attract Level 1 and Level 2 analyst skills. In fact, the level of knowledge, skill, and ability, is far lower than the average skill level employed by high maturity companies. Level 1 and Level 2 organizations cannot hire efficiently since they have poorly defined analyst roles, required skills, testable competencies, expected services to be delivered by practitioners, and measurement of results. How can an organization hire competently when it does not know what to hire? Conversely, very few organizations surveyed with a maturity level over 3.5 had poor staff capability. Higher maturity organizations are simply better at hiring and developing excellent requirements definition and management capability. Having good people without the other factors does not materially change level of maturity: Great people in a poor requirements maturity organization can only produce mediocre results particularly in comparison to this same level of skill in a high requirements maturity organization. Without question, hiring great people will have some positive benefits, but this will only take the organization a half step forward in maturity. i.e., a Level 2 maturity organization that hires Level 4 competency analysts will get Level 2.5 results. The other capability areas of the Requirements Maturity Model clearly have a significant performance multiplier effect on the skills of analysts. The figure to the right illustrates LOWER skill level analysts OUTPERFORMING significantly higher skill level staff working for lower requirements management maturity companies. Again, requirements management maturity capability areas have an accelerating effect on performance: Maturity across multiple capability areas accelerates personal performance dramatically. Significant immaturity in individual areas stifles performance dramatically.

Performance of People is Impacted Significantly by Overall Requirements Discovery & Management Maturity
85% 80% 75% 70% 65% 60% 55% 50% 45% 40%
Per Cent of Per Cent of Per Cent of Per Cent of Projects On Projects On Projects Projects Time Budget Delivering Meeting Required Business Functionality Objectives

Performance of Level 3 Skilled People in a Level 2 Maturity Organization Performance of Level 2 Skilled People in a Level 3 Maturity Organization

N=437 Source: Business Analysis Benchmark, 2009

How do these results affect outsourcing or staffing strategies? If an organization outsources requirements definition and management it would be well served to ensure the consulting firm engaged exhibits all six requirements maturity capabilities. In the absence of this breadth of competency, like using a staffing firm, it is unlikely to change overall project success rates significantly.

Page 18 of 33

IAG Consulting, 2009

www.iag.biz

The CIOs Issues in addressing Requirements Maturity very effectively dealt with some of the myths and misidentified reasons for success. As an example, we have piloted an iterative approach on some projects. These projects have been incredibly successful and now people want to do it for all projects. I am now working to ensure people understand that Iterative worked not because it is a better methodology than waterfall, but because it was the right methodology for the nature of those projects. We had a business unit that was unclear on what they wanted and needed to see it to know. That means we need to build some, then show, build some more, then show. If the requirements could have been made very clear and well understood up front then waterfall would have been a more efficient methodology. What really allowed these projects to succeed was that the BA could focus the business on getting a clear understanding of what they wanted to do, do it, then move on to the next piece and develop that clear understanding. As you noted, this all comes back to requirements maturity. The comments around the hiring challenge really addressed a couple of prevailing thoughts, one, that a good analyst makes up for all else, or even worse; two, I can get the PM to do the work (yikes). All that said, this issue was really driven home for me when I had one of my analysts approach me about a year ago. She was quite strong in her role and was being asked to take on a system she had never touched before. Everything was new and she couldnt count on her deep system knowledge to help her cut corners and anticipate landmines. In our conversation she said that in her past she would have been afraid to take on something new like this given that she would be the only analyst on the project. Up until then they had always used a knowledge based model (e.g., using personal understanding of the business and system infrastructure to drive the requirements process) and she knew that could no longer work for her in something new like this. Her comfort this time was driven by the fact that we had a process and methodology to engage the customer and elicit the requirements. By following this approach, she was confident that she would bring out the right information allowing IT to deliver the right solution. She was proven absolutely right. Angus Muir IT Director

Issue #3 Requirements Definition and Management takes a Significant Commitment of Time and Effort Maturing an organization requires Total Per Cent of Development Budget committed to that stakeholders understand the Requirements Discovery and Management, BY impact of poor requirements MATURITY LEVEL, on an Average $1 Million Project processes on their overall business 25% performance, and agree to 22.9% participate in a more mature 20% 19.9% process. For companies unfamiliar 18.7% 15% with spending $100,000 or $200,000 % of total 12.7% Budget on without a single line of code being Requirements 10% Discovery & written, this can be a significant Management 5% culture shock. However, IAG found a direct correlation between the 0% amount of effort applied to Level 1 Level 2 Level 3 Level 4 requirements definition and management, and the overall maturity of organizations generally. Higher maturity companies dedicate more effort to ensuring that requirements are right versus their low requirements maturity counterparts. The challenge for CIOs is to have the faith to front-end-load expenditure, knowing that overall costs on the project drop precipitously as a result.

Page 19 of 33

IAG Consulting, 2009

www.iag.biz

Requirements Timeliness Needs to Improve in the Industry The study shows that the average $1 million project required between twelve and nineteen weeks to prepare requirements. IAG believes this is too long and is the direct result of sub-optimal processes. Dramatic compression in time to deliver requirements can be brought to a mature organization irrespective of its development methodology. Hence, there needs to be a focus not only on establishing best practices, but continuously optimizing practices.

Full Time Equivalent Person-Weeks Required to Prepare Requirements for the Average $1 Million Assignment, by Maturity
25 20 17.8 15 12.6 10 5 0 Level 1 Level 2 Level Level 3 Level Level 4
FTE WEEKS to Do Requirements on a $1 Million Assignment

20.4 18.5 18.8 19.3

2.5 3.5 Level 5 (continuously optimizing) organizations would be far more focused on making the stakeholder experience more efficient. For example, IAG itself, on the average $1 Million assignment would complete:

Business requirements (process flow, data flow, business rules, structured statements of system capability, ERD, and the start of the data dictionary) in no more than 10 to 15 working days (as opposed to taking more than an month). Detailed requirements would not take longer than 5 to 10 weeks (about a third the time period averages found in the study).

CIOs need to expend effort to ensure stakeholders participate in gathering requirements. As a whole, the industry also needs to further optimize so that higher quality requirements can be generated with far less consumption of stakeholder time, and in far shorter time periods. Issue #4 Fixing Requirements Definition and Management Maturity is not Free The requirements practice area tends to be underfunded by organizations that are immature. Underfunding improvement creates a self-propagating cycle where the organization cannot materially change maturity levels. Underfunding results in a lack of impact across a broad spectrum of requirements definition and Spending on training and Development per management maturity areas and Analyst to Get Performance Improvement therefore limits change to overall performance on projects. The cycle of $12,000 underfunding change becomes self$10,817 $10,000 perpetuated when first forays into $9,332 improvement are met with less than stellar $8,000 success. The likely result will be waning Average $6,000 Spending on interest in continued focus on change. $6,032 Professional Development $4,000 Unlike low maturity organizations, the Per Analyst $2,614 Business Analysis Benchmark indicates $2,000 that high requirements maturity $organizations invest a tremendous Level 1 Level 2 Level 3 Level 4 amount to achieve their goals. N=437
Source: Business Analysis Benchmark, 2009

Page 20 of 33

IAG Consulting, 2009

www.iag.biz

Luckily, the business case for investment in improving requirements maturity can be easily made based on a relatively small volume of projects. Assume the following baseline scenario: an average Level 1 organization, doing three projects in a year with an application development budget of $966,000. On average, each project expected to be $250,000 likely ends up costing $322,000, and carries an additional $34,400 in maintenance post deployment to make up for missed requirements during development. The comparative: a Level 4 maturity analyst at a Level 4 company will end up doing four projects of equal magnitude in that same time period taken by the Level 1 analyst, each of which would end up costing being a mere $6,000 over expected cost on average rather than $72,000 and have negligible additional maintenance as a result of missed requirements. Therefore, if IAG assumes the company achieves the average performance for their level of maturity, the productivity and performance gain, per analyst, of improving requirements maturity to Level 4 for the Level 1 organization, would be: $199,000 savings in direct development or 20.6% increase in projects depending on what is desired $103,200 in maintenance post deployment is avoided and dropped from the maintenance budget (not included in above). 32.4% improvement in analyst productivity Over 30% improvement in time required by stakeholders to participate in requirements sessions Satisfaction rate with IT projects increases to over 80% (82.7% hit objectives of development and over 92% are considered successful) from about 50% (54% of projects hit objectives and 49% are considered successful)

Suffice it to say, for almost any reasonable investment, the benefits will more than outweigh the costs, so long as the expenditure is of sufficient magnitude to actually gain the organization Level 4 maturity. Issue #5 Everyone Recognizes Requirements are Important But Do They Take Action? The simple answer is they dont take action. Over the course of 2008 and 2009 research, and in discussing these results with tens of thousands of people, IAG concluded that while over 95% of people would say Having good requirements is important to the outcome of my project, few will change existing behavior to achieve this result. Low maturity organizations will recognize that they have poor requirements, and, that this dramatically affects project performance, yet they continue to make the same decisions that led to getting poor requirements in the first place on project after project. It falls to the savvy CIO to correct this entropy and force corrective action and the controlling of requirements-related risks. Only in the face of a visceral reaction to objective data will people take action. The issue for CIOs lies in how you set up a pilot project or other proof of concept, and how to measure and describe the benefits. Benefits must always be described in terms of whats in it for the stakeholder. For example, showing the organization that requirements change was reduced by 75% may be exciting to the IT organization, but may not lead to the needed reaction that change is needed and compliance to the new change program is warranted. If the CIO instead showed that productivity increased by 40% in all business function areas, while satisfaction with projects shifted by an order of magnitude folks are more likely to get on board. The challenge for CIOs is that this latter set of statistics is more difficult to baseline and measure. Page 21 of 33 IAG Consulting, 2009 www.iag.biz

Issue #6 There is a High Probability Your Company is a Level 2 or Lower Maturity The Business Analysis Benchmark 2009 found almost 75% of organizations at a Level 1 or Level 2 maturity in requirements definition 35% and management while relatively 30.9% 30.2% 30% few organizations had achieved 25% Level 4. Had this survey been 19.9% done 5 years ago the curve 20% would have been skewed even 15% 13% further to the left. IAG believes 10% that great strides have been 4.3% 5% taken over the last 5 years to 1.6% improve overall industry maturity 0% in requirements definition and management. However, the vast majority of companies have a big N=437 job ahead of them if they wish to Source: Business Analysis Benchmark, 2009 significantly improve.

Conclusions for the CIO: Getting on the Path to Success

Ask yourself the question: Is the business predictably successful in technology endeavors? If the answer is anything other than an unqualified yes, chances are there is an issue in requirements definition and management maturity that must be addressed and resolved. The issues involved in achieving this goal are by no means simple. You cannot simply hire a few people, or throw a few training dollars in the direction of improvement. Improving requirements definition and management maturity requires changing a broad cross-section of competencies. Combine the above challenge with the amount of effort needed to bring stakeholders on-board with the initiative, and its easy to see why many organizations fail in efforts to make improvement. The CIO is accountable for being the bridge between business and IT and is focused on more predictably delivering solutions. This results in an inherent focus on maturing requirements definition and management. For 75% of organizations, implementing this strategy yields a quantum improvement in productivity both within the analyst function, and across development and maintenance organizations. It is a simple fact that, not only do well planned and described projects run better, but few other IT investments will yield the magnitude of return of a requirements transformation program properly designed to improve requirements maturity. For the CIO, the six issues outlined in this report establish guildelines for a program of change. CIOs need to be the change catalyst in this program. If you wish to accelerate your organization down a path toward success, IAG Consulting has the following message: 1) CIOs own the path of improving requirements maturity. While the process of conducting requirements and optimizing this process is ultimately done in conjunction with the business stakeholders, the magnitude of business impact to the CIO and the organization of poor requirements practices is too great to simply shrug and say, whatever you guys want is okay by me 2) CIOs can help by communicating the steps for effective action and allocating sufficient resources to make improvement. Requirements improvement can only be made by Page 22 of 33 IAG Consulting, 2009 www.iag.biz

affecting all six requirements capability areas. To achieve gains, CIO need to communicate to the organization that the path toward success includes sustained improvement in all six areas. This means each capability area will need to be resourced and budgeted in addition to setting up the monitoring of performance change. 3) It is the CIO that is best suited to sell path to success to the organization, and show the business benefits of effective action. For too many analysts, organizational resistance and participation problems in the requirements process is so great, it is near impossible for the analyst to be successful. In lower level maturity organizations, it requires the CIO to break through organizational resistance and showcase improved development effectiveness. The consequences for the low maturity organization are simply too great to ignore or to consider inaction. CIOs must Own the Path and Sell the Path to success to their organization.

IAG is spot on with its research and results. I really enjoyed the reading. This Business Analysis Benchmark2009 report does an excellent job of both identifying the benefits and actionable activities needed to improve Requirements Management. It also quantifies them for both business and technical executive management. The report provides the type of information needed by an organization that is taking a 'lean' approach to software and application developmentshowing where to add value and where to eliminate waste. The report identifies both of the critical activities facing requirements: 'definition' and 'management'. One, without the other, is of minimal value to the organization. This report is candid and complete; it not only provides the benefits, but provides the justifications needed to pursue the noted activities. After building a strong requirements management capability, I personally have found several additional benefits besides those noted in the report: The first is that of 'auditability'. My teams have passed multiple audits by walking the auditors through our existing processes and sharing our existing documentation. We did not have to produce new artifacts. The second is that the requirements processes and artifacts are very 'transferable.' In addition to the SDLC independence, the requirements are also transferable from technology to technology as well. The third, and most important to many of my business associates, is the protection of the organization's Intellectual Property. My teams have not only been able to protect the value of the company's IP, but have added real asset value to the company by providing the quantifiable artifacts needed to procure patents, copyrights and the like. Stan Schowalter AVP, Enterprise Applications Development One of the Three Largest Global Mutual Fund/Investment Companies

Page 23 of 33

IAG Consulting, 2009

www.iag.biz

ThePathtoSuccessProjectManagers andAnalystLeadership
If you are handed the task of making transformational change to requirements definition and management, then this section of the Business Analysis Benchmark - Path to Success is for you. Conceptually the job is quite easy: 1) Baseline the organization to assess its current level of maturity in each requirements capability area (illustrated on Page 25); 2) Develop a plan of action for improving maturity within each capability area; 3) Execute the plan and measure results. Unfortunately, plans tend not to run as desired, and funding is often sub-optimal for the task at hand. Implementation teams need to balance the tendency of organizations to be impatient for immediate change, with the likelihood it will take years of thoughtful execution and continuous improvement to progress from Level 1 maturity to Level 4. Fortunately, 57% of organizations that participated in this survey made significant improvements in the area of business analysis, requirements definition and management over the last 12 to 18 months. Further, these improvements resulted in BOTH improvement in stakeholder satisfaction with development and on-time/on-budget performance of projects. This is great news the majority of companies out there are making improvements to requirements definition and management and, these changes are resulting in tangible improvements in overall performance.

The Essential Baseline

IAG has found over years of creating transformation programs that organizations which do not create an effective baseline have difficulty in sustaining their improvement program over the long term. What is clear from the Business Analysis Benchmark findings is that sustained requirements definition and management maturity improvement is not a once-anddone activity requiring a little training, some new templates, and completed in 6 months. Performance transformation activities need to be specifically targeted on the weak areas impeding maturity, and, have tangible and valuable shortterm improvement goals plotted onto a timeline of longer-term change. In the absence of this approach, organizations invest for a short period, miss key barriers in implementation planning, fail to show material performance change, lose momentum for change, backslide in maturity, and fail to realize significant benefit on much of their initial investment.
We have been actively engaged over the past 2 years in addressing the quality of our requirements and have focused on the 6 attributes that are outlined in the IAG maturity model. The survey results provide validation that we are going in the right direction but also provide some great insights into where we need to turn our focus to obtain the next level of maturity. Director, Requirement Quality Major North American Financial Institution

Many companies have a chicken-and-egg problem in setting their own baseline. If they had mature requirements definition and management practices, they would have the data and baseline for change. Since they dont have clear performance metrics and a baseline, they have difficulty building and sustaining the case for change. Its a vicious cycle. Page 24 of 33 IAG Consulting, 2009 www.iag.biz

To break this cycle, IAG developed the Requirements Management Maturity Assessment. This approach leverages the IAGs Requirements Maturity Model (illustrated to the right) and assesses the strength of the six capabilities in each of the areas of the requirements lifecycle and determine overall maturity. Combined with the data from the Business Analysis Benchmark, IAG establishes a measurable baseline founded on industry average performance at a specified level of maturity and uses this to target change practices. However a company chooses to build this baseline, it is essential. Complex companies simply dont jump from foundering to perfection in 6 months.

Metrics ->

Targeted Approach to Improvement

To highlight where organizations are having success, the table below shows the average responses to the areas of change discussed in the survey:
Area of Improvement Standardizing requirements processes and methodology Completing business analysis training or enhancing professional development programs Implementing requirements management or modeling software Implementing improvements in business analysis or requirements infrastructure (governance, RCoE, RMO) Improvement to analyst/requirements measurement or scorecarding Implementing practices and technique standards and/or guidelines for analysis, definition and management of requirements Implementing improved documentation standards Improving communication of requirements best practices to business stakeholders
Not applicable No impact Some impact Moderate impact Significant impact Huge impact

26%

3%

25%

16%

25%

5%

37%

6%

19%

19%

15%

3%

57%

5%

16%

10%

9%

3%

47%

4%

19%

15%

13%

2%

58%

7%

13%

10%

11%

2%

29%

5%

24%

19%

18%

5%

25%

5%

23%

22%

20%

5%

31%

6%

20%

20%

17%

6%

Page 25 of 33

IAG Consulting, 2009

www.iag.biz

From IAGs Analysis of the underlying data, there are a number of essential discussions for practitioners and Analyst leadership.

The Maturity Multiplier Effect

As requirements maturity increases, so too does the magnitude of the impact in every category of change. Organizations with very high levels of maturity are far more likely to report Huge Impact than organizations with poor maturity. The table above is most representative of the sample average organization (a Level 2 organization) not all organizations. Level 1 organizations are worse at implementing successfully, versus Level 4 organizations which have stellar results in each category of planned improvement. This data shows that there is a fly-wheel effect in requirements. While it is difficult to get momentum for low requirements maturity organizations, once momentum is built, it becomes easier and easier to build on this momentum by making further improvement to the requirements definition and management process. Low maturity organizations should never underestimate the effort needed to make improvement. Once this improvement translates itself into higher levels of performance, the results of future effort will also be higher.

Companies Tend to Over-Emphasize Documentation Standards

The most tried strategy is to make improvement in documentation standards. However, its a failure to visualize requirements as simply a document or artifact: Requirements are the result of a process. A great process for requirements has a chance at getting great results (the results being the documentation of requirements found and agreed-to during that process). A lousy process for requirements has a very high probability of producing lousy results. If an organization is overly focused on the documentation standard at the exclusion of other parameters it will get poor overall results. Those companies that simply introduced a new template for requirements likely met with limited success.

The Impact of Technology Implementation

A mere 15% of organizations report having excellent requirements software and only 13% would agree with the statement our business analysis tools are integrated to one another as well as integrated to the tools we use in development and testing. In this survey, this area is one of the least tried as a strategy for improvement (last year IAG found the number was closer to 20%). The results indicate a degree of dissatisfaction with existing business analysis tools. In candor, the results are likely under-representing what tools can do for business analysts and organizations. 1. For very low maturity organizations (Level 1), tools will tend to have a greater than expected impact. While this survey finding is unexpected, it is likely the result of the tool bringing its own discipline to the requirements process. Forcing analysts to use a tool enforces a set of practices and standards on the organization that can be productive. In cases where the implementing organization uses the tool as a catalyst to also improve other areas of requirements capability, the activity can be seen as highly productive and significantly impactful. In IAGs data, low maturity organizations reporting our automated business analysis tools are integrated to one another as well as integrated to the tools we use in development and testing performed far better than otherwise Page 26 of 33 IAG Consulting, 2009 www.iag.biz

expected based on their current level of maturity on the success rates of their projects. For this kind of tool, the effect is most significant in the ability of the organization to achieve the objectives of the business. 2. For low maturity organizations, overemphasis on requirements definition tools may impede performance, particularly in managing the magnitude of time overrun and magnitude of budget overrun. The implementation of a tool will pull some focus away from directing requirements definition efforts and divert these toward managing the information content within the tool. Low maturity companies reporting excellent implementations of requirements definition and modeling tools, report having far poorer project time and cost controls than would be predicted by other companies with similar maturity. 3. For higher maturity organizations (Level 4) tools are a mechanism for accelerating both the pace of requirements definition and documentation and the efficiency of communicating and managing these requirements within the organization. Here, a well defined process of requirements is institutionalized through the corporate adoption of a tools standard, and there is evidence that success rates are higher as a result. While past tools were largely limited to requirements management applications, IAG has seen a significant change in the number of tools available for business analysts for the definition, visualization, modeling and validation of business requirements. We expect to see similar improvements over the coming years and greater impact in this area.

Scorecarding What is it, and why it may be a valuable tool

Scorecarding is about establishing and managing projects and resources through a performance management system based on performance indicators aligned to the strategy of the company. The general idea is that there is often a lag between operational activity in one area and a tangible result being achieved by the company as a whole. At root, scorecarding introduces a focus on making tangible improvement on performance indicators within a specific operational groups control that are strategically aligned and known to deliver improvement to the overall organization. When analyzing the data, IAG was continuously looking for outliers companies which outperformed their peer group radically. While there are only five Level 2 companies which reported using scorecarding always or on most projects, this small subset of participants deserves special mention: they outperformed expected on time performance, on budget performance, on function performance by between 18% and 26% while raising the project success rate by an average of 16% versus their Level 2 peers. IAG makes the following conclusions from the above findings combined with the insight from the table showing both low adoption of scorecarding and relatively low perceived success rates: 1. Scorecarding can be highly effective but may be particularly difficult to implement effectively.

Page 27 of 33

IAG Consulting, 2009

www.iag.biz

2. The effect of scorecarding may be under-appreciated. While organizations that scorecard universally perform dramatically better than expected, it appears that they do not attribute significant value to the scorecard activity. Like many metrics gathering and management frameworks, scorecarding is subject to a number of weaknesses that can undermine its effectiveness.

Barriers and Accelerators Focusing on the Right Areas at the Right Time
To assemble a roadmap of tactics designed to lead an organization from Level 1 through to Level 4, IAG carefully analyzed barriers and accelerators to organizational change programs. IAG believes the needs or emphasis of organizations shift as organizations mature through each level. Hence, while every program should hit all six requirements definition and management capability areas, programs should emphasize a subset of areas of focus for making dramatic change and thereby accelerate the overall development of maturity within the organization.
IAG has presented a compelling argument on the value of improving business requirements discovery and acquisition processes. As we continue to improve on our current project outcomes and predictability their recommendations on what areas to improve during each RMM level will help us be even more successful. Ralph Maltese Project Management Office Director Large North American Consumer Products Company

Lets be clear: at any given level there is always improvement being driven across all six capabilities (the elements of people, process, and technology); but, depending on the current level of maturity, the emphasis of the program of improvement changes. This resulting roadmap is read in the following way:

1) Light Green Cells: These are areas of enhanced change acceptance. Companies that focused on change in this area at this point in maturity report experiencing better than average returns. 2) Dark Red Cells: These are points of resistance. Organizations that focused on this area at this level of maturity experienced poorer results than expected. 3) Maximum Interest: throughout the lifecycle from Level 1 to Level 4 there is a distribution curve of interest from organizations that PLAN to focus on investment in this area. This curve typically rises to the point of maximum interest then falls off.6
ChangeBA/ Requirements infrastructure (governance, centerof excellence, requirements management office)

MaturityLevel Level1 Level2 Preparingfor3 Level3 Preparingfor4 Level4

Implement standardized requirements processesand methodologies

Businessanalysis trainingand formal development program

Implement requirements managementor modeling software

Practicesand Implement techniques measurementor standardization scorecarding and/orguidelines

Change communication of requirementsbest Change practicesto Documentation business standards stakeholders

MaximumInterest MaximumInterest MaximumInterest MaximumInterest MaximumInterest

MaximumInterest

MaximumInterest

MaximumInterest

MaximumInterest

Change documentation standards is the exception to the rule it has a bi-nodal distribution and two peak points of interest in changing standards.
6

Page 28 of 33

IAG Consulting, 2009

www.iag.biz

4) White cells These are neutral points. Investment levels and return is neither better nor worse than the average for the specific level of maturity. Interpretation: Take the example of requirements software in the chart above. There is a point at which Level 1 maturity organizations benefitted strongly from implementing software followed by a period of organizational resistance. However, peak interest in software as a focal point for improvement does not occur until organizations are relatively mature in requirements definition and management and are preparing to institutionalize their requirements definition and management practices. The above chart should be read as what the organization is focusing on institutionalizing at any given level of maturity. Investment in each of the eight characteristics are important regardless of the level of maturity, however, the focus of investment will change as the maturity of the organization shifts. What is interesting in this model is that the aggregation of collective experience from more than 400 participants creates a discernable pattern: peak interest areas tend to fall at or before times of enhanced return, and are usually preceded by some period of organization resistance. From this data the following prescription for change emerges:

Optimizing the Path of Success

At Level 1: Making Sure Requirements are Performed Focus on change to documentation standards and automation around these standards. You will get resistance around standardizing techniques and practices, especially since the processes are immature. Stay away from trying to use advanced techniques like UML modeling when more simple techniques will work. As you implement, make sure a process is defined with adequate definition of roles and accountability and sufficient training put in place to make the new standards stick. Also, think in terms of the governance structure perhaps auditing requirements or peer reviewing them. These are secondary and supportive activities to getting basic guidelines in place for requirements documentation. At Level 2: Having a Process that is Defined, and Look Toward Getting This Implemented Level 2 is the combination of Level 2 and preparing for 3 in the chart above. First: the organization needs to set up some measures for success and service delivery. These are likely to be well received. Keep the newly defined process technique and technology agnostic for the short term until it can gain acceptance through your trial periods. At Level 2, the company is working to DEFINE its approach to requirements definition and management, whereas at Level 3 the company has IMPLEMENTED it. As the organization attempts to jump from Level 2 to Level 3, it needs heavy emphasis on: 1) Implementing defined requirements processes and standards. (i.e., define what analysts are expected to do) 2) Applying defined practices and/or techniques will enable the process (i.e., define how analysts will do their work reinforces the high level process) 3) Adjusting documentation standards to align with these standards (i.e., how analysts record the outcome of their efforts) 4) Communicating these standards to stakeholders (e.g., ensure stakeholders understand their role, the process, and the results of following the process)

Page 29 of 33

IAG Consulting, 2009

www.iag.biz

As the above four items are being implemented, carefully manage any supporting activities such as targeted training, improvements in governance, and communications. At Level 3: Have a Process that is Implemented, and Look Toward How to Best Institutionalize this Process The organization has IMPLEMENTED its approach to requirements definition and management, now it needs to begin INSTITUTIONALIZING this approach. This is a combination of Level 3 and Preparing for 4 above. First: Training, Training, Training. In order to go from a defined process which is inconsistently executed to a defined process that is consistently executed across a large company, the organization needs to invest heavily in resource training and development. This is an essential bridge where hundreds of analysts, project managers, stakeholders, etc. get trained on the process and are educated on what the impact of this process is expected to be. Second: as the organization progresses on this training and roll-out, it will experience a period where an emphasis on communication with stakeholders will experience very strong and positive benefits. Requirements practices must not roll out exclusively within the analyst function. To implement, the organization must focus on broad-based education. Part of this education is explaining to executives what improved requirements means for them. This is a fundamental activity that will broadly differentiate the somewhat successful from the VERY successful. While in this stage, organizations have positive returns from further tweaking their standardized requirements processes adapting master standards and gating systems to deal with more refined situations such as packaged buy versus maintenance upgrade requirements. As these process changes are adopted, templates are evolved to deal with these emergent standards. Towards the latter stages of implementation, it will become increasingly obvious that, as greater centralization is achieved, it is necessary to implement technology for the management of the requirements life cycle, business analyst resources and requirements governance. The team will also see greater benefit in analyst technologies at this point, since there is an increasingly stable environment and investment in tools and infrastructure will be seen as logical and necessary. At Level 4: Have a Process that is Institutionalized, and Look for Opportunities to Optimize and Continuously Improve this Process. Investment in training and development of business analysts remains an area of above average return as the organization progresses into Level 4. An organization can adopt more complex practices, more efficient definition and management practices and be more efficient in the rollout of these activities. While the organization is highly successful in implementing most things requirements-related, its attention will turn increasingly toward scorecarding or other measures for diagnosing current performance and seeking opportunities for continuous improvement. The resulting organization is highly standardized yet flexible in its processes, documentation standards and communication with stakeholders. So, while a Level 4 organization gets very high positive return from further investment in these areas like process (compared to lower maturity organizations) it will tend not to focus on and invest as heavily in these areas relative to other initiatives.

Conclusions for Project Managers and Analyst Leadership the Path to Success
The research highlights a path in growing and developing organizational maturity that is dynamic. Staying with dogged focus on a single strategy will repress maturity improvement, Page 30 of 33 IAG Consulting, 2009 www.iag.biz

rather than enhance the pace of change. IAG found that overall organization maturity will tend to fall to the lowest common denominator of the six capability areas. If an organization had excellent (Level 4) documentation, yet limited or no processes or techniques for executing on this documentation standard, it would typically operate at a Level 1 or 2 in terms of performance and performance would not be enhanced until such time as investment was made in the other five capability areas. As overall maturity rises, so too do the performance metrics; however be wary of committing to performance change in the face of inadequate critical mass of change. Broad-based statements of performance improvement without being able to back these with real field experience will undermine an organizations path to success.

The path to success may not be easy, but it is well defined. It is clear that very significant benefits arise to organizations which choose to successfully navigate the path. IAG applauds those organizations able to successfully make and integrate change within their organization and we work continuously to update our own understanding of how to optimize the path of improvement for clients. To this end, we provide this 2009, Business Analysis Benchmark the Path to Success.

Page 31 of 33

IAG Consulting, 2009

www.iag.biz

ContactingtheAuthor
We encourage those with questions on the survey to reach out to the author and IAG. Send us your feedback, success stories, and experiences in making improvement. Personally, Im an avid collector of data on requirements and business issues of performance change, so Im always interested in hearing about how your organization improved. If you wish to send me an email, go to www.iag.biz and select [contact us] and send an email to the address listed with IAG Business Analysis Benchmark in the subject line.

AboutTheSurveyDesign
IAG designed the survey using our Business Intelligence methodology, first and foremost establishing the business questions the company felt were essential to answer, then mapping these back to domains of knowledge and optimizing these domains into a question set that captured this knowledge efficiently. IAGs executive have decades of experience in survey research and survey research design. Given the dimentionality of the survey instruments, IAG is publishing a fraction of the data available in this Business Analysis Benchmark. Each dimension can, in fact, be BA Skill/Competency assessed against RMM Level Organization requirements results Overall Skill Level Overall Level Size Training Process Level or assessed in a # BAs Skill in each IIBA Practice Level # Resp for Bus RDM defined skill area: multidimentional Technology Level Avg Size of Projects BA Planning Org Level RM Industry analysis to improve Staff Level Enterprise Analysis Country Elicitation Deliverables Level information available. State Req Analysis Customized data segmentation and analysis can be developed for clients by contacting your IAG account executive.
Solution Assess & Validate

Technology Usage BA Infrastructure

Formal BA/RM Organization RMO/RCoE Reporting relationship of BA Quality standards ETC

Requirements Results
% proj on time % proj on budget % proj on function % proj on objectives Overrun if overun on any of above Avg time to def req. % project budget on requirements Level of quality of Bus requirements Success rate of projects

Modeling SW RM SW Integration of above Implement/plan to impl

Methods
Agile Waterfall Iterative Prototyping/ Visualization Usage now and trend

Practices
Practice guidelines UML/BPMN Use Case modeling User Stories Process modeling Data modeling Req Mgmt planning Requirements discovery practices Review practices Prioritization Change management Scorecarding Etc.

Deliverables
Clarity, accuracy, completeness Documentation standards defined

Processes
Defined process for elicitation Roles are defined Change processes defined Service levels defined Stakeholder communication processes ETC.

Page 32 of 33

IAG Consulting, 2009

www.iag.biz

AbouttheAuthor
Keith Ellis is a Vice President at IAG Consulting, a company that specializes in eliciting and managing business requirements for technology initiatives. Mr. Ellis was co-founder of the elicitation company Digital Mosaic (merged with IAG in 2007) and has extensive experience in technology research, and business analysis issues. He regularly publishes articles, white papers and other research findings in these areas and speaks to audiences around the world on best practices for business analysis. Mr. Ellis can be reached at (905) 842-0123 x228

AboutIAGConsulting
IAG Consulting is a market leading professional services firm specializing in requirements definition and management. Established in 1997, IAG has over 40 senior consultants, has worked with the majority of the F500 companies, trained over 20,000 Business Analysts, and written over 1,200 requirement specs. IAG helps US and international organizations define their business and system requirements for IT projects and organizational change initiatives. IAG provides consulting expertise, best practices, maturity assessment, and transformational programs to the IT and business community. Specifically, IAG brings measurable gains by: Reducing time needed to complete requirements Ensuring completeness in documentation and reducing change requests Issuing RFPs where vendors can bid accurately and clients get better terms Reducing costs in systems development Salvaging troubled projects Benchmarking the maturity of requirements definition & management Transforming the capabilities of organizations in requirements definition and management

Based in New Castle, Delaware, IAG is privately held with offices in the US and Canada. Contact IAG or the author at www.iag.biz or 1.800.209.3616.

Page 33 of 33

IAG Consulting, 2009

www.iag.biz