A PRACTICAL GUIDELINE FOR A SUCCESSFUL ROOT CAUSE FAILURE ANALYSIS

by David L. Ransom
Senior Research Engineer Mechanical and Materials Engineering Division Southwest Research Institute San Antonio, Texas

David L. Ransom is a Senior Research Engineer at Southwest Research Institute, in San Antonio, Texas. His professional experience over the last 10 years includes engineering and management responsibilities at Boeing, Turbocare, and Rocketdyne. His research interests include rotordynamics, structural dynamics, seals and bearings, finite element analysis, and root cause failure analysis. He has authored eight technical papers in the field of rotordynamics and thermodynamics. Mr. Ransom received his B.S. degree (Engineering Technology, 1995) and M.S. degree (Mechanical Engineering, 1997) from Texas A&M University. He is also a licensed Professional Engineer in the State of Texas.

they also represent organizational inability to successfully manage the competing interests of time, quality, and money. Therefore, in the interest of continuous improvement, it is in ones best interest to learn all one can from these failures, allowing us to avoid making the same mistake twice. The objective of this tutorial is to provide the reader with a practical guide for performing root cause failure analysis and determining the appropriate corrective/preventive action necessary to avoid the same failure in the future. The root cause failure analysis (RCFA) process begins with the collection phase, followed by the analysis phase, and concludes with the solution phase. Each of these phases is shown in Figure 1.

ABSTRACT
Root cause failure analysis is a process for identifying the true root cause of a particular failure and using that information to set a course for corrective/preventive action. From a technical standpoint, it is usually a multidisciplinary problem, typically focused on the traditional engineering fields such as chemistry, physics, materials, statics, dynamics, fluids, etc. However, it seems that too often the analysis stops with the technical aspects that are easily understood in an engineering environment, where the real root cause may exist in the human organization. In this tutorial, a practical guide to root cause failure analysis will be provided, followed by case studies to demonstrate both the technical and organizational nature of a typical root cause failure analysis.

INTRODUCTION
Despite the best efforts to avoid them, failures are still a common occurrence in every industry. Of course, there are the more obvious and well-publicized failures in the automotive, petrochemical, aerospace, and mining industries, just to name a few, but there are also many less catastrophic failures occurring at any point in time. Failures not only represent imperfection in the technical attempt to design and operate complicated systems, but
149

Figure 1. RCFA Flow Chart.

150

PROCEEDINGS OF THE THIRTY-SIXTH TURBOMACHINERY SYMPOSIUM 2007

Within collection, there are several key steps including team forming, problem definition, and of course data collection. The analysis phase is simply represented by determining the immediate, contributing, and root causes of the defined problem. The solution phase consists of determining corrective/preventive action, and then testing and implementation, which of course is the final step in RCFA. In each phase of the process, there are critical steps and simple guidelines to consider that will keep the investigation focused and practical. There are also some practical methods for organizing the investigation, depending on the size of the system under review. Finally, two turbomachinery related case studies are presented and discussed throughout the length of the tutorial to assist in demonstrating the overall process and guidelines.

questions or assist in the development of viable solutions. These expert team members do not need to be permanent members, and can be released once their contribution is complete. Their role is to support the investigation so that it is not halted for technical reasons.

CASE STUDIES
As stated in the introduction, two turbomachinery related case studies are threaded throughout the length of the tutorial, and are used to demonstrate steps and guidelines along the way. The first case study (Case 1) is from Alpha Company, who manufactures after-market parts for industrial turbomachinery. Just like every other day, an order comes in for the manufacture of a product for which drawings already exist in the engineering files. The engineering department pulls the drawings, selects the materials, and issues the work order to the shop. The shop, in turn, manufactures the hardware per the print and sends all the pieces to final inspection. During the course of final inspection, it is found that the pieces of the assembly do not fit. This is particularly troublesome for two reasons. First, this is a priority job and must ship immediately. Second, this is a product that has been manufactured in the past, so any defects in the design or manufacturing process should have been worked out in the past through the engineering change request (ECR) process. The second case study (Case 2) is from the Bravo Power Company, owner of several power generation gas turbines operating in combined cycle. After completing the spring outage (just in time for the summer heat wave), one of the gas turbines begins to exhibit a higher than normal temperature spread in the turbine exhaust. The unit is shutdown and inspected, and is found to have cracked crossfire tubes on one of the combustors. The damaged hardware is replaced, and the unit is returned to service. However, the temperature spread is still unacceptably high. A second unscheduled outage reveals that yet another combustor has cracked crossfire tubes. This time, the full set of combustors is replaced, and the unit is returned to operation without further anomalies.

Figure 2. Team Expansion for Either Technical or Organizational Limitations. The second reason to add team members is to increase the circle of influence of the team. As the investigation matures, it may become apparent that the real root cause lies outside of the current teams influence. For example, an engineering investigation may point to an issue in manufacturing. In such a case, it is important to add a team member that has the desired influence (i.e., manufacturing lead/manager), so that the investigation is not prematurely halted due to organizational boundaries. In Case 1 above, the natural team formed includes the engineering manager, two designers, one sales staff, and the quality manager. In this case, the natural team seems to have the ownership, technical knowledge, and the influence necessary for the resolution of this problem. In Case 2, the natural team is smaller, consisting only of the plant manager and the operations manager. Since neither of them has the technical knowledge necessary for the investigation, a third party consultant is also hired for the investigation. It is also possible to have an investigation team lead by an outside, independent investigator. This is most likely to occur when there is suspicion of overall organizational failure, and is typically imposed by a superior authority, such as senior management, industry regulator, etc. In such a case, the investigation may be lead by another division manager or a recognized industry expert. In this scenario, the people who would have otherwise been likely candidates for the natural team become technical team members, and will likely only be involved in the actual data collection phase. The next step in the collection phase is to define the problem. Defining the problem is a team activity, usually requiring some amount of brainstorming to come up with just the right definition. The quality of the investigation depends heavily on the quality of the problem definition. A good problem definition is short, simple, and easy to understand. In fact, if a problem statement is complicated, it merely reflects a poor understanding of the real problem. It is important that everyone on the team understands and agrees with the problem statement. The problem statement must also not be biased toward a specific solution. The consequence is the potential to either completely

STEPS TO ROOT CAUSE FAILURE ANALYSIS

As described in the introduction, RCFA is generally divided into three major phases: collection, analysis, and solution. Each of these steps is described in detail below. It is best to proceed through the phases as they are presented (i.e., one should not consider solutions until the analysis is complete), but it is not a one way path. There are plenty of reasons to possibly back up and repeat or revisit any step in the process before proceeding further. Collection Collection is used to describe all of the work necessary to prepare for the analysis phase. Naturally, the first step is to form a team that will participate in the RCFA. Team members should have ownership of the problem, and will therefore probably include engineers, technicians, operators, sales, management, etc. These team members are considered the natural team, as they have a first hand interest in the results of the RCFA. There are two main reasons why other team members might be added over the course of the investigation (Figure 2). First, it may be necessary to bring expertise into the team to help resolve key

A PRACTICAL GUIDELINE FOR A SUCCESSFUL ROOT CAUSE FAILURE ANALYSIS

151

miss the real root cause, or at a minimum, miss some important contributing causes. In Case 1, the problem statement is determined to be, Why did the product fail to pass final inspection? On the contrary, if the team jumped immediately to what they intuitively determined as the root cause, the problem statement might be, Why didnt engineering update the drawings after the first manufacture? As will be shown later, this second problem statement prevents the team from identifying an important contributing cause. The final portion of the collection phase is the actual data collection. Table 1 is a listing of the most common sets of data that should be collected for an industrial turbomachinery failure analysis. Generally speaking, there are three common types of data: physical evidence, recorded evidence, and personal testimony. Table 1. Common Data Types.

In addition to the failed components, it may be important to the investigation to provide good, undamaged components for study as well. For example, in blade failure analysis, the most effective method for evaluating blade modes is to rap test a good blade. Undamaged parts can also be important for extracting geometric information to be used in a computer simulation (finite element analysis [FEA], computational fluid dynamics [CFD], etc.). Depending on the type of failure, it may also be important to capture physical evidence such as lube oil samples, water samples, air filter samples, deposit samples, etc. Generally, it is experience that will determine what other physical evidence needs to be retained. If there is doubt, it is certainly better to retain the samples. They can always be discarded once the investigation is over. Recorded evidence is the next significant type of data to be collected. Pictures are clearly necessary for the investigation. The tendency is to take too few pictures, because at the time, it seems impossible to forget what is being witnessed. However, experience will show that there cannot be too many pictures. There are two good concepts to keep in mind when taking pictures. First, for each detail picture, include a series of pictures that start from a very large view, and then gradually (perhaps three steps) zooms into the desired level of detail (Figure 4). This technique is vital to maintaining perspective and orientation.

The most critical aspect of collecting the physical evidence is to resist the urge to clean. Although it may seem desirable to provide clean, easy to handle samples to the various technical experts for review, the odds are that valuable data will be lost in the cleaning process. Figure 3 is an example of just such an occasion. The air compressor impeller and the stationary passage are contaminated with chlorides and sulphur, leading to the stress corrosion cracking (SCC) failure of the impeller blades. Cleaning these parts before the completion of the investigation will add uncertainty to the metallurgical analysis, as well as eliminate the evidence of the corrosive source (contaminated air). There are times when it is necessary to further damage evidence just to remove it from the scene. In this case, care should be taken to not impact the actual damaged portions of the evidence.

Figure 4. Photo Sequence Captures Orientation. Another important concept is to take pictures in orthogonal views, as if they are intended to be used as manufacturing drawings. Although isometric views are handy for seeing the overall layout, they are very difficult to scale. Orthographic views can easily be used as pseudo drawings, especially if there are at least three views recorded (front, top, and side). In Figure 5, the top photograph shows an isometric view of this small, auxiliary power unit (APU) gas turbine. Although this view is helpful to see where the fuel lines are located, it is very difficult to extract line dimensions from this view. On the other hand, the lower two photos provide the proper view, allowing for dimensions to be scaled, if necessary.

Figure 3. SCC Failure Due to Chloride and Sulfur in Air Stream.

152

PROCEEDINGS OF THE THIRTY-SIXTH TURBOMACHINERY SYMPOSIUM 2007

collect personal testimony will definitely have adverse effects on the quality of the testimony. Second, it is important to stay focused only on data collection, building a consistent timeline, etc. Any premature discussion of the cause of failure will likely adversely impact the interview process. It is up to the investigating team to resolve all conflicts in the data, whether it is in the personal testimony, in the operator logs, etc. Unfortunately, due to the human influence, none of the data sources will be pristine. But, by comparing all of the data, filling in the gaps, and resolving the conflicts, a clear and consistent picture of the failure can be obtained. Analysis The analysis phase is solely focused on using the collected data to build the cause chain and determine the immediate, contributing, and root causes of the failure. The immediate cause is typically the first one in the cause chain, thus directly leading to the failure. The root cause is the last one in the cause chain, while the contributing causes are the ones in between the immediate and the root. Although the process is referred to as root cause failure analysis, it is important to identify all of the causes. There are several common structures used in the analysis phase. The why chart is a simple series of questions that guides the team to the root cause. This is generally applied to small systems, or problems that do not span over to more than a couple of systems. This method is generally useful for most rotating machinery failures. The chart begins with the first problem statement, followed by the first answer to the problem statement. The questions are answered in small steps, which help to prevent missing any contributing causes. For Case 2, the why chart starts with the event question, Why did the gas turbine register an increased T5 spread? The rest of the chart is provided in Figure 7.

Figure 5. Orthographic Views Are Easier to Scale. The other forms of recorded data (operator logs, supervisory control and data acquisition (SCADA) logs, etc.) can be critical to the complete understanding of operating conditions at the time of failure. Figure 6 is an example plot from a pump SCADA system. These data are used to assist in a failure analysis of an overheating bearing. Since log data typically are dated, they are ideal for generating a timeline of events. However, due to the costs associated with data storage and management, high resolution data are often retained only for a short period, replaced by lower resolution data for long term storage. Therefore, it is critical to capture these electronic data as soon as possible.

Figure 6. Example SCADA Plot. The other important category of data to collect is the personal testimony. Theoretically, since everyone involved is discussing the same event, all of the various stories should converge. If the information from the various personnel does not agree, it may be a sign of multiple failures. Obviously, there is significant potential for finger pointing, or at least perceived finger pointing during this phase of data collection. To minimize this perception, it is important that the interviews be conducted by a rational, coolheaded person. Sending in an irritated and irrational person to

Figure 7. Case 2 Why Chart. At the conclusion of this RCFA, the immediate cause is determined to be uneven combustion, and the root cause is

A PRACTICAL GUIDELINE FOR A SUCCESSFUL ROOT CAUSE FAILURE ANALYSIS

153

determined to be an assumption of service provider quality practices, not simply a technical flaw. This is important to note, as it drives the types of solutions that are considered in the next phase. Notice also that the contributing cause (poorly welded parts delivered by the service provider) lies outside of the current teams influence. At this point, it is recommended that the service provider be included in the team. The why chart is simple to apply, and will work for most of the turbomachinery related failures found in industry. For much larger systems, it may be more practical to use a fault tree. Fault trees are commonly used in the aerospace and nuclear power industries, since these systems are typically very complicated and much more difficult to investigate. Basically, the fault tree method requires that the team start with the fault, and then work backward to identify all possible causes of this fault. For large, difficult to understand systems, this provides a map for dividing up the investigation. As the investigation proceeds, teams gradually rule out each potential cause, and mark it off on the tree. By the end, there should remain a short list of potential root causes. Figure 8 is an example of a fault tree for Case 2. In this case, each of the events is preceded by either an AND or OR logic statement. For example, Uneven Combustion can be caused by either Uneven Fuel Delivery OR Uneven Air Delivery. On the other hand, the Poor Weld Quality is the result of both Improper Welding Technique AND Inadequate Quality System. Although this fault tree is incomplete, it demonstrates the level of detail required when using this approach. Each branch must be expanded and evaluated by the team. By closing out each branch of the tree, the actual failure cause chain becomes apparent.

Since this method is more complex and relies on a system of symbols (similar to a process flow chart), there are many commercial software packages available to assist in the process. Due to the size and complexity of the systems for which fault trees are used, the investigation is usually managed by an experienced investigator. Another popular structure for the analysis phase is the cause and effect diagram (also known as a fishbone diagram). Where fault trees are useful for complex systems, cause and effect diagrams are useful for incorporating cross-functional influences. As seen in Figure 9, the head of the fish is the problem to be investigated, and each of the main branches (bones) represents a specific functional area. To complete the fishbone diagram, the investigation team continues to list all of the possible connections each functional area might have with the failure. This format allows the team to see the overall picture and begin to focus the investigation as each of the functional branches is evaluated. In this case, it is clear from the beginning that the failure is contained within the engineering function, eliminating the need to further investigate the other branches.

Figure 9. Case 1 Cause and Effect (Fishbone) Diagram. There are some other important key points to remember during the analysis phase. It is helpful to keep these handy as the investigation proceeds, so that each team member is reminded of these guidelines.

Follow the dataThe most difficult aspect of the analysis phase

is avoiding preconceived notions regarding the root cause. It is up to the team members to protect each other from this trap. The investigation team must stick to the data and exclude gut feel from the investigation.

Consider

both technical and organizational causesFinding the technical answer is often difficult, but the investigation should not stop there. Organizational influences can be just as significant and must also be included in the investigation.

Concentrate on analysisSave the problem solving for the next Really

phase. The key at this point is to identify the immediate, contributing, and root causes. operator or maintenance error?It is rarely actually an operator or maintenance craftsmen error. We all work in organizations with norms, procedures, and external pressures. What appears to be operator error is most likely a broken process, missing check, or unclear expectations. The analysis phase is complete once the immediate, contributing, and root causes are identified. Keep in mind that the root cause is dependent on the reach of the team. If the last contributing cause exists at a boundary that cannot be crossed (by either adding technical or organizational influence), then it is effectively the root

Figure 8. Case 2 Fault Tree.

154

PROCEEDINGS OF THE THIRTY-SIXTH TURBOMACHINERY SYMPOSIUM 2007

cause. In other words, this is where the solution phase should focus. It is only of academic value to identify a root cause over which the team has no influence. For example, consider again the cause chain for Case 2 (Figure 7). The contributing cause is identified as GT service provider delivered poorly welded combustors. Clearly, this cause lies outside of the current teams influence. If the gas turbine (GT) owner hopes to eliminate this cause, they must convince the service provider to join the investigation. Solution Certainly, failure prevention starts with good design, manufacture, and operation practices, providing protection against the already considered failure modes. For complicated systems, cause and effect diagrams and fault trees are used to study potential failure modes, allowing them to be incorporated into the design phase. However, it is just as important to learn from experience, preventing the recurrence of the same mistake, and this is where the solution phase of root cause failure analysis fits in. The fundamental objective of the solution phase is to break the cause chain. Of course, this means that the quality of the solution depends heavily on the quality of the cause chain developed in the analysis phase. To emphasize this point, consider again the problem statement for Case 1. As shown in Figure 10, the correct initial problem statement is Why did the product fail to pass final inspection? Suppose the team instead started with Why werent the drawings corrected after the first manufacture? This statement is located in the lower half of the why chart, below a critical split. Such a poorly posed problem statement would prevent the investigating team from identifying the real root cause, which is the assumption that all previously manufactured products have updated drawings.

Therefore, using the well-developed cause chain as a starting point, the solution phase begins by identifying all the possible ways to break the chain. These solutions are referred to generically as corrective/preventive actions. Each corrective/preventive action must be evaluated for effectiveness (i.e., does it reduce the likelihood of the event recurring to an acceptable level) and realism (i.e., is it reasonable to implement with respect to cost, time, organizational influence, technical requirements, etc.). Table 2 is a list of corrective/preventive actions for both of the case studies, along with an assessment of effectiveness and realism for each potential action. Table 2. Corrective/Preventive Actions.

Figure 10. Case 1 Why Chart. Another important feature of the cause chain is that since most failures are the result of both root and contributing causes, there are usually multiple areas that can be addressed. This is important to recognize in the solution phase, as it helps to open up the number of possible solutions to the original problem. It is also possible that preventing some of the contributing causes can also lead to improved reliability in other areas not presently considered. For example, in Case 1, one of the possible contributing causes is Shop fixed the problem without engineering involvement. Clearly, if this occurs, there is no feedback through the ECR process, and the design cannot possibly be corrected. Although this did not occur in this particular case, it is identified as a potential failure mode, and preventive action is taken to minimize the possibility of this failure.

For Case 1, perhaps the most obvious, or at least the initial response, is to thoroughly check each drawing before it is issued to the shop. But, when this possible solution is evaluated for effectiveness and realism, it becomes clear that it is a poor option. It is later determined that the best corrective/preventive action plan is to eliminate the ECR back log (and discontinue practice of adding to the back log), and begin to review previous job files for each repeat order, thus significantly improving the probability of eliminating repeat mistakes. For Case 2, although the root cause is an assumption of service provider quality, it is also clear from the cause chain that there are two reasonable approaches to the corrective/preventive action plan. First, it might make sense to begin the practice of inspecting combustors for weld quality before they are installed. This provides the GT owner with the direct control of the combustor weld quality. However, it is also in the GT owners interest to get involved in the service providers quality system, making it possible to eliminate the extra level of quality inspection. Obviously, this approach requires teaming with the service provider, and therefore may not be relied upon for the immediate correction that is necessary before the next outage.

SUMMARY
Failures in human-made systems reflect both technical and organizational flaws. Although it is unreasonable to expect perfect performance with perfect reliability from these systems, it is just as unreasonable to allow the same failure to occur multiple times. Therefore, the objective of this tutorial is to provide the reader with a practical guide for performing root cause failure analysis and determining the appropriate corrective/preventive action necessary to avoid the same failure in the future.

A PRACTICAL GUIDELINE FOR A SUCCESSFUL ROOT CAUSE FAILURE ANALYSIS

155

RCFA starts with the collection phase, consisting of team forming, problem definition, and of course data collection. Next is the analysis phase, determining the immediate, contributing, and root causes of the defined problem. Finally, the solution phase consists of determining the appropriate corrective/preventive action plan that will effectively break the cause chain. In each phase of the process, there are critical steps and simple guidelines to consider that will keep the investigation focused and practical. These, of course, are the key characteristics of a successful root cause failure analysis.

BIBLIOGRAPHY
AlliedSignal Aerospace FM&T, 1997, Root Cause Analysis and Corrective/Preventive Action Workshop. Vesely, W. E., Goldberg, F. F., Roberts, N. H., and Haasl, D. F., 1981, Fault Tree Handbook, NUREG-0492, U.S. Nuclear Regulatory Commission, Washington, D.C.