FAILURE REPORTING, ANALYSIS

AND CORRECTIVE ACTION SYSTEM

SEMATECH and the SEMATECH logo are registered trademarks of SEMATECH, Inc.
WorkStream is a registered trademark of Consilium, Inc.
 FAILURE REPORTING,
ANALYSIS, AND
CORRECTIVE ACTION SYSTEM
ii
 CONTENTS

Page
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

FAILURE REPORTING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Collecting the Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Reporting Equipment Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Reporting Software Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Logging Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Using FRACAS Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

ANALYSIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Failure Analysis Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Failure Review Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Root Cause Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Failed Parts Procurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

CORRECTIVE ACTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

FRACAS DATABASE MANAGEMENT SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Configuration Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Events Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Problems Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Report Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

GLOSSARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

FRACAS Functions and Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . inside back cover

iii
iv
 ACKNOWLEDGMENTS

In putting together this document, there were many obstacles that kept getting in the way of
completing it. Without the support of my family, Sonia, Jason, and Christopher, this would
never have been accomplished.

Mario Villacourt
External Total Quality and Reliability
SEMATECH

I would like to thank my wife, Monique I. Govil, for her support and encouragement in
completing this work. Also, I want to thank SVGL for supporting my participation with this
effort.

Pradeep Govil
Reliability Engineering
Silicon Valley Group Lithography Systems

v
vi
 EXECUTIVE SUMMARY

Failure Reporting, Analysis, and Corrective Action System (FRA-

CAS) is a closed-loop feedback path in which the user and the
supplier work together to collect, record, and analyze failures of
both hardware and software data sets. The user captures predeter-
mined types of data about all problems with a particular tool or
software and submits the data to that supplier. A Failure Review
Board (FRB) at the supplier site analyzes the failures, considering
such factors as time, money, and engineering personnel. The
resulting analysis identifies corrective actions that should be
implemented and verified to prevent failures from recurring. A
simplified version of this process is depicted in Figure 1.

Reliable
Equipment

Test &
Engineering In-house Test
Change & Inspections
Control

Design & Field

Production Operations

Corrective Failure
Action Reporting

Analysis

Figure 1 FRACAS Process

Page 1
Failure Reporting, Analysis, and Corrective Action System
Since factors vary among installation sites, equipment users must
work closely with each of their suppliers to ensure that proper data
is being collected, that the data is being provided to the correct
supplier, and that the resulting solutions are feasible.

Unlike other reliability activities, FRACAS promotes reliability

improvement throughout the life cycle of the equipment. The
method can be used during in-house (laboratory) tests, field (alpha
or beta site) tests, and production/operations to determine where
problems are concentrated within the design of the equipment.
According to the military,

Corrective action options and flexibility are greatest during

design evolution, when even major design changes can be
considered to eliminate or significantly reduce susceptibility
to known failure causes. These options and flexibility become
more limited and expensive to implement as a design
becomes firm. The earlier the failure cause is identified and
positive corrective action implemented, the sooner both the
producer and user realize the benefits of reduced failure
occurrences in the factory and in the field. Early implementa-
tion of corrective action also has the advantage of providing
visibility of the adequacy of the corrective action in the event
more effort is required. 1

In addition to its military use, FRACAS has been applied extensive-

ly in the aerospace, automotive, and telecommunications industries.
Elements of FRACAS can be found throughout all industries, but
this limited use usually means that the only data received by the
manufacturer are complaints logged with the field service or
customer service organizations.

FRACAS can provide control and management visibility for

improving the reliability and maintainability of semiconductor
manufacturing equipment hardware and software. Timely
disciplined use of failure and maintenance data can help generate
and implement effective corrective actions to prevent failures from
recurring and to simplify or reduce maintenance tasks.

1. Quoted from MIL-STD-2155(AS), Failure Reporting, Analysis and Corrective Action System.

Page 2
Failure Reporting, Analysis, and Corrective Action System
 FRACAS objectives include the following:

Providing engineering data for corrective action

Assessing historical reliability performance, such as mean time

between failures (MTBF), mean time to repair (MTTR), avail-
ability, preventive maintenance, etc.

Developing patterns for deficiencies

Providing data for statistical analysis

Also, FRACAS information can help measure contractual perfor-

mance to better determine warranty information.

FRACAS provides a complete reporting, analysis, and corrective

action process that fulfills ISO 9000 requirements. More and more
companies are requiring their suppliers to meet ISO 9000, for
example, the SEMATECH Standardized Supplier Quality Assess-
ment (SSQA) and the Motorola Quality Systems Review (QSR).

FAILURE REPORTING
In-house Test
& Inspections
All events (failures) that occur during inspections and tests should
be reported through an established procedure that includes
Field
Operations collecting and recording corrective maintenance information and
times. The data included in these reports should be verified and
then the data should be submitted on simple, easy-to-use forms that
Failure are tailored to the respective equipment or software.
Reporting

Page 3
Failure Reporting, Analysis, and Corrective Action System
Collecting the Data
Many problems go unnoticed because insufficient information was
provided. The FRB must know if, for example, someone was able
to duplicate the problem being reported. There are three common
causes for missing essential data:

Inspection or testing began before a procedure was in place to

report problems.

The reporting form was difficult to use.

The person who filled out the form had not been trained.

Operators and maintenance personnel are usually the first to

identify problems and, therefore, they should be trained to properly
capture all of the information needed for an event report. The
contents of this report are more fully described in the next section.

However data is collected, one constant should be emphasized:

consolidate all the data into a central data logging system. When
field failures occur, two types of data collection are usually present:
(1) the users system that includes equipment utilization and
process-related information, and (2) field service reports (FSRs) that
include parts replacement information. The supplier has great
difficulty addressing problems when FSRs are the only source of
data. It is helpful if the user provides the supplier with detailed
logs for problem verification. The supplier should correlate both
types to help determine priorities for problems to be resolved.

The supplier should be included in the formation of the users data

collection system while the product is being installed. Ideally, the
equipment should log all information automatically and then
download the data to the suppliers data collection system. This
would eliminate the need for paper forms and also the confusion
caused by duplicate data sets.

Page 4
Failure Reporting, Analysis, and Corrective Action System
Reporting Equipment Failures
Collecting and sharing appropriate data through event reports are
essential components of an effective FRACAS process, both for the
supplier and for the user. There are common elements in every
report (when the event occurred, what item failed, etc.) that the
user and the supplier both use to analyze failures. Other crucial
information includes the duration of the failure, the time it took to
repair, and the type of metric used (time or wafer cycles).

To simplify the effort of collecting data and to minimize any

duplication of effort, use the following guidelines:

Capture data at the time of the failure.

Capture utilization data through the users factory tracking

system (for example, WorkStream).

Avoid duplicating data collection between the suppliers and

the users systems.

Make failure and utilization data available to both the supplier

and the user in a standardized format.

Exchange failure and utilization data between the supplier and

user frequently.

Link event reports to the corresponding utilization data in the

factory tracking system through an identifying event number
that is captured on the equipment.

Automate both the users and the suppliers systems to

maximize efficiency, minimize paper tracking, and avoid dual
reporting.

In addition, make full use of reporting tools that already exist (to
which little or no modification may be necessary), such as

In-house test reports

Field service reports

Page 5
Failure Reporting, Analysis, and Corrective Action System

Customer activity logs

Customer equipment history

Any other pertinent information you may need may be determined

by asking the design engineers what information they need for root
cause analysis. Data log files (history kept by the software) should
accompany the failure report to provide software engineers with
information about events that preceded the malfunction.

As a result of the data collected, minimum reliability performance

metrics (such as MTBF, MTTR, and uptime percentage) could be
determined for each system.

Every piece of data helps. In addition to these common elements,

there are specific types of data needed only by the user or only by
the supplier.

The supplier should acquire detailed failure information from the

event report that should become an integral part of the equipment
log file. The details help determine the state of operation before and
after the failure, which is necessary to address the root cause of
equipment failures efficiently and effectively.

Table 1 shows the types of detail that the supplier needs to see in
the event report (in-house or field). Figure 2 is an example of an
event report form. The format of the form is important only to
simplify the task of the data recorder. You may want to computer-
ize your data entry forms to expedite the process and also minimize
failure description errors.

Page 6
Failure Reporting, Analysis, and Corrective Action System
 TABLE 1 EVENT REPORT FIELDS

Event Report
Fields Example Description

Serial Number 001 System-unique identifier that

indicates who the customer is
and where the system is located

Date and Time March 12, 1994 at

of Event 4:00 PM

Duration of 01:21 Usually in hours and minutes

the Event
(Downtime)

Failed Item Handler-Stocker-Robot- Describes function and location

Description Motor of failed item from top to bottom;
that is, field to lowest replaceable
unit

Reliability/ AH-STK-ROT-MOT-021 Known problem categorized by

Fault Code reliability code and failure mode
of Known number
Problems

Life Cycle Field Operations See equipment life cycle in the

Phase of the Glossary
System

Downtime unscheduled Usually shown as either

Category unscheduled or scheduled
(Maintenance (preventive maintenance)
Action)

Part Number 244-PC Part number of replaced item (if

any)

Operators M. Lolita Person reporting problem

Name

Service S. Spade Person performing repair action

Engineers
Name

Event Report FSR-002 Suppliers report number

Identification
Number

Event What happened? Description of all conditions

(Problem) prior to failure and how observer
Description thinks it failed

Repair Action What did you do to Description of any repair or

Description repair item? maintenance attempts

Page 7
Failure Reporting, Analysis, and Corrective Action System
 EVENT RECORD
Serial Number: 9003 Reliability Code: DMD-XS-HRN-M3S-002

Date: 09/01/93 Down Time Category: Unscheduled

Time: 09:00: Life Cycle Phase: OPERATION

Duration: 04:30: Relevant: Y

System: DMD PM: N

Subsystem: XS Part Number: 650-000029

Assembly: HRN Operator: FRANK BELL

Subassembly: M3S FSE Name:

Sub-subassembly: FSE Number:

Problem: PREVIOUS ATTEMPT TO RESOLVE TRANSPORT LOAD FAILURE DID

NOT WORK. REQUIRES SWITCH REPLACEMENT.
Repair REPLACED SWITCH, CURED LENS 2, ALIGNED COLUMN, CHECKED
Action: JET ALIGN, MADE TEST MILLS AND DEPOS.

Figure 2 Example Event Report

The user analyzes this data to evaluate management of equipment

and resources and to determine its effectiveness in meeting internal
commitments. Internally, user data may be handled differently
than described in SEMI E10-92, but as a minimum for sharing data
with the supplier, the major states in Figure 3 should be followed.

NON-SCHEDULED

UNSCHEDULED
DOWNTIME
equipment
downtime SCHEDULED total time
DOWNTIME (168 hrs/wk)

ENGINEERING operations
time
equipment STANDBY
uptime

PRODUCTIVE

Figure 3 Equipment States Stack Chart

Page 8
Failure Reporting, Analysis, and Corrective Action System
Reporting Software Problems
When reporting software problems, too much detail can be
counterproductive. An error in software does not necessarily mean
that there will be a problem at the system (equipment) level. To
quote Sam Keene, 1992 IEEE Reliability Society president,

If Im driving on a two-lane road (same direction) and

switch lanes without using the signal indicator and
continue to drive, Ive committed an error. But if I do the
same when another car is on the other side and I happen
to crash into it, then I now have a problem.

Your existing software error reporting process should become an

integral part of your overall FRACAS. (Rather than reiterating the
software reliability data collection process, a practical approach can
be found in A.M. Neufelders work.2)

FRACAS helps you focus on those errors that the customer may
experience as problems. One focusing mechanism is to track the
reason for a corrective action. This data can be collected by review-
ing the repair action notes for each problem in the problem report.
Engineering and management can categorize these reasons to
determine the types of errors that occur most often and address
them by improving procedures that most directly cause a particular
type of error. The process of analyzing this data is continuous.

Among the reasons for corrective action are the following:

Unclear requirements

Misinterpreted requirements

Changing requirements

Documented design not coded

Bad nesting

Missing code

2. Refer to Chapter 7, Software Reliability Data Collection, of Ensuring Software Reliability, by

A. M. Neufelder (Marcel Dekker, Inc., New York, 1993).

Page 9
Failure Reporting, Analysis, and Corrective Action System

Excess code

Previous maintenance

Bad error handling

Misuse of variables

Conflicting parameters

Software or hardware environment change

Specification error

Problem with third-party software

Documented requirements not designed

Software environment error (for example, an error in the

compiler)

Reasons should also be ranked in terms of the criticality or severity

of the error. This information helps management predict those
errors that will cause downtime (as opposed to all errors, including
those that may not cause downtime).

Severity rankings are:

1. Catastrophic

2. Severe

3. Moderate (has work-around)

4. Negligible

5. All others (for example, caused by a misunderstood new feature

or unread documentation)

Also, tracking the modules or procedures modified for each

corrective action helps schedule pre-release regression testing on
these changes, which results in more efficient test procedures and
more effective test results.

Page 10
Failure Reporting, Analysis, and Corrective Action System
 Logging Data
Whether software or equipment errors are being reported, once the
supplier receives the reports, all data should be consolidated into
one file. The suppliers reliability/quality organization should
oversee the data logging.

A FRACAS database is recommended for this part of the process.

This is not a product-specific database but rather a database for
tracking all products offered by a company. Figure 4 shows how
the status of problems is kept updated in the database.

PROBLEMS RECORD
Reliability Code Title/Description
ATF-STK-PRI-RT-XAX-001 LIMIT FLA SENSORS FAIL
System: ATF Initial Status: I
Subsystem: STK Fault Category: Facilities Elec
Assembly: PRI Error Codes: errcode 2
Subassembly: RT Assignee: Eric Christie
Sub-subassembly: XAX
DATES
ROOT CAUSE SOLUTION (RCS)
Insufficient Info: 11/04/93
Original Plan: / /
Sufficient Info: / /
Current Plan: / /
Defined: / /
Complete: / /
Contained: / /
Retired: / /
RCS Documentation:

Comments:Not enough information at this time. However, supplier is

investigating vendor part #po09347-90.
STATISTICS
Current Status: I First Reported: 11/04/93
Number of Fails: 2 Last Reported: 11/11/93
Number of Assists: 0
Number of Others: 0
Total Events: 0
Total Downtime Hours: 2.72

Figure 4 Problems Record

Page 11
Failure Reporting, Analysis, and Corrective Action System
 Using FRACAS Reports
The FRACAS database management system (DBMS) can display
data in different ways. This section describes some of the report
types you may find useful. A complete failure summary and other
associated reports that this DBMS can provide are described in
more detail beginning on page 27.

The graphical reports provide a quick snapshot of how the equip-

ment or software is performing at any given point. Figure 5 shows
the percentage rejection rate on a monthly basis as actual-versus-
target rates. Figure 6 depicts the number of failure modes or
problems on a weekly basis. Figure 7 points out the number of
events being reported weekly by life cycle phase. Figure 8 provides
a one-page status outline for tracking by the FRB. Whichever
report you choose, it should be tailored to provide summaries and
special reports for both management and engineering personnel.

Factory Test Tracking

Initial Test Reject Rate




 D Monthly Rate



+ 3 Month Rate






Reject %












Target

 Rate (8%)



Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan 86
Volume 566 97 679 598 22 123 126 333 98 98 145 88

Production HP3060 Test

Figure 5 ExamplePercentage Rejection Rate, Actual Versus Target

Page 12
Failure Reporting, Analysis, and Corrective Action System
 By Reliability Code
From 08/30/93 to 10/03/93
8

Events
4

0
1 2 3 4 5
Week
DMD-OP-IBIGA-SRC-002
DMD-OP-IBIGA-SRC-003 DMD-VS-HC-001

Figure 6 ExampleNumber of Events Weekly

Events by Life Cycle Phase by Week

From 09/27/93 to 10/17/93
4

0
1 2 3
Phase II Phase III

Figure 7 ExampleNumber of Events Weekly by Life Cycle Phase

Page 13
Failure Reporting, Analysis, and Corrective Action System
 FAILURE/REPAIR REPORT
NO.
V H/W V S/W V SE V TE V OTHER
1. PROJECT 2. FAILURE DATE 3. FAILURE GMT 4. LOG NO. 5. REPORTING LOCATION V OTHER
DAY ___ HR ___ MIN ___ V SUPPLIER
6. SUBSYSTEM 7. 1ST TIER 8. 2ND TIER 9. 3RD TIER 10. 4TH TIER 11. DETECTION AREA
H/W, S/W H/W, S/W H/W, S/W H/W, S/W
A REFERENCE V BENCH V ACCEPT
DESIGNATIONS
V FAB/ASSY V QUAL
I ORIGINATOR

B NOMENCLATURE
V INTEG V SYSTEM
C SERIAL NUMBER
V FIELD V OTHER
D OPERATING TIME
CYCLES

12. FAILURE DESCRIPTION

13. ENVIRONMENT V ACOUSTIC V VIE V SHOCK V HUMID 14. ORIGINATOR 15. DATE 16. COGNIZANT ENGINEER
V AMB V TEMP V EMC/EMI V OTHER

17. VERIFICATION AND ANALYSIS

18. ASSOCIATED
II VERIFICATION

DOCUMENTATION

18. ITEM A. ITEM NAME B. NUMBER C. SERIAL NO. D. CIRCUIT DESIGNATION E. MANUFACTURER G. DEFECT
DATA

70. FAILURE CAUSE V PART FAILURE V ADJUSTMENT V S/W V SE FAILURE

V DESIGN V WORKMANSHIP
V LRU FAILURE V OPERATING TIME V TEST ERROR V DOCUMENTATION
V DAMAGE V MFG PROCESS
(MISHANDLING) 21. SIGNATURE 22. DATE
V EQMT FAILURE
V OTHER

23. ACCUMULATED A. ON LOCATION B. DEPOT SHOP C. MANUFACTURING SHOP

MAINTENANCE
a NAME
III REPAIR

b DATE

c HRS/MINS
d LOCATION
e WORK
PERFORMED

24. CORRECTIVE ACTION TAKEN

IV CORRECTIVE ACTION

25. DISPOSITION OF SUBSYSTEM OR ASSEMBLY V NO ACTION

V REDESIGNED V REWORKED V READJUSTED V SCRAPPED V RETESTED V OTHER

26. EFFECTIVITY V FAR NO. 27. SAFETY

V THIS UNIT V ALL UNITS V OTHER V RFC NO.

28. SIGNATURE COG ENGR 29. SEC 30. DATE 31. SIGNATURE COG SEC ENGR 32. DATE 33. SUBSYSTEM
RATING

34. SYSTEM ENGR 35. DATE 36. PROJECT RELIABILITY ASSURANCE 37. DATE 38. RISK ASSESSMENT

COPY DISTRIBUTION WHITE PFA CENTER, CANARY PROJECT OFFICE, PINK ACTION COPY
GOLDENROD ACCOMPANY FAILED ITEM

Figure 8 ExampleFailure/Repair Report

Page 14
Failure Reporting, Analysis, and Corrective Action System
 Failure ANALYSIS
Reporting

Failure analysis is the process that determines the root cause of the
failure. Each failure should be verified and then analyzed to the
Analysis extent that you can identify the cause of the failure and any
contributing factors. The methods used can range from a simple
investigation of the circumstances surrounding the failure to a
sophisticated laboratory analysis of all failed parts.

Failure Analysis Process

Failure analysis begins once an event report is written and sent to
the FRB. It ends when you sufficiently understand the root cause so
you can develop logically derived corrective actions.

It is important to clearly communicate the intent and structure of

the failure analysis process to all appropriate organizations. These
organizations should review and approve the process to confirm
their cooperation. During and after the analysis, the problem
owner, associated FRB member, and reliability coordinator must
ensure that the database is maintained with the current applicable
information.

The analysis process is:

1. Review, in detail, the field service reports.

2. Capture historical data from the database of any related or

similar failures.

3. Assign owners for action items.

4. Do a root cause analysis (RCA).

5. Develop corrective actions.

6. Obtain the failed items for RCA (as needed).

7. Write a problem analysis report (PAR) and, if needed, a material

disposition report (MDR).
Page 15
Failure Reporting, Analysis, and Corrective Action System
Developing proper forms for tracking the failed parts and reporting
the problem analysis results is essential. Figures 9 and 10 are
examples of these types of forms.

MATERIAL DISPOSITION REPORT

S.NO. REWORK ORDER NO

FAILURE DATE J37511

z RETURNED PART DESCRIPTION

PART NUMBER
QTY RETURNED
PART DESCRIPTION
FSR NUMBER

z REASON FOR RETURN

V DEFECTIVE V ENG CHANGE
V OVERISSUE V OVERDRAWN
V OTHER

z USED ON
EQUIPMENT S.NO.
ASSEMBLY NO.
SITE NUMBER

z MATERIAL DISPOSITION INSTRUCTIONS DATE

V RETURN TO STOCK
V SEND TO FRB FOR FAILURE ANALYSIS
V SEND FOR VENDOR FAILURE ANALYSIS
V SCRAP
V RETURN TO VENDOR
V SEND FOR REPAIR

z RELIABILITY INFORMATION DATE

TASK NUMBER
FRB MEMBER

ORIGINATOR DATE
DEFECT DESCRIPTION (FILLED BY THE ORIGINATOR)
z DISTRIBUTION

V RELIABILITY
V MDC
V FS
V QUALITY

Figure 9 MDR Forms

Page 16
Failure Reporting, Analysis, and Corrective Action System
 PROBLEM ANALYSIS REPORT
PROBLEM NUMBER: ___ ___ ___ ___ ___ ___ ___

ERROR CODE: PROBLEM TITLE:

FRB MEMBER: ASSIGNEE:

TYPE OF PROBLEM: F/A j O j CURRENT FIX DATE/SW REV.

COMMON: j UNIQUE: j CATEGORY: WORK: j PROC: j DESIGN: j

MODULE CODE: ____ ____

INSUFFICIENT INFO: j DATE: SUFFICIENT INFO: j DATE:

COMMENTS:

j DEFINITION OF PROBLEM:

DEFINED: j DATE

j CONTAINMENT:

CONTAINED: j DATE

j ROOT CAUSE:

j CORRECTIVE ACTION:

VERIFICATION: RETIRED/CLOSED: j DATE:

DOCUMENTATION: j ECN: j FRI j OTHER:

REVISION: A: j B: j C: j D: j DATE:

ENTERED IN DB:

*** IF MORE SPACE IS NEEDED, PLEASE ATTACH SEPARATE PIECE OF PAPER ***

Figure 10 PAR Form

Page 17
Failure Reporting, Analysis, and Corrective Action System
Failure Review Board
The FRB reviews failure trends, facilitates and manages the failure
analysis, and participates in developing and implementing the
resulting corrective actions. To do these jobs properly, the FRB
must be empowered with the authority to require investigations,
analyses, and corrective actions by other organizations. The FRB
has much in common with the techniques of quality circles; they are
self-managed teams directed to improve methods under their
control (see Figure 11).

FRB meets
regularly

Review in-house
& field reports

Reliability FRB establishes

coordinator preliminary
brings reports assignments

Yes Does FRB

require
additional
data?

No

FRB confirms
problem
assignments within
24 hours

Figure 11 FRB Process

Page 18
Failure Reporting, Analysis, and Corrective Action System
 A

Yes
Sufficient
Enter into information?
DBMS & write
PARs

No

After suitable period FRB assigns

if additional events problem owner
have not recurred,
retire problem

Design Software Mfg.

problems problems issues

Operator/
field Others
services

Establish
containment, RCA
Update DBMS plan/schedule

FRB reviews &

Distribute ensures reliable
updates to solution
FRB

RCA complete
Begin ECN Process

Figure 11, continued FRB Process

Page 19
Failure Reporting, Analysis, and Corrective Action System
The makeup of the FRB and the scope of authority for each member
should be identified in the FRACAS procedures. The FRB is
typically most effective when it is staffed corporate-wide, with all
functional and operational disciplines within the supplier organiza-
tion participating. The user may be represented also. Members
should be chosen by function or activity to let the composition
remain dynamic enough to accommodate personnel changes within
specific activities.

The FRB composition may be product specific, if desired. For

example, a suppliers different products may be complex and
therefore require expert knowledge. In this scenario, though,
overall priorities must be managed carefully, since the FRB may
require the same expert resources for root cause analysis.

Generally, the following organizations are represented on the FRB:

Reliability Quality Assurance

Field Service Manufacturing
Statistical Methods Marketing
Systems Engineering Test
Design Engineering*

The FRB should be headed by the reliability manager. One of the

main functions of this reliability manager is to establish an effective
FRACAS organization. The reliability manager must establish
procedures, facilitate periodic reviews, and coach the FRB mem-
bers. Other responsibilities include:

Assign action items with ownership of problem/solution

(who/what/when)

Allocate problems to appropriate functional department for

corrective action

* Hardware, software, process, and/or materials design, depending on the type of system
being analyzed.

Page 20
Failure Reporting, Analysis, and Corrective Action System

Conduct trend analysis and inform management on the types
and frequency of observed failures

Keep track of failure modes and criticality

Issue periodic reports regarding product performance (for

example, MTBF, MTTR, productivity analysis) and areas
needing improvement

Ensure problem resolution

Continually review the FRB process and customize it to fit

specific product applications

The FRB operates most efficiently with at least two recommended

support persons. A reliability coordinator can procure failed items
for root cause analysis, ensure that the event reports submitted to
the FRB contain adequate information, stay in contact with internal
and external service organizations to ensure clarity on service
reports, and track and facilitate RCA for corrective action imple-
mentation.

A database support person should be responsible for providing

periodic reports and for maintaining the database (importing/ex-
porting data, backing up files, archiving old records). This will
ensure that data available from RCA and corrective action is kept
current in the database.

Responsibilities of the FRB members include the following:

Prepare a plan of action with schedule

Review and analyze failure reports from in-house tests and field
service reports

Identify failure modes to the root cause

Page 21
Failure Reporting, Analysis, and Corrective Action System
Furthermore, each FRB member should

Have the authority to speak for and accept corrective action

responsibility for an organization

Have thorough knowledge of front-end systems

Have the authority to ensure proper problem resolution

Have the authority to implement RCA

Actively participate

The FRB should meet on a regular basis to review the reported

problems. The frequency of these meetings will depend on the
number of problems being addressed or on the volume of daily
event reports. Problem-solving skills and effective meeting
techniques can ensure that meetings are productive. If you need
assistance in these areas, SEMATECH offers short courses3 that are
affordable and can be customized for your application.

The top problems, based on pareto analyses (statistical methods)

derived from the database, should be used to assign priorities and
to allocate resources.

The reporting relationship of the FRB with other FRACAS functions

is shown in Figure 12.

3. Contact the SEMATECH Organization Learning and Performance Technology Depart-

ment, 2706 Montopolis Drive, Austin, TX 78741, (512) 3567500

Page 22
Failure Reporting, Analysis, and Corrective Action System
 Reliability
Manager

Failure
Reliability
Review
Coordinator
Board

Problem Database
Owner Support

Problem In-house
Owner Test
Inspections

Problem Field
Owner Operations

Figure 12 FRACAS Functions Responsibilities

Root Cause Analysis

In failure analysis, reported failures are evaluated and analyzed to
determine the root cause. In RCA, the causes themselves are
analyzed and the results and conclusions are documented. Any
investigative or analytical method can be used, including the
following:

Brainstorming

Histogram

Flow chart

Force field analysis

Page 23
Failure Reporting, Analysis, and Corrective Action System

Pareto analysis

Nominal group technique

FMEA4

Trend analysis

Fault tree analysis

Cause and effect diagram (fishbone)

These tools are directly associated with the analysis and problem-
solving process.5 Advanced methods, such as statistical design of
experiments, can also be used to assist the FRB. The Canada and
Webster divisions of Xerox Corporation extensively use design of
experiments as their roadmap for problem solving;6 they find this
method helps their FRBs make unbiased decisions.

Any resulting corrective action should be monitored to ensure that

causes are eliminated without introducing new problems.

Failed Parts Procurement

A failed parts procurement process should be established to assist
the FRB in failure analysis. Failed parts may originate from field,
factory, or company stock, or they may become defective during
shipment. Each of these scenarios requires different action. The
procedure should clearly define the process, including the timeline
and responsibilities for at least the following tasks:

Shipment of failed parts from field to factory

Procurement of failed parts from various factory locations

4. Failure Mode and Effects Analysis. More information is available through the Failure
Mode and Effects Analysis (FMEA): A Guide for Continuous Improvement for the Semiconductor In-
dustry, which is available from SEMATECH as technology transfer #92020963A-ENG.
5. Analysis and problems solving tools are described in the Partnering for Total Quality: A To-
tal Quality Toolkit, Vol. 6, which is available from SEMATECH as technology transfer
#90060279A-GEN.
6. A fuller description of the Xerox process can be found in Proceedings of Workshop on Acceler-
ated Testing of Semiconductor Manufacturing Equipment, which is available from SEMATECH as
technology transfer #91050549AWS.

Page 24
Failure Reporting, Analysis, and Corrective Action System

Failed part reports from each location

Communications between material disposition control and

reliability engineering

Sub-tier supplier failure analysis

Procurement of sub-tier supplier PAR and implementation of

corrective action

Disposition of failed part after RCA is completed

Corrective CORRECTIVE ACTION

Action

When the cause of a failure has been determined, a corrective action

plan should be developed, documented, and implemented to
Design & eliminate or reduce recurrences of the failure. The implementation
Production
plan should be approved by a user representative, and appropriate
change control procedures should be followed. The quality
Test &
Engineering assurance organization should create and perform incoming test
Change
procedures and inspection of all redesigned hardware and soft-
Control
ware. SEMATECH has developed an Equipment Change Control
System7 that you may find useful (see Figure 13).

To minimize the possibility of an unmanageable backlog of open

failures, all open reports, analyses, and corrective action suspension
dates should be reviewed to ensure closure. A failure report is
closed out when the corrective action is implemented and verified
or when the rationale is documented (and approved by the FRB) for
any instances that are being closed without corrective action.

7. Refer to Equipment Change Control: A Guide for Customer Satisfaction, which is available
from SEMATECH as technology transfer #93011448AGEN.

Page 25
Failure Reporting, Analysis, and Corrective Action System
 An effective change control system incorporates the following characteristics.
An established equipment baseline exists with current and accurate
V Design specs and drawings
V Manufacturing process instructions
V Equipment purchase specifications

All changes are identified by using

V The equipment baseline V Calibration systems
V Raw materials supplier change control V Traceability systems
V Procurement methods and procedures V Materials review process
V Incoming quality control
All changes are recorded, forecasted, and tracked in an equipment
change database incorporating
V Name of change V Qualification data
V Change description V Implementation plan
V Reason for change V Start date
V All equipment affected V Completion date
V Retrofit required/optional V Status of change
V Concerns/risks

Proper investigation of all changes is performed by a Change Evaluation

Committee made of experts from
V Equipment manufacturing V Environment, health, and safety
V Equipment design V Procurement
V Software design V Process engineering
V Quality and reliability V Marketing
V Service and maintenance

A Change Evaluation Committee is chartered to

V Validate change benefits using statistical rigor as required
V Specify change qualification and implementation plans
V Manage and drive change implementation schedules
V Update equipment specifications and drawings
V Maintain the equipment change database
V Communicate with the customer

In summary, a change control system is functioning well when it systemati-

cally provides
Identification of all changes relative to an equipment baseline
Recording, forecasting, and tracking of all changes
Proper investigation of all changes
Communication to the customer

Figure 13 Equipment Change Control Characteristics

Page 26
Failure Reporting, Analysis, and Corrective Action System
 FRACAS DATABASE MANAGEMENT SYSTEM

The FRACAS Database Management System (DBMS) facilitates

failure reporting to establish a historical database of events, causes,
failure analyses, and corrective actions. This pool of knowledge
should minimize the recurrence of particular failure causes. This
DBMS is not product specific. It is available from SEMATECH8, or
you can implement the same concept in whatever software you
choose. However, the database should have the same software
requirements and characteristics at both the suppliers and users
sites.

This section describes characteristics of the SEMATECH product. If

you are creating your own implementation, you should develop the
following aspects. Sections follow that describe each table more
fully.

Configuration Table

Events Table

Problems Table

Reports

8. Available from SEMATECH Total Quality division. Software transfer and accompanying
documentation in press.

Page 27
Failure Reporting, Analysis, and Corrective Action System
Configuration Table
The configuration table accepts information about each machine or
software instance. There are six standard fields, as shown in the
following table. In the SEMATECH FRACAS database, there are
seven optional fields that can be customized for your purposes,
such as customer identification number, software revision, process
type, engineering change number, site location, etc.

TABLE 2 STANDARD FIELDS OF THE CONFIGURATION TABLE

Field Name Description

Serial Number User input.

Site Enter the location of the machine.

System User input.

Phase User input. Enter the current Life Cycle Phase of

the machine. Required field.

Phase Date User input. Enter the date the machine went into
the current Life Cycle Phase.

Weekly Operational User input. Enter the number of hours the

Time machine is scheduled to be in operation.

Page 28
Failure Reporting, Analysis, and Corrective Action System
Events Table
The primary purpose of the FRACAS application is to record
downtime events for a particular system. The fields of the Events
Table are described in Table 3. To understand these fields fully, you
need to understand the relationship between the problem fields
(System, Subsystem, Assembly, Subassembly, and/or Sub-subas-
sembly) and the reliability code.

The reliability code field contains one or more of the problem field
names. Data (if any) from the problem fields automatically helps
you select or create the reliability code. Once you select a code,
associated fields are updated automatically.

TABLE 3 EVENTS TABLE FIELD DESCRIPTIONS

Field Name Description

Serial Number User input. The serial number of machine having

event.

Date User input. The date the event occurred. Enter in a

MM/DD/YY format.

Time User input. The starting time of the event. Entered

in 24-hour time (HH:MM:SS). Seconds are not
required.

Duration User input. The duration of the event. Entered in a

HH:MM:SS format. Seconds are not required.

System Automatic input based upon the serial number.

Subsystem User input. The optional subsystem code (if the

part is at a subsystem level) involved in the event.
Related to the System codes.

Assembly User input. The optional assembly code (if the part
is at an assembly level) involved in the event.
Related to the selected System and Subsystem
codes.

Operator User input. Name of the operator when the event

occurred.

continued on next page

Page 29
Failure Reporting, Analysis, and Corrective Action System
 TABLE 3 EVENTS TABLE FIELD DESCRIPTIONS, CONTINUED

Field Name Description

FSE Name User input. Name of field service engineer who

responded to the event.

FSR# User input. The field service report number.

Subassembly User input. The optional subassembly code (if the

part is at a subassembly level) involved in the
event. Related to the selected System, Subsystem,
and Assembly codes.

Sub-subassembly User input. The optional sub-subassembly code (if

the part is at a sub-subassembly level) involved in
the event. Related to the selected System,
Subsystem, Assembly, and Subassembly codes.

Reliability Code User input. The problem reliability code. This is

validated against current reliability codes. A new
one may be added by selecting New from the
pop-up list box.

Life Cycle Phase The current phase of the system. Defaults to

current phase from Configuration file.

Down Time Category User input. Downtime code. Either scheduled or

unscheduled. Scheduled downtime should be used
for holidays, weekends, or any other time when the
machine is not scheduled to be in operation.

Part Number User input. The machine part number.

Relevant Failure User input. Whether the failure was relevant or

not. Y for yes; N for no.

PM User input. Whether the downtime was due to

preventive maintenance. Y for yes; N for no.

Problem User input. The description of the problem. This

can be any number of characters.

Repair Action User input. The description of the repair action.

This can be any number of characters.

Page 30
Failure Reporting, Analysis, and Corrective Action System
Problems Table
In the SEMATECH implementation, the Problem Record allows you
to view, modify, or add records for problems. The fields of the
Problems Table are described in Table 4.

TABLE 4 PROBLEMS FIELD DESCRIPTIONS

Field Name Description

System User input from Events screen. If adding from

Events screen, automatic input based upon the
serial number.

Subsystem User input. Optional subsystem code (if part is

subsystem level) involved in event. Related to
System code. See also Reliability Code.

Assembly User input. Optional assembly code (if part is at

assembly level) involved in event. Related to
selected System and Subsystem codes. See also
Reliability Code.

Subassembly User input. Optional subassembly code (if part is at

subassembly level) involved in event. Related to
selected System, Subsystem, and Assembly codes.
See also Reliability Code.

Sub-subassembly User input. Optional sub-subassembly code (if part

is at sub-subassembly level) involved in event.
Related to selected System, Subsystem, Assembly,
and Subassembly codes. See also Reliability Code.

Reliability Code Automatically generated from the above fields. A

sequence number will be added to the code.

Title/Description User input. Simple title or description of event.

Initial Status User input. Required. Status of problem when first

reported. If problem is being added from Event
screen, date of event is entered automatically as
initial status date.

Fault Category User input. Event category.

Error Codes User input. Error code for selected tool.

Assignee User input. Staff member who assigned this prob-

lem.

Insufficient Information Date when insufficient information status was

declared.
continued on next page

Page 31
Failure Reporting, Analysis, and Corrective Action System
 TABLE 4 PROBLEMS FIELD DESCRIPTIONS, CONTINUED

Field Name Description

Sufficient Information Date when sufficient information status was

declared.

Defined Date when defined information status was

declared.

Contained Date when contained information status was

declared.

Retired Date when retired information status was declared.

Root Cause Original (planned) date for fixing problem.

Solution Original

Root Cause Current date for fixing problem.

Solution Current

Complete Completion date for fixing problem.

RCS Documentation User input. Points to documentation of fix; for ex-

ample, Engineering Change Numbers (ECNs).

Comments User input. Memo area for problem comments.

Can be any number of characters.

Report Types
Your reporting mechanism for this database should be flexible
enough to accommodate various customer types and their needs.
For example, you need to be able to pull reports for an FRB analysis
as well as for the users, sub-tier suppliers, and those who are
addressing corrective actions. At a minimum, the following reports
should be available:

Trend charts to show a number of events during a time period.

Event history to show a list of events for during a time period.

Problem history to show a list of problems during a time period.

Reliability statistics to demonstrate the system performance in

terms of MTTR, MTBF, mean time to failure (MTTF), availabil-
ity, etc.

Page 32
Failure Reporting, Analysis, and Corrective Action System
 GLOSSARY

Corrective Action A documented change to design, process,

procedure, or material that is implemented and
proven to correct the root cause of a failure or design
deficiency. Simple part replacement with a like item
is not considered corrective action.

Equipment Life Cycle The sequence of phases or events

constituting total product existence. The equipment
life cycle is divided into the following six phases:

Concept and feasibility

Design

Prototype (alpha-site)

Pilot production (beta-site)

Production and operation

Phase-out

Error Human action that results in a fault (term is usually

reserved for software).

Error Code Numbers and/or letters reported by the equipments

software that represent an error type; code helps
determine where the fault may have originated.

Failure An event in which an item does not perform its

required function within the specified limits under
specified conditions.

Failure Analysis A determination of failure cause made by use of

logical reasoning from examination of data, symp-
toms, available physical evidence, and laboratory
results.

Page 33
Failure Reporting, Analysis, and Corrective Action System
Failure Cause The circumstance that induces or activates a failure.
Examples of a failure cause are defective soldering,
design weakness, assembly techniques, and software
error.

Failure Mode The consequence of the failure mechanism (the

physical, chemical, electrical, thermal, or process
event that results in failure) through which the
failure occurs; for example, short, open, fracture,
excessive wear.

Failure Review Board (FRB) A group consisting of representatives

from appropriate organizations with the level of
responsibility and authority to assure that failure
causes are identified and corrective actions are
effected.

Fault Immediate cause of failure. A manifestation of an

error in software (bug), if encountered, may cause a
failure.

Fault Code Type of failure mechanism categorized to assist

engineering in determining where the fault may
have originated. For example, a fault may be
categorized as electrical, software, mechanical,
facilities, human, etc.

FMEA Failure Modes and Effects Analysis. An analytically

derived identification of the conceivable equipment
failure modes and the potential adverse effects of
those modes on the system and mission.

FRACAS Failure Reporting, Analysis, and Corrective Action

System. A closed-loop feedback path by which
failures of both hardware and software data are
collected, recorded, analyzed, and corrected.

Page 34
Failure Reporting, Analysis, and Corrective Action System
Laboratory Analysis The determination of a failure mechanism
using destructive and nondestructive laboratory
techniques such as X-ray, dissection, spectrographic
analysis, or microphotography.

Mean Time to Repair (MTTR) Measure of maintainability. The

sum of corrective maintenance times at any specific
level of repair, divided by the total number of
failures within an item repaired at that level, during
a particular interval under stated conditions.

Mean Time Between Failures (MTBF) Measure of reliability for

repairable items. The mean number of life units
during which all parts of the item perform within
their specified limits, during a particular measure-
ment interval under stated conditions.

Mean Time to Failure (MTTF) Measure of reliability for nonrep-

airable items. The total number of life units of an
item divided by the total number of failures within
that population, during a particular measurement
interval under stated conditions.

Related Event A failure that did not cause downtime. A related

event or failure is discovered while investigating a
failure that caused downtime. A related event could
also be a repair or replacement during downtime
caused by the main event.

Relevant Failure Equipment failure that can be expected to occur

in field service.

Reliability The duration or probability of failure-free perfor-

mance under stated conditions.

Reliability Code A functional traceable description of the

relationships that exists in the equipment. Codes are

Page 35
Failure Reporting, Analysis, and Corrective Action System
 derived in a tree fashion from top to bottom as
System, Subsystem, Assembly, Subassembly, and
Sub-subassembly (typically lowest replaceable
component). Codes vary in size; however, a
maximum three alphanumeric descriptor per level is
recommended. For example, AH-STK-ROT-XAX-
MOT for Automated Handler-Stocker-Robot-X_Axis-
Motor.

Repair or Corrective Maintenance All actions performed as a

result of failure to restore an item to a specified
condition. These can include localization, isolation,
disassembly, interchange, reassembly, alignment, and
checkout.

Root Cause The reason for the primary and most fundamental
failures, faults, or errors that have induced other
failures and for which effective permanent corrective
action can be implemented.

Uptime Time when equipment is in a condition to perform

its intended function. It does not include any
portion of scheduled downtime and nonscheduled
time.

Glossary References:
MIL-STD-721C Definitions of Terms for Reliability and Maintainability
(1981).

Omdahl, T. P. Reliability, Availability, and Maintainability (RAM)

Dictionary, ASQC Quality Press (1988).

SEMI E10-92. Guideline for Definition and Measurement of Equipment

Reliability, Availability, and Maintainability (RAM), SEMI (1993).

Page 36
Failure Reporting, Analysis, and Corrective Action System
 FRACAS Functions and Responsibilities
Responsibilities Function Description

Failure Operators Identify problems. Call for maintenance.

Annotate incident.
Maintenance Correct problem. Log failure.

Failure Report Maintenance Generate report with supporting data (time,

place, equipment, etc.).

Data Logging Reliability Log all failure reports. Validate failures and
forms. Classify failures (inherent, induced,
false alarm, etc.).

Failure Review Failure Review Board Determine failure trends. Prepare plans for
action. Identify failures to the root cause.

Failure Analysis Reliability and/or Review operating procedures for error. Pro-
Problem Owners cure failed parts. Decide which parts will be
destructively analyzed. Perform failure anal-
ysis to determine root cause.
Quality Inspect incoming test data for item.

Corrective Action Design Redesign hardware/software, if necessary.

Sub-tier Prepare & provide new part or test proce-
Supplier dure, if necessary.
Quality Evaluate incoming test procedures. Inspect
redesigned hardware/software.

Post-data Review Reliability Close loop by collecting and evaluating post-

test data for recurrence of failure.
 2706 Montopolis Drive
Technology Transfer Austin, Texas 78741 Printed on Recycled Paper
#94042332A-GEN (512) 3563500 Copyright 19931994, SEMATECH, Inc.