LASCAD
LASCAD
International Journal of Project Management Vol. 14, No. 2, pp. 103-110, 1996 Copyright 1996 Elsevier Science Ltd and IPMA Printed in Great Britain. All rights reserved 0263-7863/96 $15.00 + 0.00
0263-7863(95)00067-4
This paper illustrates the dangers of trying to implement high technology projects, without sufficient technical research and development, adequate management organisation and due regard to social factors. The London Ambulance Service is the largest in the world. During the early 1990s, it attempted to introduce a computer-aided despatch system. This system was finally commissioned in October 1992, about 9 months late and failed within 2 weeks. This case study traces the history of the project and identifies the flaws that led to its failure. Copyright Elsevier Science Ltd and IPMA
Keywords:hightechnologyprojects, LondonAmbulanceService, computer-aideddespatch, cultural change, business process re-engineering, information technology
The London Ambulance Project arose from a need to increase performance and reliability of a service which had an extremely large, complex task to execute. The organisation of the service had remained largely stagnant for 10 years and had been demoralised by a debilitating industrial relations dispute. The project attempted to introduce a computerised system to receive emergency calls and send ambulance crews to required destinations. At the same time, new techniques were introduced to monitor the location and availability of the emergency services. A 'big-bang' approach was adopted with all systems intended to go live on 8 January 1992. This was attempted, despite the known existence of major flaws in the system and the lack of trials that could have demonstrated other serious inherent problems. The implementation on day 1 was not successful and a phased introduction was adopted. Some actions taken to retrieve the situation made problems worse. The project passed the point of no return and for various reasons the situation could not be recovered. The contingency action taken in the early part of 1992 could not compensate for the fundamental shortcomings of the technical solution, together with inadequacy of the management organisation. A major crisis point was reached in October of that year. There were a large number of reasons why the system failed, many of which would not have been serious on their own. Taken collectively they proved to be catastrophic. The main reasons were: The management did not appreciate that this was a
business re-engineering project and tried to run it as a project for the purchase and installation of information technology. Due to external pressure to achieve results, insufficient time was allowed for development and proving of the extremely complex technical solution. There was a lack of disciplined technical approach. There was little attempt to manage the changes in the organisation's culture as part of the project. These all contributed to the problem but perhaps the single most important factor was the inadequacy of the organisation to control such a large and technically complex operation.
*Entrant for the Sir Monty Finniston Award 1995 sponsored by IBM
The London Ambulance Service (LAS) was founded in 1930, and was expanded to take in all or part of eight other services in 1965 when the Greater London Council was also established. From 1974-1990, (when the project began), LAS had been managed by the South Thames Regional Health Authority (STRHA) on behalf of the National Health Service. It was a quasi-independent body with its own board, managed only at arm's length by the RHA. Little change had been made to their way of operating during the 10 years preceding 1990. 103
Performance requirements
The nationally recognised standards of performance for ambulance services issued by the Operational Research Consultancy (ORCON) call for the time taken from receipt of an emergency call to the issue of instructions to an ambulance station or crew to be no more than 3 min. An ambulance is required to arrive at the scene within 14 rain. This is an onerous requirement that could not be met by LAS with its existing manual system o f activating the
Table 1 Staff numbers Staff category Numbers Ratio %
Operational (accident and emergency) Operational (patient transport service) Control assistants Managers (accident and emergency) Managers (patient transport) Administration and clerical Maintenance and ancillary
54 22 8 4 4 4 4
LAS was trying to introduce a major innovation. How different was it and what were the risks? In order to understand the new computerised system it is first necessary to look at the manual system that the computerised version was intended to replace.
Answering a call for help--the manual system The manual system is illustrated in Figure l. When a '999'
305 445 2 4 9 8 2
1
Accident and emergency ambulances Patient transport service ambulances Emergency control vehicles Emergency equipment vehicles Rapid response units Driver training units Motorcycle response units Helicopter 104
39 57
call comes into Central Ambulance Control, the call taker, a control assistant, writes the details on a form. A book of maps is used to locate the incident and provide a reference. The completed incident form is placed on a conveyor which transports it to a central collection point. The forms are collated and the details reviewed to eliminate duplication and assigned to a resource allocator, based on the three London divisions--north east, north west and south. Each resource allocator has an 'activation box' containing forms
Reference
Added
toCentral Point
,.~
Eliminated
__
Resource I Despatched by Telephone Figure 1 Operation of the manual system giving the status and location of their resources. This information is provided continuously by a radio operator. The allocator examines the forms for his/her own area and decides which resource should be mobilised. He/she notes this on the form and then passes it to a despatcher who then telephones the relevant ambulance station or passes mobilisation instructions to the radio operator if the ambulance is already mobile. There are some obvious problems with a totally manual system: At times a caller gives incomplete or inaccurate details; the identification of the precise location can be difficult and a number of alternatives have to be explored in the book of maps. The physical movement of paper forms around the control room is inefficient. Maintaining up-to-date vehicle status and location data is slow and laborious, relying on allocators' intuition and reports from ambulances as relayed through the radio operators. Voice communication with ambulances is time consuming, leading to mobilisation queues at peak times. Identification of duplicate calls is prone to error relying on human judgement and memory. Call-backs are not handled efficiently, with call administrators leaving their posts to talk to allocators. The identification of special incidents requiring a rapid response unit, a major incident team or a helicopter, relies totally on human judgement. However, there are situations where human brain power is superior to computer logic. These are where the human operator knows other unrelated information (such as the existence of a local football match or a major road works) and can apply it effectively to the current situation. The reduction of the facility for human intervention in the allocation of resources was to prove a major disadvantage in the troubled times ahead.
The aims and objectives of the new computerised system The aim was to create an almost totally automatic system, so the majority of calls to Central Ambulance Control could be dealt with by the call taker, who would despatch the most suitable ambulance resource. Only in the most complex cases would an allocator be needed to identify and allocate the best resource (and he/she would have superior tools at his/her disposal). All other allocations would be carried out by the call taker, who would take the call and see the incident through to completion. The computerassisted despatch system was therefore to provide the following command and control functions automatically:
Call taking and verifying incident details including location. Resource identification, determining which ambulance to send. Resource mobilisation, communicating details of the incident to the ambulance to be dispatched. Ambulance resource management, primarily the positioning of suitably equipped and staffed vehicles to minimise response times. Provision of management information to assess performance and help in medium- and long-term resource management and planning. The new system was to provide computer-aided despatch, computer map display and an automated vehicle location system. These new elements should interface with the existing mobile data terminals (installed as part of a previously aborted system) and integrated communications 105
Servers I
Sites
(5 ofO
Mo ech I--
Interface System
Radio
Station Printers
Automatic
I D' trib tr I
Call Taker
Emergency Call
Time scale
The LAS was under significant pressure from all directions to achieve a rapid improvement in their service. Accordingly, the target programme dictated by their senior management was as shown in Table 3. With this short time scale it was inevitable that the programmes for policy making, planning and implementation would overlap. We shall review how this programme evolved in the next section.
Table 3 Milestone Milestone competition dates
Completion of system requirement specification Completion of system design specification System implementation
London Ambulance Service computer-aided despatch system: M Hougham Project management and control
A team was set up to prepare the system requirements specification, (SRS), composed of: Director of Support Services (chair) Systems manager A contract analyst The control room services manager Staff representing training, communications and other areas administrator. Other departments appear to be well represented but in the face of the major deficiencies inherent in the technical solution, they were powerless to prevent the problems soon to be revealed. The computer-aided despatch system and its software development were to be carried out using the PRINCE project management methodology. No clarification was made as to how this was to be applied and there were no personnel in the LAS team experienced in its application. A short training course was carried out but this proved to be insufficient and the disciplines of PRINCE were not adhered to or enforced. Throughout the contractor selection process, there had been ambiguity over the managerial role of the contractor. LAS had intended that the contractor's associate in the consortium, a large hardware manufacturer, would take the role of prime contractor and lead the project. In the event, the contract was placed with a small software house who subcontracted the hardware back to the other party. The contractor was therefore not prepared or set up to execute a prime contractor role and rapidly withdrew from activities of this nature. Responsibility for the overall coordination therefore reverted to the Director of Support Services who found himself spending more and more of his own time doing low level co-ordination and progress chasing. He had no middle management level personnel to do this for him. The Director of Support Services found his time increasingly under pressure as he had to carry out his normal duties as well as managing the project. With a more experienced professional project manager, many of the problems would have been identified earlier.
Invitations to participate were given to union representatives but there was little involvement from ambulance crews as there were problems with the staff consultation process at the time. The contract analyst did most of the work with direct assistance from the systems manager. As part of the development of the system requirements specification a companion paper containing a revised operational method of working was produced. This was aimed at both the Central Ambulance Control staff and ambulance crews but although it would impact seriously on the way they worked there was little consultation with the latter. A small sub-team consisting of the contract analyst and the systems manager was set up to handle the contract placement with additional help from a supplies manager from regional recruitment. The prime responsibility for the technical evaluation of the tenders fell on the contract analyst and the systems manager. The experience of the representative from regional supplies was in procurement in its most general sense rather than specific IT procurement. The systems manager was a career ambulance man who had taken over responsibility for LAS systems and communications some years before. Thus, a contractor and a systems manager (who incidentally, was arguably unsuitably qualified and knew that he was to be replaced and made redundant) were put in charge of the procurement of an extremely complex and high-risk computer system. An audit of the selection process was carried out by the systems manager of the Scottish Ambulance Service. The technical aspects of the tenders should have been adequately covered, however there was no one with specific project management or contract management experience involved in the process. This gave rise to serious deficiencies: Choice of a small IT contractor to manage a major programme. Acceptance of a programme which proved unachievable. Lack of definition of contractors' responsibilities. Acceptance of a contract price which, although close to the available budget, was unrealistic in comparison with the other quotations. When the project moved into the development stage, the project team was changed to: Director of Support Services LAS contract analyst Control room services manager Director of operations Training manager Public affairs manager CTS representative (communications) Administrative support Supplier representatives
Technical development
Work on the preparation of the system's requirements specification (SRS) went ahead in 1990 against a background of the recent abandonment of a previous computer-aided despatch system dating from the 1980s which could not technically handle the demands placed on it. It was an ambitious document, the functionality of the proposed system being greater than that available from any existing system. There was no evidence of any formal sign-off of the SRS. Other ambulance services do have systems support, but none of them could meet the LAS requirement. They were therefore discounted. The SRS was very detailed and contained a high degree of precision on how the system was intended to operate, being quite prescriptive, reflecting the culture of the organisation and providing little scope for additional ideas to be incorporated from prospective suppliers. However, as is normal in any SRS, there were areas not fully defined; notably details on the relationship and interfaces to other LAS systems such as: Radio interface system Communication system Patient transport system The full systems design specification (SDS) was prepared by the contractor during June and July 1991. The computer hardware was considered to be very powerful and generally resilient. The LAS network was not large in comparison with those of other organisations. The workstations were 486-based, running at 25 MHz using a Microsoft Windows environment that allowed an attractive and friendly graphical user interface and simultaneous 107
It is noticeable that there was no direct representation from the ambulance drivers (the users) and no contract
The programme
It is clear that neither the computer-aided despatch team, the users nor the hardware were ready for commissioning of the system on the due date of 8 January 1992 or on the revised date for full implementation of 26 October 1992. Significant facts, dates and activities were as follows: (a) Work on a previous system that had been on-going since the early 1980s was abandoned in 1990 due to its lack of capacity for dealing with the anticipated workload. (b) The new management team was formed in 1990 and the system requirement specification was completed in February 1991. (c) The system development and procurement contract was first advertised in the European Journal on 7 February 1991. The recommendation to award the contract was endorsed by the LAS Board on 28 May 1991. The contract for the radio interface system was placed on 16 September 1991. (d) The contractor prepared the full system specification during June and July 1991. He then had 5 months to achieve full system implementation by the due date of 8 January 1992. (e) By mid-December 1991 it was clear that this date could not be achieved. At that point the computer-aided despatch software was incomplete and largely untested. The radio interface system hardware and software were not fully delivered and tested. The installation of the ambulance tracking equipment (Datatrak) was incomplete and its reporting accuracy under question. (f) In January 1992 attempts were made at functional and load testing of the system. These tests were inconclusive and, over the next few months, various system elements were tested, but never as a fully integrated whole. (g) During the first 9 months of 1992 the system was implemented piecemeal across different London Ambulance divisions. (h) The system was fully implemented at 07:00 hours on 26 October 1992. As the day wore on the system became overloaded and it was necessary to revert to semi-manual operation. This solution worked with reasonable success up to the early hours of 4 November at which time it slowed significantly and then locked up altogether.
Full system implementation The key changes made in order to enable the system to go live were:
Re-configuring the control room, separating resource allocators from radio operators. Installing more terminals and radio interface screens. Discontinuing the use of the manual paper activation box back-up system. Operating over the whole of London rather than in three divisions. Only using system proposed allocations with call takers able to allocate resources directly if they were less than 11 min away from the incident. Separate allocators handling 999 calls, doctors' urgent calls and patient transport calls. When the full system was implemented, instructions were given for minimal use of voice radio communications. The physical changes to the control room meant that staff were working in unfamiliar positions, without paper backup and less able to work with colleagues with whom they had previously jointly solved problems. The system itself did not fail in a technical sense, but much of the design had fatal flaws that cumulatively led to the symptoms of system failure. The system needed perfect information. However, because of the problems with communications, vehicle status and location reporting, it rapidly knew the correct status and location of fewer and fewer vehicles. The changes to Central Ambulance Control prior to going live made it extremely hard for staff to intervene. Furthermore, the restrictions placed on voice radio communications removed the essential feedback loop of real information. The overall effect on 26 October 1992 was that, as the day's normal workload increased, the whole system rapidly became unusable. A major reason for this was that errors could not be identified or corrected, which had the following impact: The system made incorrect allocations, sending multiple vehicles to the same incident or not selecting the closest vehicle. The system had fewer and fewer resources to allocate and did not know their correct status. Failures due to incorrect or missing information caused the system to generate large numbers of exception messages. Unrectified exception messages themselves generated more 109
Phase /--Using the call-taking software and gazetteer to help with the recording and location of incidents. Printers would then pass information to the allocators, who, using their traditional methods would mobilise the crews by radio or telephone. Phase 2 - - A s Phase 1, but with vehicle locations reported by Datatrak and mobilisation notification using the mobile data terminals. Human allocators would determine the optimum resources to be used as previously. Phase 3--Full implementation, whereby call takers would allocate using automated resource proposals provided that resources could arrive within 11 min of activation. Otherwise the allocators would identify and allocate the most suitable resources. Phase 3 was designed to operate without paper backup.
This phased implementation should have been planned in
exception messages which added to the queue of items requiring attention. The congestion that could not be cleared became so great that it was found necessary to abort full implementation and revert to the semi-manual system of operation. All then went reasonably well until the early hours of 4 November, when the system slowed significantly and then locked up altogether. Attempts to restart it failed and Central Ambulance Control, on advice from senior management, reverted to a fully manual, paper-based system with voice or telephone mobilisation. (a) What was the cause o f the failure? The single technical fault (which was the final item to make the system inoperable) was a minor programming error. Three weeks previously a programmer had inadvertently left a programme code in the system that caused a small amount of server memory to be used up and not released each time a vehicle was mobilised. Over a 3-week period this had gradually used up all available memory, causing the system to crash. (b) Why did the back-up server not then come into operation ? This was due to the addition of the printers that had never been envisaged for the paper-less system and were added as a short-term expedient to implement partial operation on 8 January 1992. The fall-back to the second server was never implemented as part of this level of operation.
The former Chief Conciliation Officer of the Advisory Conciliation and Arbitration Service. Appropriate support was utilised from other personnel and confidential evidence was taken from a range of organisations associated with the project. The report was issued in February 1993. Several recommendations were made covering a range of topics including the functions of the LAS Board, management responsibilities, industrial relations and the immediate and future resource implications of the implementation of the report, and number of conclusions were drawn and recommendations made. The most significant conclusions were: LAS should continue to plan the implementation of the computer-aided despatch system. A suitably qualified and experienced project manager should be appointed immediately to co-ordinate and control the implementation of the first stage of computer-aided despatch and an IT director, with direct access to the LAS Board, should be recruited. Any new system should be introduced over a reasonable time scale, must have total ownership by managers and staff and be introduced in a stepwise approach. Where possible, steps giving maximum benefit should be introduced first. A renewed attempt at implementing the system is now proceeding, taking full account of these recommendations.
The enquiry
An enquiry team was set up by the South West Thames Regional Health Authority in November 1992: 'To examine the operation of the computer-aided despatch system, including: the circumstances of its failures on Monday 26 October and Wednesday 4 November 1992; the process of its procurement; and to identify lessons to be learned for operation and management of the LAS against the imperatives of delivering service at the required standard, demonstrating good working relationships and restoring public confidence.' The team comprised: The Chief Executive of South Yorkshire Metropolitan Ambulance and Paramedic Service NHS Trust. A senior computer audit partner from a reputable firm of consultants.
Bibliography
The material used in the preparation of this case study was taken from the report of the Inquiry into the London Ambulance Service dated February 1993.
Michael Hougham 's early career was in military electronics with GEC. He moved into construction engineering project management in 1977 and worked with both John Brown E & C and Stone & Webster Ltd on a wide variety o f projects in the north sea, on land in the UK and overseas. His specialisation is project management controls. For the past three years he has managed Greenlands Management & Engineering Consultants Ltd. He has recently accepted the position of Executive Officer with the Association of Project Managers.
110