Evaluation of the Impact of the National Resource for Infection Control

Gemma Madle, Patty Kostkova, Sue Wiseman, Gawesh Jawaheer October 2008

Executive Summary This report describes the impact evaluation of the National Resource for Infection Control undertaken in 2008. Sixty five NRIC users were recruited to take part and the key results are described below. Most of the study participants were experienced in their profession and reported that access to the internet and employer support for using resources like NRIC in their workplace is generally good. How is NRIC used in the community? NRIC is used mostly for finding evidence to support policy development, to implement a change in practice at work or for personal education. In addition a significant number of visitors use NRIC to search for whats new. NRIC is often used in preference to source websites but the awareness of NRIC in the user community could be improved. How well received is NRIC and its services? NRIC is generally perceived to be a useful resource and provided relevant information in over 65% of visits. However awareness of all services but particularly the monthly newsletter and training courses/conferences information could be improved. A regularly updated home page is important for attracting users. Clarification is required about the purpose of the document reviews (Reviewers Assessments) as compared with the electronic catalogue card for each document. Coverage of NRIC is generally perceived to be good with few gaps in content. Popular suggestions for development include adding a discussion board and developing the template policy section into national model policies. Another suggestion was that NRIC could allow users to select the frequency of email alerts for new documents rather than just offer a monthly email. Access to external documents can cause problems for some users. Usability and navigation: The users in this study spend a significant amount of time per visit (average over 12 minutes) and visit on average 3 documents in one session. But only 10.4% of available document reviews are accessed. NRIC is generally perceived as an easy to use website but the main issue is with the display of the search results which are currently not ordered by date. It was suggested that workshops could be run to increase awareness of NRIC and help users navigate it more effectively. In terms of getting around the library browsing is more common than searching and when NRIC is browsed rather than searched it is more likely to have an impact on user knowledge. NRICs impact: NRIC had an impact on user knowledge in 52.8% of visits. The main reasons for no impact were that not enough information was found or the user couldnt access the document. NRIC has a positive impact in many areas of user work including policy development, training and education, implementing changes in practice and business case or proposal preparation. A significant proportion of NRIC use is to find information either on behalf of someone else or to pass information/advice on to patients or colleagues and users also direct others to NRIC to find information themselves. Recommendations: To promote policy sharing either by way of a clearing house or by only sharing model policies vetted by a reputable body Raise awareness of NRIC within the user community, particularly with students and less experienced staff Increase the educational remit of NRIC by extending links with educational bodies and providing more educational materials Change the order the search results are displayed in to a by date order Run training workshops in how best to navigate NRIC and produce an online guide

Introduction The National Resource for Infection Control (NRIC) was launched in May 2005 in response to National Audit Office,1,2 (2000/04) recommendations for a national infection control manual. The project funded by the Department of Health (UK) and endorsed by the UK National electronic Library of Infection (www.neli.org.uk) covers a broad range of infection prevention & control and infectious diseases information The importance of infection prevention and control policies and guidance cannot be over emphasised. They form the basis for all safe evidence based practice in any healthcare setting, outline the reasons why certain practice is important and also attempt to define all patient care within safe boundaries. NRIC has, over the 2 years since its launch, helped to disseminate the plethora of new Infection Prevention & Control guidance published and become a useful resource both in Primary and Secondary Care Trusts as well as the independent sector. The need to evaluate the impact of NRIC on professionals in the infection prevention and control field is essential to identify how successful it is and where future investment can be targeted to improve its impact. Therefore an impact evaluation was undertaken during Spring 2008. Data Collection Potential participants were recruited by an advert on the NRIC website, through the NRIC mailing list (approximately 1800 members), via a post on the Infection Prevention Society (IPS) website forums and through contacts in the IPS South West branch. The impact evaluation was conducted using the Impact-ED model for digital library evaluation (ref ECDL). This provides a set of criteria around which questionnaires and interviews are designed to collect appropriate data. The criteria can be found in Appendix 1. The methods used in the impact evaluation were as follows: Study registration (Feb 08) and end questionnaires (May 08) to find out who uses NRIC, how and when they were using it and for what reasons, compare answers before and after the study and to provide an opportunity for users to comment on services and suggest improvements. Information seeking/knowledge gain task (May 08) to examine how well users could complete an information seeking task i.e. find specific documents and find answers to questions using NRIC Pre and post questionnaire (Feb-May 08) to discover for what reasons people use NRIC at the point of use in their own words, what they know already to compare with what they think they have learnt from using NRIC and how they will apply this to their work Web server log collection (Feb May 08) to collect data on how the participants actually navigate the library and see how this compares with how they report using it and the impact it has on their work Interviews (July - Aug 08) to provide more in-depth information in users own words about how the site has an impact and how it can be improved The methods were designed to collect data to meet all the evaluation criteria and the results section presents the results for each criteria.
1

Improving patient care by reducing the risk of hospital acquired infection: A progress report (A report by the Comptroller and Auditor General) National Audit Office HC 876 Session 2003-2004: 14 July 2004 http://www.nao.org.uk/publications/nao_reports/03-04/0304876es.pdf (Summary) http://www.nao.org.uk/publications/nao_reports/03-04/0304876.pdf (Full Report)

The Management and Control of Hospital Acquired Infection in Acute NHS Trusts in England. (2000) HC 230 1999/2000 http://www.nao.org.uk/pn/9900230.htm (NAO Press Notice) Full Report http://www.nao.org.uk/publications/nao_reports/9900230.pdf Executive Summary http://www.nao.org.uk/publications/nao_reports/9900230es.pdf

Results Sixty five NRIC users signed up for the impact evaluation. Of these two officially dropped out, 53 completed the registration questionnaire, 32 completed pre and post visit questionnaires of which 72 sets were matched for analysis, and 31 completed the end of study questionnaire. In addition 5 users were interviewed. Only 10 users completed the information seeking task and two questionnaires were invalid due to users completing them after viewing the website, therefore the results for this are limited and analysis is not included in this report. The study ran from February 2008 to May 2008 with interviews taking place during July and August 2008. On arriving at the NRIC library participants were asked to login with a unique username and password, purely for the purposes of tracking their activity during the study period and linking data from the different collection methods. The small numbers of this study mean that generalisation and statistical analysis of results is not appropriate, however, the results do provide a useful insight into the impact of NRIC on user work and should not be discounted due to the lack of participation. The results are organised into the criteria as presented above in the methods section. 1. Community: 1.1 How the digital library is used in the work setting and what are the reasons for use The registration and end questionnaires showed that users tended to overestimate the frequency of their visits before the study period began, with more conservative reports at the end of the study (Figure 1.1a) and that either they had overestimated the actual frequency or that they were not always logging in to the library to visit it as actual visits were much lower than the reported use (Figure 1.1b). Reasons for use varied both in the reported reasons in the registration questionnaire (Figure 1.1c) and in actual reasons for specific visits from the pre and post visit questionnaires (Figure 1.1e). The former were selected from a multiple choice question and the latter were real reasons in the participants own words which were then grouped into the reasons shown in the chart. The majority of the information sought in NRIC was evidence to support or improve existing knowledge or practice but also a significant number of visits were to support personal education or staff training.

Figure 1.1a Reported frequency of use of NRIC

Figure 1.1b Actual recorded visits to NRIC during the 3 month study period

Figure 1.1c Reported reasons for use of NRIC

Figure 1.1d Type of information sought

Figure 1.1e Actual reasons for use for specific visits The interviews support the evidence from the questionnaires for the role of NRIC in policy development and personal education:

As I said ..I also need evidence for the policies, .And I find NRIC really useful for that because its got the documents for everything in there. Its also got the policy templates and I find them very useful. (Interviewee A) I demonstrate the site to all of my students because obviously they have to find meaningful references to support their academic assignments. I would certainly share it with professional colleagues, if, for example, we were working on a policy then I would direct them to the main references that Im using and wed just use that one web address, as an easy one, short one that they could find for themselves the relevant information. I also of course use it for myself. . its a lazy quick access point for myself, and often when Ive even forgotten which file Ive actually electronically saved the document on previous occasions I find myself going back to NRIC. (Interviewee B) if Im trying to hunt for an assignment, um, hunt for an article and Im not sure what it is but I know if I go into NRIC Ill find it quite quickly. I tend to use it a lot in work really, if Im asked a question, particularly in reviewing policies which Ive been involved in then Ill access it and have a look at other policies that have been placed on there. (Interviewee C) And as a one stop shop, rather than visiting source websites: I also of course use it for myself. . its a lazy quick access point for myself, and often when Ive even forgotten which file Ive actually electronically saved the document on previous occasions I find myself going back to NRIC. (Interviewee B) Its where I go to get the main infection control guidance and adviceumor if Im not sure whats new I look there to see what there is as well because we dont always get the information from the Department of Health because it goes to the NHS automatically but not to the HPA automatically. (Interviewee E)

Key points: NRIC is used mostly for finding evidence to support policy development, to implement a change in practice at work or for personal education. A significant number of visitors use NRIC to search for whats new It is used in preference to source websites

1.2 The suitability of the work environment for integrating the digital library into work practice e.g. access to the Internet 1.3 Acceptability of the digital library by employers/colleagues i.e. support given for using the digital library in work time & space Users generally felt that access to the Internet was easy in their workplace and there was a feeling of support from employers to use such resources although participants were more likely to agree with the former than the latter (Figure 1.2). There was no difference in the impact of NRIC on users who had good access and employer support to use such resources and those who did not.

Figure 1.2 suitability of work environment However three interviewees reported distractions in the work environment as a barrier to use: .I can be halfway through looking for something and the phone will ring or my bleep will go and then Ill have to go away. (Interviewee A) I probably use it most at home for myself because thats where I tend to do my more concentrated paperwork. The working day is stuffed with distractions. Interviewee B I suppose if my activitys high it might be on the backburner and I need it, Im always meaning to look at it more than I do (Interviewee D)

Key points: Access to the internet and employer support for using resources like NRIC is generally good

1.4 Awareness of the digital library in the target community, not just those who use it already Most users disagreed or neither agreed nor disagreed with the statement that Most of my colleagues are aware of NRIC and what it provides with only 23% of those who answered (n=30) agreeing or strongly agreeing that this was the case. Key points: Awareness of NRIC in the user community could be improved

1.5 Basic demographic information of users The majority of participants were nurses with 28 of the 52 responding that this was their profession. In reality this figure was probably higher as a number of participants entered infection control specialist or infection control practitioner and may well be nurses as well. Other professions included doctors, pharmacists, managers, and microbiologists. Thirty participants were definitely from the UK with 15 unspecified of which the majority were thought to be UK based. Other countries represented included Spain, Saudi Arabia, the UAE and Egypt. The most common highest level of qualification was a taught postgraduate degree and the most frequently stated number of years of experience in their profession was over 20 years suggesting the study participants were a group of higher level staff who are experienced in their field. Key points: Most of the study participants were experienced in their profession therefore more promotion to and research with students and less experienced staff may be beneficial

2. Services: 2.1 User satisfaction with the digital library & how it meets user needs At the start of the study most users reported the NRIC library to be either very useful (40.4%) or somewhat useful (38.5%) with only two specially reporting that it is not useful. There was no significant change in these results at the end of the study period. In the seventy two visits for which pre and post visit questionnaires were collected users found relevant information in 47 visits (65.3%). Specific comments included: NRIC, um, its a really good resource (Interviewee A) I find it a very useful resource (Interviewee B) Well again it just makes my job easier to do really, I think it makes me, um, it gives me the information I need to perform my role more efficiently. (Interviewee C) It makes a difference if Im trying to find out whats going on in the NHS (Interviewee D) Key points: NRIC is generally perceived to be a useful resource and provided relevant information in over 65% of visits

2.2 Preferred resources/services already offered by the digital library 2.4 Awareness of current digital library services and content to existing users Figure 2.2 shows the reported use of the different services offered by NRIC as reported by participants answering the end of study questionnaire. These services include the monthly email newsletter, reviewers assessments of documents within the library, information on conferences and meetings and policy examples and templates. Most users reported to use the services occasionally with lack of awareness being the most common reason for non-use (18.8% of respondents) and only 8% of responses saying they did not find a particular service useful. The most commonly reported use of a service was the reviews of documents which contrasts with the actual use of reviews where the web server logs showed that only 5 of the 48 available reviews for the documents viewed in the 72

visits analysed were actually viewed by users. This conflict in results is perhaps due to a misunderstanding of the terms as each document in the library has an electronic catalogue card with information about the document which also contains an excerpt of a review (if available). Users may have been confusing the catalogue card with the actual review. The policy templates were noted as useful by two interviewees: the policy templates and I find them very useful (Interviewee A) For instance we havent got a c.diff policy.so Ive looked on NRIC.its given me lots of ideas now and Im actually going to take them forward as examples for other people to read on a working group..you know sharing information from those that are available on NRIC (Interviewee C) One interviewee mentioned the email alert as being useful: if I get the email because we do get emails from NRIC dont we, then I will look at it.its been the most helpful thing and I would, I would like definitely for that to continue..its good on conferences and study days (Interviewee D) And another the training events displayed on the home page: But NRIC.the first viewing screen constantly alters and thats a very quick way of just sort of being made aware of anything significant thats come through and the training events can also be interesting its a way of alerting you to think of something that you might otherwise not have been aware of (Interviewee B)

Figure 2.2 Use of services

Key points: Awareness of all services but particularly the monthly newsletter and training courses/conferences information could be improved A regularly updated home page is important for attracting users Clarification is required about the document reviews (Reviewers Assessments) and their purpose as compared with the electronic catalogue card

2.3 Gaps in provision Twenty seven percent of the respondents to the end of study questionnaire felt that access to documents could be improved, particularly the problems of linking through to password protected documents in the Athens system. Three users thought that the policy template section could be enhanced and another three would welcome the opportunity for a discussion board. One felt the library should have a clearer purpose. Twelve users did not respond to any of these questions. Eighty percent of the respondents thought there were few gaps in content of NRIC, 17% several gaps and only one user thought NRIC had large gaps in coverage. Suggestions for improving content coverage were made in the end of study questionnaire as shown in Table 2.3. What areas of content do you feel could be more comprehensive? can't think of any reviews of clinical practice to aid management of infection in clinical practice publicise to all in healthcare profession Specific organisms More hospital Infection society document links, more journal articles provided/ lit search facility I don't belive I have used this enough to comment spefically. Theatres and CSSD endoscopy none waste disposal policies and personal protective equipments Development of a NAtional Policy I have not identified any. More on Hospital Infections particularly C Difficile Provision for non clinical staff who are subject to the same policies/advice in areas like IC. Also more links between research findings and practical applications. templates/examples of good practice all areas Fungal infection, resistences... Table 2.3 Suggestions for improvements to content coverage The interviews also highlighted the desire for a discussion board: I wondered about some kind of discussion board on there. Because I know Ive had questions I couldnt answer there, you could maybe put them on there so that maybe other folk could see them and answer them. (Interviewee A) is there a, um, not like a chat but a uma discussionI think, wasnt there talk that there was going to be one? Yes. So Id like to get involved in things like that (Interviewee C)

Two interviewees discussed the possibility of improving the policy templates and examples: NRIC I think could actually be a genuine depository for these documents.....the Department of Health documents alone dont really do it....NRIC could be ensuring that, rather than just offering the policies it should be trying to help people establish better practice in policy writing and perhaps giving the best examples nationally rather than just the examples that are offered. (Interviewee B) I definitely would like model policies not examples of policies because there are actually hundreds of examples of policies and thats not helpful.... (Interviewee D) Other topics mentioned were extending the educational focus of NRIC and introducing daily alerts. I feel the whole education opportunity is the thing that is missed by NRIC.I would like NRIC to be a one stop resource for free educational materials, to help people privately study the subject.I think the other thing I would like would be to see an extension of the authoritative journal articles..... (Interviewee B) if there could be something on NRIC about what is happening there about these competencies and then is there going to be some sort of generic course that infection control people can sign up to but there, so I suppose its that decision of whether NRIC could become more active where I think theres a gap (Interviewee D) having looked at PVL and at that time there wasnt anything on the site that I wasnt already aware of Ive become aware of for example, patient information leaflets produced by individual pcts about pvl so it would be good if there was stuff like that available on the site or some posted on the site (Interviewee E) I think that would be really good so that when.youre on your email every day, every time something came out there was an alert that would just make you feel more on top of things.you dread things coming out and you not being aware of it (Interviewee D) Other issues that were raised by the interviewees included NRIC being clearer on the origin or nationality coverage of documents, particularly English/Scottish/Welsh documents, and access issues for non-NHS staff to Athens password protected documents. Key points: Coverage of NRIC is generally perceived to be good with few gaps in content Popular suggestions for NRIC development include adding a discussion board and developing the template policy section into national model policies Access to external documents can cause problems for some users NRIC could allow users to select the frequency of email alerts for new documents rather than just offer a monthly email

3. Technology: 3.1 Basic web access log statistics Table 3.1 shows the basic access statistics for the 72 visits to the NRIC library that were analysed by pre and post questionnaires. The average time spent (excluding the time spent completing questionnaires) was quite high being over 12 minutes with one third of users spending over 15 minutes and one third of users spending under 5 minutes in the library. Users were viewing an average of 13 pages per visit and this included 3 documents. However the majority of visits accessed

either between 0 and 5 pages (41.7%) or 6 to 10 pages (34.7%). As noted earlier access to the document reviews was low. Basic access statistics for the 72 visits analysed Mean time spent per visit Mean number different pages viewed per visit Mean number of documents viewed per visit Median number of documents viewed per visit Total number of reviews available Number of reviews visited % available reviews visited Table 3.1 Basic access statistics Key points: These users spend a significant amount of time per visit (average over 12 minutes) On average 3 documents are visited in one session Only 10.4% of available document reviews are accessed 00:12:22 13.74 3.07 2 48 5 10.4%

3.2 Usability of the digital library On the whole there is satisfaction with the ease of use the site as shown in Figure 3.2 and the following comments: It is very easy to use and its easy to get into. (Interviewee A) its a simple website., it is a reliable site, you can get straight in there and go straight to mostly the documents youre looking for... (Interviewee B) I find it very easy to use. When I first got introduced to it my computer skills were not good, theyve improved since, theyve had to but now I find it a very easy site to use.. very easy to get around. (Interviewee C) I mean the general design of the site,..I find very easy, the way the items are listed, the titles of things that makes it easy to find. (Interviewee E) There was some issue with the ease of locating documents in NRIC (Figure 3.2) with several comments about the display of the search results table (Table 3.2). Seventeen percent felt that the search results table could be improved, specifically to order by date of publication or allow sorting by categories of the users choice, something supported by data from the interviews: the limitations are that the data capture can be far broader than the search terms, than the word terms typed in.: I think a main criticism is that, on the search facility. I know the document that Im looking for so I come to the site with some knowledge of what the references should be but if for example I typed in aseptic technique some appropriate documents will be electronically trapped by that search word but there will also be a lot of very strange and peripheral documents that actually make for a cluttered and less convincing search. (Interviewee B) when youre going through the research you know when youre looking a subject, its not always in order of date is it. That would be more useful. Sometimes you know, youve got a 1985 then a 2007

then youve got a 1986. So maybe start off with the newest ones then you can read the ones that are obviously most important first. (Interviewee C) I would like to think that things on the site were in more of a date orderit is the most recent piece of guidance, I would like it to do that (Interviewee D) Another user suggested workshops as a way to train users in navigating NRIC: sometimes I think I wish there was a workshop you could go to and say these are the things I would like NRIC to do because I always think Im sure that this is in there somewhere and I think a workshop for people to go to, to actually be talked and walked through the site would be really helpful and for lots of users to share this is what I want to use it for, how can we. it would help me if somebody did some sort of simple persons guide to it...... (Interviewee D)
Comments about the search results display and ease of use of the site. What would users like It would be helpful to be able to re-arrange documents in service area into order of date released more easily searchable tables of info Topic lists need to be enhanced ability to arrange documents in each section by date of release before browsing AS STATED PREVIOUSLY, BETTER MANAGEMENT OF DOCUMENTS E.G. CHRONOLOGICAL ORDER, MADE CLEAR WHEN SUPERCEDED, ETC. A search facility that sorts documents by date and by author. east access

Table 3.2 Comments about the usability of the library and user preferences

Figure 3.2 . Ease of locating documents in NRIC

Key points: NRIC is generally perceived as an easy to use website The main issue is with the display of the search results which are currently not ordered by date Workshops could be run to increase awareness of NRIC and help users navigate it more effectively

3.3 Search query analysis i.e. how are people searching the digital library, for what topics/parameters and are they finding what they are looking for In the 44 visits that used the search facility 88 searches were performed. Only one of these searches used the filtering options provided on the search page to filter by publication type (a review). The most common way to search was to enter a phrase (56.8%) or a single keyword (36.4%) with only four search queries using a Boolean AND structure. Table 3.3 shows what users were searching for. Relevant information was found in 61.4% of visits that searched, however in those that only searched and did not browse to navigate as well (n=17) information was found in only 9 visits (52.9%). Category of keyword Number of searches Document/Campaign 9 Infection/organism 23 Medication/treatment 5 Other 9 Procedure 15 Setting 10 Specific item 17 Table 3.3 Search keyword categories Key points: Users mostly search by phrase or single keyword The most commonly searched for topics are specific infections, organisms or items % of searches 10.2% 26.1% 5.7% 10.2% 17.0% 11.4% 19.3%

3.4 Navigation pathway analysis to identify how users are navigating the digital library and what services are commonly used The most popular method of navigating the website was to browse and search (27 visits) with 24 visits only browsing and 17 only searching. The most common pathway for browsing was to browse the resource pages with 25 visits (34.7%) accessing these pages. Table 3.4a shows the use of all the browsing options. Category Number of visits Browsed Resources 25 Browsed Policy 19 Browsed Settings 16 Browsed Clinical Practice 16 Browsed Diseases 13 Browsed Transmission 12 Table 3.4a. Popularity of browsing categories % of visits 34.7% 26.4% 22.2% 22.2% 18.1% 16.7%

Browsing was more effective than searching in terms of whether or not NRIC had an impact as shown in Table 3.4b, perhaps due to the issues with the search results display highlighted above.

Category Browsed only Searched only Browsed and searched

Confirmed/strengthened or changed knowledge 45.8% 29.4% 33.3%

Gained knowledge 50.0% 23.5% 37.0%

No impact 37.5% 52.9% 48.1%

In terms of how people actually use the library within the constraints of the work environment, one interviewee reported that: usually what Ill do is Ill try and save the documents and then Ill go back to them. So I hunt out all the different ones and rather reading them at the time I can save them and then go back when Ive got more time. (Inerviewee A) Key points: Browsing is more common than searching When NRIC is browsed rather than searched it is more likely to have an impact

4. Content: 4.1 knowledge and attitude changes as a result of using the digital library e.g. actual or selfreported User knowledge was confirmed, strengthened or changed in 36.1% of visits and knowledge was gained by the user in 37.5% visits. In all there was an impact on user knowledge in 52.8% of the 72 visits. Where knowledge was gained most commonly this was adding to the users existing knowledge about their query (20.8% of visits) although on 6 occasions NRIC did add to user knowledge in unrelated areas. Where there was no impact on user knowledge reasons were investigated and are shown in Figure 4.1. In the information seeking task there was a correlation between the number of relevant documents viewed and improvements in knowledge with the more documents viewed the greater the number of questions answered correctly although the number of users involved (n=8) was too small for proper analysis.

Figure 4.1 Reasons for lack of impact of NRIC on user knowledge

Key points: NRIC had an impact on user knowledge in 52.8% of visits The main reasons for no impact were that not enough information was found or the user couldnt access the document

4.2 The impact of using the digital library on user work e.g. clinical decision-making and patient care, use in creating documents such as policies, guidelines or coursework An impact score was calculated for different areas of user work to show how NRIC is having an impact in these areas. The score was calculated by dividing the number of visits in which there was an impact on user knowledge (i.e. confirmation, strengthening, change or gain) by the number of visits related to a specific area of work. A maximum score of 1 can be achieved. Figure 4.2 shows the impact of NRIC in the different work areas. Comments from the questionnaires are shown in Table 4.2 whilst interviewees comments are shown below. actually saved me a lot of time because the resources I need were all in one area. Instead of having to go into Google or whatever and look forever, or go across to the library.so that was good, saved me a lot of time. (Interviewee A) . one can always refer people and say youll find the documents there. So its a useful website, its a simple website. I think certainly it has helped me in terms of efficiency to speedily find some references when Ive most needed them..within seconds I can find the answer to a question quite easily whereas doing something like a full electronic literature search or wading through a Department of Health or Health Protection Agency website that would be a slower process. (Interviewee B)

If I need some information, if Ive been writing a report, or you know, Im preparing something to do training or want to give staff advice, its been good to get the research base behind me so I know Im coming from the right direction really...Ive actually printed off a couple of the c.diff policies that are out there, so its given me lots of ideas now and Im actually going to take them forward as examples for other people to read on a working group to develop our own policy. (Interviewee C) Although NRIC appears to save time, as would be expected time is also a barrier to use with 5 users reporting that lack of time was a barrier to the benefit of NRIC helping with their work. Navigation and lack of computer access are also barriers for 5 and 2 users respectively.

Figure 4.2. Impact of NRIC in different work areas (maximum score of 1 possible) What impact do you think using NRIC has on your work? Please give examples if available. excellent for finding evidence to base new local policies on gives good, up to date advice and resources limited impact prefer using search engines on the internet which have a major impact on my clinical work useful tool Helped me to develop evidence-based policies and patient information sheets - especially in the areas of hand hygiene and Clostridium difficile Useful one stop resource, but on ewhich could be greatly enhanced and developed Quick link to topic specific info Used to get accesst to some useful documents. Helpful for up to date info and leads to other sites Used for implementing hand hygiene able to locate information quickly and all in one plece updatiding me in with recent information & guidlines which will help in decision making Has made researching issues far easier Assited me finding information on HII re the Scottish site.

ALERTS FROM NEWSLETTER TO NEW GUIDANCE, EVIDENCE, ETC. VERY USEFUL AS ANOTHER WAY TO SEARCH FOR SOMETHING. It enables me to find the right information and to guide colleagues to this information, e.g. the PCT librarian. It provides the evidence based information that is required in answering clinical questions. It also alerts users of the Library and within the Trust to the Specialist Library for Infection and they can look also. Gives me information when I require it I don't think it has impacted alot, but if you are not aware of recent guidance it is useful I am now more aware of its usefulness and expect to use it more in the future It has been very useful when developing new policies or updating current ones, where i can easily see all current research/evidence base available I have only been in post for 3 years, so initially I used it a lot whenever I was unsure of a particular infetion great impact I am Infection Control Lead for Medicines Management within PCT.Information obtained from NRIC helps with this role. In my work as an infectious disease specialis, decisions are always changing, and I use to make some research before taking important decision. NRIC is one of the database in which I usually search Table 4.2 Impact of NRIC on user work Key points: NRIC has a positive impact in many areas of user work including policy development, training and education, implementing changes in practice and business case or proposal preparation

4.3 The dissemination of information found in the digital library to others Thirteen of the 72 visits were to find information to enable the user to pass on or give advice to other colleagues or patients. One specific visit was purely to find a url link to a Department of Health document for a colleague. In addition users direct others to the NRIC site rather than pass on the information themselves. Three interviewees reported that: I find it a very useful resource, in as much that its a one stop direction that I can give to colleagues, to students so that they can perhaps easily find the key references. Its easier, than to say, direct them to the Department of Health which is a very big website and is perhaps less likely to give them a successful experience. (Interviewee B) Ive recommended it to my colleagues as well. (Interviewee A) very often later on when Im with staff they will tell me theyve found something on NRIC even though I know Ive told them about it but theyve gone to NRIC and found it themselves and I think that probably makes it a bit more meaningful than me just trolling it out to them. So I think it is a useful resource for me to give to staff, clinical staff I think have used it quite a lot (Interviewee D) Key points: A significant proportion of NRIC use is to find information either on behalf of someone else or to pass information/advice on to patients or colleagues Users also direct others to NRIC to find information themselves

Discussion: Impact of the National Resource for Infection Control This research has shown that NRIC does currently have a positive impact on those that use it and that it is contributing in many ways to the infection prevention and control community. However, whilst the feeling toward NRIC is generally that it is a popular and easy to use resource several areas for development have been identified. If NRIC is going to fulfil its potential as a one-stop resource integrated into this community then continued investment and resource is necessary to allow it to develop in-line with user needs and wants. The following recommendations are made as a result of this research and user feedback. Recommendations for NRIC Development Policy sharing Develop the area of policy templates and policy examples. Currently a small selection of policies are available from two trusts and whilst this area of NRIC is thought to be useful, criticism was received for the lack of transparency of the selection process for these policies and suggestions for the involvement of NRIC were made. The first approach is that NRIC could be encouraging the community to share their policies with one another using NRIC as a platform. Taking this approach the community can comment on policies in a review system i.e. the Amazon approach and the end result can be a selection of policies that have been refined by the community themselves (i.e. like a clearing house). The difficulty with this approach is encouraging people to share their policies in the first place and then resolving differences in opinion without moderators. However, if moderators for specific policies could be appointed from the community, perhaps in conjunction with the Infection Prevention Society this could help the community take ownership of this part of the library and also provide educational development opportunities in the form of CPD. The alternative approach is that NRIC only posts model policies i.e. policies that have been vetted by an organisation such as the IPS, however this approach is relying on such policies being developed and is not taking an active role in developing the guidance in the infection prevention and control domain. Training workshops Workshops could be run at conferences or meetings to help users learn how best to navigate NRIC and find information. This would also provide a forum for regular feedback from users as to how NRIC is meeting their needs and how it can be developed or improved. It would also increase awareness of NRIC and provide a means for recruiting members of the community to contribute to the library. A simple user guide could also be produced to accompany these workshops and also be available on the website for all users. Raising awareness Awareness of NRIC is still not high and further work is needed to embed this resource in the user community. More links could be made with educational establishments to increase the awareness of NRIC among students studying infection prevention and control courses. Conference attendance has shown in the past to be a very effective way of increasing awareness and the presence of NRIC at relevant events could be beneficial. However promotion to less experienced members of the community who may not be attending conferences is also key therefore posters and user guides could be distributed to course providers, hospitals and other relevant organisations. Changing order of search results The search results table needs changing so that either results are ordered by date or ideally so the user can sort results by whatever category they choose. This is of increasing importance as NRIC grows and the list of results becomes unmanageable. Extending educational remit The training and education section of NRIC (Training in Infection) requires updating and better integration into the NRIC website. Fostering relationships with course providers could enable this to

be improved and ensure any development is meeting the needs of students. Student representatives could also be invited to join the Advisory Board. For those professionals using NRIC for developing training materials a clearing house similar to that suggested for the policy templates could be developed for professionals to share educational material they create.

Conclusion This report has shown that NRIC is a popular, easy to use, library that is having a positive impact on user knowledge and work. However, further sustained investment is required if NRIC is to fulfil its potential as a one-stop resource in the infection prevention and control community. Raising awareness is key to encouraging the involvement of this community in developing NRIC and ensuring that any development is in-line with user needs.

Appendix 1 Impact-ED Model Evaluation Criteria 1. Community: 1.1. How the digital library is used in the work setting and what are the reasons for use 1.2. The suitability of the work environment for integrating the digital library into work practice e.g. access to the Internet 1.3. Acceptability of the digital library by employers/colleagues i.e. support given for using the digital library in work time & space 1.4. Awareness of the digital library in the target community, not just those who use it already 1.5. Basic demographic information of users 2. Services: 2.1. User satisfaction with the digital library & how it meets user needs 2.2. Preferred resources/services already offered by the digital library 2.3. Gaps in provision 2.4. Awareness of current digital library services and content to existing users 3. Technology: 3.1. Basic web access log statistics to see frequency of use and patterns of use 3.2. Usability of the digital library 3.3. Search query analysis i.e. how are people searching the digital library, for what topics/parameters and are they finding what they are looking for 3.4. navigation pathway analysis to identify how users are navigating the digital library and what services are commonly used 4. Content: 4.1. knowledge and attitude changes as a result of using the digital library e.g. actual or self-reported 4.2. the impact of using the digital library on user work e.g. clinical decision-making and patient care, use in creating documents such as policies, guidelines or coursework 4.3. the dissemination of information found in the digital library to others