Dependency Drug Court: Evaluation Report
Dependency Drug Court: Evaluation Report
Dependency Drug Court: Evaluation Report
Evaluation Report
December 2012
Table of Contents
Overview ................................................................................................................................................ 1 Collaborators and MIS Barriers ................................................................................................................................... 1 Changes to the Superior Court System ....................................................................................................................... 2 Systemic Data Barriers................................................................................................................................................... 3 Descriptive Analysis............................................................................................................................... 4 Demographics ................................................................................................................................................................. 4 Client Satisfaction ........................................................................................................................................................... 4 Overall Satisfaction.................................................................................................................................................... 4 Service Use and Helpfulness .................................................................................................................................... 8 Satisfaction Survey Summary ................................................................................................................................. 10 Lessons Learned & Recommendations................................................................................................ 11 Potential Data Collection Strategies .......................................................................................................................... 11 Potential Research Questions ..................................................................................................................................... 12 Appendix A: Client Satisfaction Form ..................................................................................................14
Overview
The San Francisco Superior Court engaged Davis Ja and Associates, Inc. (DYJA) to implement the outcome evaluation for the Dependency Drug Court (DDC) project. DYJA reviewed the initial work plan, the goals and objectives of the project and assigned evaluation staff to work with the Superior Court on the evaluation. Under this work plan, DYJA staff was responsible for developing consent forms, and data collection protocols and assumed some of the data entry procedures for specific measures. DYJA was also responsible for determining satisfaction through client surveys, conducting data analysis and compiling interim reports and a final evaluation report. DYJA staff worked with the program staff to establish consent procedures, assessment protocols and procedures and to utilize existing measures that would capture the data necessary to answer the evaluation and research questions indicated in the goals and objectives. However, in reviewing the process to capture adequate data necessary to fulfill goals and objectives, DYJA determined that the existing database and management information systems (MIS) of the Superior Court was inadequate and could not provide the data necessary. This was due in large part to the fact that the Superior Court system was never intended to capture outcome data, but to document court proceedings, charges and judgments. Attempts were made to revise and make changes, but authorization for these changes was difficult to obtain. Ultimately, the most significant change was the inclusion of the Addiction Severity Index (ASI), an outcome measure used by members of the collaborators of this project.
Although HPP routinely utilizes the ASI for their intake information, the data was primarily used to determine clinical needs and services. In addition, much of the services that the HPP provided for the Superior Court was case management which meant providing linkages for those referred by the court to services with HSA and also other departments of the City and County of San Francisco such as mental health services, community services and Department of Public Health Services (DPH). Unfortunately, HPP was unable to collect much data over time or to collect additional assessments of the clients in order to determine changes in outcome. Furthermore, the existing ASI were never entered into their database given their existing use as a clinical tool. Instead, the collected ASI data were entered into the DDC specific MS Access database strictly for evaluation purposes. HPP did not have a system that adequately collected ASI data and also did not have the resources necessary to conduct ongoing ASI assessments as well as enter them into a database. Although HPP provided excellent services (see satisfaction data analysis), their database was not a priority and had minimal dedicated resources for this specific project. DYJA worked with Superior Court staff to adapt the existing DDC-specific database from Microsoft Access that would be able to capture the information necessary to respond to the performance measures. But in order to enter the data needed, the measures and data collection process from HSA, DPH and HPP would have to be provided to the Superior Court in a timely and ongoing fashion. Given the leave of absence of the key leadership role on the project, especially as the project began implementation, it became increasingly difficult to coordinate and ensure changes in database development and data collection strategies from all parties involved with the project.
returned, her responsibilities remained the same but with tremendously reduced resources. There was little time or resources that could be devoted to the DDC evaluation and measurement issues and concerns.
Descriptive Analysis
Demographics
DDC served a total of 83 clients over 24-month period between August 2010 and June 2012. Sixty females and 23 males were served. The average age was 36, with ages ranging from 21-60. The average age for females was 34, and the average age for males was 41. The clients served were 45% Black (37 clients), 28% White (23), 18% Hispanic (15), 5% Asian or Pacific Islander (4) and 2% American Indian (2) with 2% (2) not reporting their ethnicity. The average amount of education completed by the clients served was 11 years, with amounts ranging from 0-14 years. Forty-three percent of clients (36) reported having some form of employment over the past three years, 43% of clients (36) reported being unemployed, 4% (3) reported being students, 2% (2) reported being retired, 1% (1) reported being in a controlled environment, with 6% (5) not reporting. Sixty-one percent of clients (51) reported having some form of income, with the average monthly income for clients being $662.27. Thirty-six percent of clients (30) reported having no source of income and 2% (2) did not report. Sixty-one percent of clients (51) reported being incarcerated at some point in their lives, with the average length of incarceration being 40 months. Six percent (5) of clients reported being detained or incarcerated during the past 30 days with an average of 11 days spent incarcerated.
To the question, Do you feel that staff respects all clients in the program, regardless of ethnic, gender, sexual orientation, and lifestyle differences? 52% (12) responded Greatly, 35% (8) responded Very Much, 9% (2) responded Somewhat, and 4% (1) responded A Little.
Do you feel that staff respects all clients in the program, regardless of ethnic, gender, sexual orientation and lifestyle differences?
Response Not At All A little Somewhat Very Much Greatly Total Frequency 0 1 2 8 12 23 Percent 0% 4% 9% 35% 52% 100%
Client Response Frequency 14 12 10 8 6 4 2 0 Not At All A little Somewhat Very Much Greatly
Clients were asked: How well do you feel staff understands your needs and concerns?. Forty-three percent (10) responded Greatly, 22% (5) responded Very Much, 26% (6) responded Somewhat, 4% (1) responded A Little, and 4% (1) responded Not at All.
How well do you feel staff understands your needs and concerns?
Response Not At All A little Somewhat Very Much Greatly Total Frequency 1 1 6 5 10 23 Percent 4% 4% 26% 22% 44% 100%
Client Response Frequency 12 10 8 6 4 2 0 Not At All A little Somewhat Very Much Greatly
To the question, In general, how well do you feel the staff has helped you meet your needs? 48% (11) of clients responded Greatly, 13% (3) responded Very Much, 30% (7) responded Somewhat, 4% (1) responded A Little, and 4% (1) responded Not at All.
In general, how well do you feel staff has helped you meet your needs?
12
Frequency 1 1 6 5 10 23
Percent
Client Response Frequency
Clients were asked the question: Do you feel that your quality of life has improved since being in DDC?. Forty-eight percent responded Greatly, 17% (4) responded Very Much, 22% (5) responded Somewhat, 4% (1) responded A Little, and 9% (2) responded Not at All.
Do you feel that your quality of life has improved since being in DDC?
12
Frequency 2 1 5 4 11 23
Percent
Client Response Frequency
When asked, Do you feel that working with the court program has helped you with your own recovery? 52% of clients (12) responded, Greatly, 13% (3) responded Very Much, 26% (6) responded Somewhat, 4% (1) responded A Little, and 4% (1) responded Not at All.
Do you feel that working with the court program has helped you with your own recovery?
Response Not At All A little Somewhat Very Much Greatly Total Frequency 1 1 6 3 12 23 Percent
Client Response Frequency 14 12 10 8 6 4 2 0 Not At All A little Somewhat Very Much Greatly
To the question, In general, how well do you feel that the court program has helped you address your own needs? 30% of clients (7) responded Greatly, 26% (6) responded Very Much, 22% (5) responded Somewhat, 17% (4) responded A Little, and 4% (1) responded Not at All.
In general, how well do you feel the court program has helped you address your own needs?
Response Not At All A little Somewhat Very Much Greatly Total Frequency 1 4 5 6 7 23 Percent 4% 17% 22% 26% 30% 100%
Client Response Frequency 8 7 6 5 4 3 2 1 0 Not At All A little Somewhat Very Much Greatly
clients (8) reported using court dates monthly. Forty-four percent of clients (10) reported court dates as being Greatly helpful, 26% of clients (6) reported them as Very Much helpful, 17% of clients (4) reported them as Somewhat helpful, 9% (2) reported them as A Little helpful, and 4% (1) reported them as Not at All helpful.
HPP Case Management Services: Fifty-seven percent of clients (13) reported using HPP Case Management
services weekly; 22% (5) reported using them every two weeks; 13% (3) reported using them monthly; 4% (1) reported using them less than once a month, while 4% of clients (1) reported never using the services. Of those clients who used HPP Case Management services at least once, 59% (13) reported them as Greatly helpful, 23% of clients (5) reported them as Very Much helpful, 9% of clients (2) reported th em as Somewhat helpful, 5% (1) reported them as A Little helpful, and 5% (1) reported them as Not at All helpful.
Referrals for Drug Treatment: When clients were asked about referrals for drug treatment 4% of clients (1)
reported using the service daily; 4% (1) reported using it weekly; 9% (2) reported using it every two weeks; 4% (1) reported using it monthly; 30% (7) reported never using the service, while 39% of clients (9) selected Not Applicable. Of those clients who used referrals for drug treatment at least once, 53% (8) reported them as Greatly helpful, 7% of clients (1) reported them as Very Much helpful, 27% of clients (4) reported them as Somewhat helpful, 7% (1) reported them as A Little helpful, and 7% (1) reported them as Not at All helpful.
Referrals for Housing: When clients were asked about referrals for housing 9% of clients (2) reported using
the service weekly; 13% (3) reported using it monthly, 39% of clients (9) reported using it less than once a month; 13% (3) reported never using the service, while 26% of clients (6) selected Not Applicable. Of those clients who used referrals for housing at least once, 57% (8) reported them as Greatly helpful, 14% of clients (2) reported them as Very Much helpful, 14% of clients (2) reported them as Somewhat helpful, and 14% (2) reported them as A Little helpful.
Referrals for Therapy: Thirteen percent of clients (3) reported using referrals for therapy services weekly; 4%
(1) reported using it every two weeks; 4% (1) reported using it monthly, 17% reported using the service less than once a month; 22% (5) reported never using the service, while 39% of clients (9) selected Not Applicable. Of those clients who used referrals for therapy at least once, 46% (6) reported them as
Greatly helpful, 15% of clients (2) reported them as Very Much helpful, 8% of clients (1) reported them as Somewhat helpful, 15% (2) reported them as A Little helpful, and 15% (2) reported them as Not at All helpful.
Referrals for Other Services: When clients were asked about referrals for other services 13% of clients (3)
reported using the service weekly; 9% (2) reported using it every two weeks; 13% (3) reported using it monthly; 22% (5) reported using the service less than once a month; 4% (1) reported never using the service; while 39% of clients (9) selected Not Applicable. Data on this item was missing for one client. Of those clients who used referrals for other services at least once, 67% (8) reported them as Greatly helpful, 25% of clients (1) reported them as Very Much helpful, and 8% (1) reported them as Not at All helpful.
Emergency Needs:
Clients were asked about their use of services for emergency needs including food,
supplies, strollers, furniture or other supplies/support. Four percent of clients (1) reported using the service weekly; 4% (1) reported using it every two weeks; 30% (7) reported using it monthly; 26% reported using it less than once a month; 17% (4) reported never using the service, while 17% of clients (4) selected Not Applicable. Of those clients who used emergency needs support services at least once, 53% (8) reported them as Greatly helpful, 20% of clients (3) reported them as Very Much helpful, 20% of clients (3) reported them as Somewhat helpful, and 7% (1) reported them as A Little helpful.
Parenting Classes: When clients were asked about their use of parenting classes, 26% (6) reported using them
weekly; 4% reported using them every two weeks; 13% (3) reported using them monthly; 4% (1) reported using them less than once a month; 22% (7) reported never using them, while 30% of clients (7) selected Not Applicable. Of the clients who attended parenting classes at least once, 82% (9) reported them as Greatly helpful, 9% (1) reported them as A Little helpful, and 9% (1) reported them as Not at All helpful.
For those clients who have used Parenting Classes, how helpful were they?
90%
Frequency 1 1 0 0 9 11
Percent
Percentage of Clients
9% 9% 0% 0% 82% 100%
General Support Services to Increase Visitations: Four percent of clients (1) reported using general support
services to help increase visitations daily; 13% (3) reported using them weekly; 17% (4) reported using them every two weeks; 4% (1) reported using it monthly; 17% (4) reported using the service less than once a month; 17% (4) reported never using the service, while 26% of clients (6) selected Not Applicable. Of those clients who used general support services for increasing visitations at least once, 53% (8) reported them as Greatly helpful, 7% of clients (1) reported them as Very Much helpful, 27% of clients (4) reported them as Somewhat helpful, 7% (1) reported them as A Little helpful, and 7% (1) reported them as Not at All helpful.
Drug Treatment: When clients were asked about drug treatment services, 44% (10) reported using them
daily; 17% (4) reported using them weekly; 9% (2) reported using them every two weeks; 4% (1) reported using it monthly; 4% (1) reported using the service less than once a month; 9% (2) reported never using the service, while 13% of clients (3) selected Not Applicable. Of those clients who used drug treatment services at least once, 63% (12) reported them as Greatly helpful, 5% of clients (1) reported them as Very Much helpful, 21% of clients (4) reported them as Somewhat helpful, 5% (1) reported them as A Little helpful, and 5% (1) reported them as Not at All helpful.
Psychotherapy: Forty-four percent of clients (10) reported using psychotherapy services weekly; 13% (3)
reported using them every two weeks; 9% (2) reported using it monthly; 13% (3) reported never using the service, while 22% of clients (5) selected Not Applicable. Of those clients who used psychotherapy services at least once, 63% (10) reported them as Greatly helpful, 13% of clients (2) reported them as Very Much helpful, 13% of clients (2) reported them as A Little helpful, and 13% (2) reported them as Not at All helpful.
DDC Intake/Waitlist Group: Clients were asked about their use of the DDC Intake/Waitlist Group. Four
percent (1) reported using the group daily; 13% (3) reported using it weekly; 4% (1) reported using it less than once a month; 17% (4) reported never using the service, while 61% of clients (14) selected Not Applicable. Of those clients who used the DDC Intake/Waitlist Group at least once, 44% (4) reported it as Greatly helpful, 22% of clients (2) reported it as Very Much helpful, 11% of clients (1) reported it as A Little helpful, and 22% (2) reported them as Not at All helpful.
10
last six months, with many using services at higher frequencies. Of clients using services at least once, a majority reported for each of the services that they found them to be Greatly helpful.
11
For the Superior Court, the collaborative partners must have a shared understanding and interest in the evaluation. Another recommendation for future evaluations would be to tie funding to data collection for collaborating agencies to ensure that all data, particularly follow-up data, is collected in a timely manner. If resources and staff time becomes problematic for the intervention, whether case management, or treatment, than perhaps an outside evaluator needs to be involved in conducting all post baseline assessments. Incentives would be built into each assessment with the clients and the follow-up assessments and other data needs would be entered directly by the evaluators themselves. Although this may prove problematic in terms of evaluation expense, for programs that have difficulty in supporting or implementing outcome assessments, this may prove to be an option. One of the foreseeable barriers to this endeavor is the ease of sharing information. Even among the closest collaborators in this project, HPP and the Superior Court, data sharing proved to be a challenge. Ideally DDC would be able to readily access accurate data on an individual client basis. One suggestion would be to create or further refine DDCs Access database to track more in-depth information on each client. Perhaps a shared online database on a cloud system using a secured third-party database would avoid duplication of data entry and provide a shared file for tracking the client over time in terms of services as well as outcome.
How can the Superior Court better understand the outcomes of the clients they see and refer for services? How can the Superior Court better establish data sharing linkages in the future particularly given consent and confidentiality issues? How can the Superior Court insure that their collaborating partners collect adequate outcome data and make this a priority?
12
How can resources be increased to help refine and develop the Superior Court MIS to include outcomes for the people they see (i.e., reunification, etc.)? Can the Superior Court establish a protocol for all future collaborations particularly with other nonprofits and with other Departmental agencies? Do service referrals and frequent monitoring provided by the Superior Court result in better outcomes for families versus those statutorily mandated?
13
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
2. Do you feel that staff respects all clients in the program, regardless of ethnic, gender, sexual orientation, and lifestyle differences?
NOT AT ALL
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
3. How well do you feel staff understand your needs and concerns?
NOT AT ALL
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
4. In general, how well do you feel the staff has helped you meet your needs?
NOT AT ALL
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
5. Do you feel that your quality of life has improved since being in DDC?
NOT AT ALL
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
6. Do you feel that working with the court program has helped you with your own recovery?
NOT AT ALL
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
7. In general, how well do you feel that court program has helped you address your own needs?
NOT AT ALL
A LITTLE
SOMEWHAT
VERY MUCH
GREATLY
14
8. In the last 6 months, how often have you used the following services?
Court Dates
Never
Month -ly
Every 2 weeks 4
Weekly
Daily
N/A
Emergency Needs (i.e. food, supplies, strollers, furniture, other supplies/support) Parenting Classes
Psychotherapy
15
9. Of the services you have used in the last 6 months, how helpful were they? Court Dates HPP Case Management Referrals for Treatment Referrals for Housing Referrals for Therapy Referrals for other services Emergency Needs (i.e. food, supplies, strollers, furniture, other supplies/support) Parenting Classes General support to help increase visitations Drug Treatment Psychotherapy DDC Intake Group (the waitlist group) m. Other (please specify): _______________________________ _______________________________ _______________________________
Not at all 1 1 1 1 1 1 1
A little
Somewha t 3 3 3 3 3 3 3
Very much 4 4 4 4 4 4 4
Greatly
N/A
2 2 2 2 2 2 2
5 5 5 5 5 5 5
6 6 6 6 6 6 6
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 4 4 4 4 4
5 5 5 5 5 5
6 6 6 6 6 6
Thank you!
DDC Satisfaction Survey Final
16