Pathways to Performance: Making a Difference through Program Evaluation
PROCESSES TO SYSTEM:
QUALITY ASSURANCE (QA) IN STATE VOCATIONAL REHABILITATION AGENCIES
REHABILITATION SERVICES ADMINISTRATION
OFFICE OF SPECIAL EDUCATION AND REHABILITATION SERVICES
U.S. DEPARTMENT OF EDUCATION
WEDNESDAY, SEPT. 22, 2010
Today’s Presentation
FY 2008 QA Pilot Information—Janette Shell
participants, providers and themes
Summary of the targeted evaluations -VR agency QA practices
Information from 107 Monitoring—Brian Miller
QA observations, recommendations, findings
what RSA looks at and looks for when reviewing QA
Components of a QA System—Terry Martin
identification and management of linked activities

QA Pilot FY 2008
Participating Agencies: 
Alabama Department of Rehabilitation Services;
Nebraska Division of Rehabilitation Services;
New York Office of Vocational and Educational Services for Individuals with Disabilities;
New York Commission for the Blind and Visually Handicapped; and
Wyoming Division of Vocational Rehabilitation.

QA Pilot FY 2008
Participants chose technical assistance from a variety of providers:
one in the field of QA—no vocational rehabilitation (VR) experience
one with extensive experience in VR
one with research and business background in QA

QA Pilot FY 2008
Pilot Technical Assistance (TA) providers: 
American Society for Quality—Alabama and Nebraska;
Regional Rehabilitation Continuing Education Program (RRCEP) Region 1—New York; and
Utah State University and the National Clearinghouse of Rehabilitation Training Materials—Wyoming.

QA Pilot FY 2008
TA providers were charged to: 
provide expertise in QA;
utilize an individualized approach;
assess existing QA processes;
develop and implement action plans to integrate/align systems; and 
evaluate the impact of the initiative.
QA Pilot FY 2008
Themes in changes made by agencies: 
looking at QA in a different way; 
use of available data; 
evaluating community rehabilitation programs (CRPs); 
communication strategies; 
educating staff on their role in QA; and
improved consumer satisfaction information.

QA Pilot FY 2008
Themes from follow-up calls: 
broader view of QA; 
consensus on what quality means; 
improved communication between staff, partners and consumers; 
improved tracking of quality indicators; 
difficult to assess impact of pilot. 

Unpublished QA Study
Targeted Evaluations of State VR Agency Practices
Quality Assurance
Barbara Elliott, Michael Tashjian, Becky Hayward, and Lita Jans
InfoUse and RTI International

STUDY QUESTIONS
Question 1
To what extend do agencies perform QA and other functions to ensure that services are delivered in accordance with statutes and regulations and in keeping with agencies’ policies and practices?

Number of Agencies
Reporting Quality Assurance Activities 

Number of Agencies Reporting Quality Assurance Activities (cont’d.)

Breakdown of QA Practices
Further analysis of:
methods for sampling files for review;
features of case management systems;
methods for obtaining consumer input;
agency requirements for vendors;
methods for monitoring vendor performance;
topics of special studies; and
uses of performance data.

Uses of Performance Data

Study Questions
Question 2
What effects or outcomes are attributable to QA and/or similar program functions?

Number of Agencies Identifying Areas for Improvement Through Various Methods

Effects and Outcomes
Improvements in:
organizational culture;
communication with staff and stakeholders;
business processes;
case work and case services; and
availability and use of data
Study Questions

Question 3
To the extent it is possible to determine from this evaluation, what practices are effective?

Effective Practices
multiple quality assurance activities focused on the same goal;
case file review that emphasizes training and skill building;
case management systems that provide staff at all levels with access to real-time data;
timely feedback from consumers;
better methods of communicating with, and obtaining input from, staff; and
staff involvement in quality assurance activities

Study Questions
Question 4
How well are QA functions performed?

Areas for Improvement
development of comprehensive quality assurance “systems”;
case management systems;
availability and use of data;
timeliness and comprehensiveness of consumer satisfaction data;
quality of case work and services; and
staff involvement in quality assurance activities

QA and the RSA Monitoring Process
Questions:
What is RSA looking for when reviewing QA?
What tools does RSA use to review QA?
What has RSA seen in the way of QA in the course of its reviews over the past four years?
How is QA addressed in the RSA monitoring report?
What kinds of QA recommendations or findings have RSA’s reviews generated?
QA and the RSA Monitoring Process
Monitoring history: 
service record reviews
compliance oriented
Case Service Record (CSR) instruments still in use today

QA and the RSA Monitoring Process
What we’re looking for today:
Move to systems approach.
Link QA to performance.
What does QA mean to the agency?
What processes are in place?

QA and the RSA Monitoring Process
What we’re looking for today (continued):
Who is involved with QA?
What actions result from QA processes?
To what degree are the results of QA processes shared with staff members?
How well are program and fiscal QA processes integrated?

QA and the RSA Monitoring Process
What tools does RSA use to review QA?
agency organizational charts
face-to-face interviews
QA discussion guides
internal QA reports—fiscal and program
reports to RSA:
Client Assistance Program (CAP) 
Protection and Advocacy (P&A) 
State Rehabilitation Council (SRC) 
hearings and appeals 
data

QA and the RSA Monitoring Process
What has RSA seen in the way of QA during its reviews over the past four years?
36	of 58 agencies that were reviewed in FY 2007-FY 2009 have QA observations
QA is rarely seen as a system, but a set of procedures or activities
QA activities not defined as QA 
compliance not performance based
QA is seen as task of one person

QA and the RSA Monitoring Process
What has RSA seen in the way of QA during its reviews over the past four years (continued)?
lack of integration between program and fiscal operations
consumer input limited to customer satisfaction surveys with low return rates
consumer satisfaction surveys limited in scope
service record review centric
VR staff members often see QA as punitive because results are linked to staff member performance evaluations

QA and the RSA Monitoring Process
What has RSA seen in the way of QA during its reviews over the past four years (continued)?
VR staff members and managers are often wary of data-driven QA processes, more comfortable with qualitative approaches
performance is usually limited to quantity measures – employment outcomes
RSA standards and indicators dominate performance discussions
data collection is limited by antiquated case management systems
QA of CRPs is very limited

QA and the RSA Monitoring Process
How is QA addressed in the RSA monitoring report?
executive summary including strengths and challenges
promising practices 
technical assistance (received and requested)
VR chapter on service delivery system, QA, data management, and planning

QA and the RSA Monitoring Process
How is QA addressed in the 
RSA monitoring report (continued)?
observations and recommendations
findings are not likely
Independent Living (IL) and Older Individuals Who Are Blind (OIB) chapters
fiscal chapter

QA and the RSA Monitoring Process
What kinds of QA recommendations or findings have RSA’s reviews generated?
Ex. RSA recommends that:
Review and refine QA instruments for conducting reviews;
Incorporate new QA instruments into agency goals and priorities;

QA and the RSA Monitoring Process
What kinds of QA recommendations or findings have RSA’s reviews generated (continued)?
Example: RSA recommends that:
develop mechanisms to collect and aggregate the results of the QA process and utilize them to inform the design and evaluation of training, service delivery and strategic planning;
enhance its QA system to incorporate the on-going monitoring of the quality and quantity of vendor performance;
develop and implement assessment criteria and performance measures for providers;

QA and the RSA Monitoring Process
What kinds of QA recommendations or findings have RSA’s reviews generated (continued)?
Example: RSA recommends that:
use results of CRP monitoring to identify training needs for the provider community to improve the quality of services to VR consumers;
develop a QA manual that identifies and describes the various QA processes employed by the agency to facilitate the creation of a QA system rather than just a set of processes; and
ensure that QA activities involve all staff, and emphasizes program improvement rather than personal performance and compliance.

QA and the RSA Monitoring Process
Future monitoring:
RSA is currently reviewing its monitoring protocol and processes to support program improvement.
RSA will solicit input from stakeholders.
Process Approach
Monitoring and Measurement

QA and the RSA Monitoring Process
Future monitoring (continued):
Customer Focus
Partner Relationships
Document Control
Continuous Improvement

Process Approach
“For an organization to function effectively, it has to identify and manage numerous linked activities. An activity using resources and managed in order to enable the transformation of inputs into outputs can be considered as a process…

Process Approach
“…The application of a system of processes within an organization, together with the identification and interactions of these processes and their management, is referred to as the process approach.”
Peterson, A.J. (2003). Jump start your process approach. 
Fairfax, VA: QSU Publishing Company.

Monitoring and Measurement
Defined requirements—specifications and standards for the output of each process;
verification process and instrumentation; and
use of results.

Customer Focus
Meet customer requirements and strive to exceed customer expectations;
research and understand customer needs and expectations;
communicate customer needs and expectations throughout the organization; and 
measure customer satisfaction and act on the results.
(www.iso.org)

Partner Relationships
Communication of requirements—specifications and standards;
evaluation and monitoring—verification of standards and specifications; 
quality of personnel; and
qualification of facilities, equipment, and processes.

Document Control
Prevent unintended use and safeguard confidentiality; and
ensure access to most current information.

Continuous Improvement
Results from monitoring processes:
Who collects data?
Who accesses results?

QA Summary
“An organization can never know what it thinks or what it wants until it sees what it does”
Weick, K.E. (1977). Re-punctuating the problem. (p. 193) 
In P. S. Goodman, J. M. Pennings, and Associates. New perspectives on organizational effectiveness. San Francisco, CA: Jossey-Bass Publishers.

Resources
National Clearinghouse of Rehabilitation Training Materials (NCRTM)—Utah State University
http://ncrtm.org
Community of Practice—NCRTM
Technical Assistance and Continuing Education (TACE) Centers
Research and Technical Assistance Center (RTAC) VR Management 

Conclusion
Questions?