The Statistics RRTC conducted an online survey to explore data quality issues related to the new RSA-911 case services report form. Specifically, we were interested in examining the variations between agencies in using VR service categories (data elements 51-190). The online survey was distributed through the VR program evaluator’s network.
A total of 43 complete responses were received from 25 states (based on 50% of the respondents who chose to identify their state) and respondents identified themselves as data managers (39%) Program evaluator (39%), decision maker (25%), agency manager (30%) and other (18.8%).
Most agencies used state-specific software for case management and reporting (42%). Approximately 35% of the agencies used Aware and another 12% were using Libera.
About 72% of the respondents indicated that they were somewhat satisfied with the quality of data reported using the new service categories of the RSA-911; 20% were highly satisfied and 8% were not satisfied.
When asked about any concerns they may have about the quality of data reported using the RSA 911 service categories, 32% of the respondents said that they had no concerns about the quality of the data, while the other 68% had some very specific concerns. The most common issues raised were related to the clarity in the new service categories, especially rehabilitation technology, training, transition, technical assistance and assessment. A few respondents raised concerns about the quality of data reported since the new definitions were implemented mid-year. Respondents pointed out that it is likely that the data quality may be compromised due to conversion of old expenditure codes and the need to estimate costs for services provided in addition to services purchased.
Some of the concerns raised by respondents are highlighted below:
– Challenging to consistently and accurately capture no cost services such as counseling and guidance or newly created services within the agency
– “Transition” is being identified as a specific service. Transition services include a full range of allowable services including job placement, supported employment, training etc.
– Data elements are not capable of keeping up with the reality of rapidly changing and developing technologies
– Service categories are more aligned with the billing units, but do not support outcome based evaluation, like how RSA wants
– Difficult to distinguish customized employment from other job placement activity for supported employment
– The data reported is not always accurate and/or tracked in a manner that is consistent with actual job placements and providing supports in a timely manner (i.e. transportation, clothing).
– Concerns we have are all of the additional fields required concerning comparable benefits for and cost for each individual service.
– Since these are 911 elements represent services provided on closed cases, and since many of those cases will have had services provided and categorized before the revision went into effect, the data may represent ‘mixed’ categories, which could result in diminished data quality.
– There appears to be such specific guidance on how to code certain type of services that when a service that we can provide doesn’t fit into one of those categories it’s hard to know how to code it. Some of the guidance for what type of assessment services seems to be vague
– The definition of rehabilitation technology including hearing aids and wheelchairs demonstrates a clear misunderstanding of what rehabilitation technology is.
– I have concerns about the way we respond to the provider/purchaser of the services. The new provider categories/comparable benefit providers doesn’t necessarily capture the entities who help pay for services – financial aid, medical insurance.
– The data does not reflect the amount of services provided. While total dollars of purchased services allow for an aggregate examination, without comparable agency level data that is classified by type of services it is difficult to paint an accurate picture of service delivery.
– The quality of data submitted over the next 3-4 years will not be the most accurate due to conversion of the old expenditure codes.
– The Breakdown of Specifics of Training Facilities (Ex. Public, Private) has created some problems in our Case Management System.
– Questions for the new taxonomy need to be developed using a hierarchical structure and the categories should be mutually exclusive. For example, Assessment (51-55) includes psychological assessments, dental, and medical exams, which may also be categorized under Diagnosis and Treatment of Impairments (56-60). The hierarchical structure used for Training and Personal Assistance Services, each with several subcategories might be extended to other types of services, such as Diagnosis and Treatment of Impairments, Transportation, Maintenance, and Rehabilitation Technology to provide more granularity. The categories for the new taxonomy could be developed through axial coding for these dimensions.
Posted in: RSA 911