— (06/12/2014) Choosing Good or Sound Evaluation Questions: We all know that evaluation can go far off track if the evaluation questions from the beginning are confusing, unclear, irrelevant, unrealistic or not sound. How do you or others in your organizations go about selecting good or sound evaluation questions? I know this includes judgment and a decision making process. I am very interested in how this judgment is made. Do you/your organization have examples of the judgment/decision-making process?

From Jennifer Beilke, MN State Services for the Blind:

The process will vary depending on the project. For an Adjustment to Blindness Customer Satisfaction Survey we started with the existing evaluation. Our state department did research with our vendors, customers, and staff to find out what was really important to them. We involved our council committee and council in the process for their input. The new questions were used in a trial run with our customers to test them out before going live.

From David Higginbotham, LA Rehabilitation Services:

We have three QA staff at this time. Each of the evaluation questions is reviewed on an ongoing basis or at least annually to assure that the item actually measures the standard for which it is intended. The text of the question must be approved by the three QA staff and the Assistant Director of LRS.

From Richard Clark, Iowa Vocational Rehabilitation Services:

In Iowa one of the ways we have grown in selecting questions for the evaluation is by bringing in a team of field staff to build the questions. This team would consist of the staff that are exceeding in the areas of the QA process. We present them with the information from the regulations and ask for feedback on how this would be handled in the field. Based on their responses we build the questions and how we would expect to see it in the case files. Then we take the questions out to the field to train the field staff on the process and questions that have been developed so that they understand what we are looking for and how it will be scored. From there we ask for feedback and tweak it as needed or further educate if the staff are just not understanding the questions. I will attach a copy of one of the forms we have built to review the quality of work with Transition. In this form you will see the questions we would ask in bold and examples of things we would look for in the file to determine if it occured or not. Not all of the examples would be required in each case but some of those activities should be occurring.

Attachments: 1

Posted in: Client Surveys