Patient Education– In the Department of Medicine

Primer for Evaluating Outcomes

By Kate Lorig, RN, DrPH, and Diana Laurent, MPH


Getting Started

There is no such thing as one questionnaire for measuring the outcomes of the CDSMP. When thinking of evaluation, every organization has to decide what is that they want to know. Before going further, let’s define some terms.

Identifying information — these are questions that ask name, address, phone number, email and sometime an alternative contact person with name and phone so you can find people if they move.

Demographic information — these are questions that ask age, education, marital status, gender, ethnicity etc.

Scale — this is one or more questions (items) that measure only one thing such as pain, self-efficacy or exercise. To build a scale or even test a single question takes lots of work — in fact there is a whole field call psychometrics devoted to building and testing scales. This means that you cannot change anything on a tested item or scale.

Questionnaire — this is made up of identifying information, demographic information and a series of scales. The problem is that there is an unending list of possible questions and outcomes, so there has to be some way of sorting these out. Thus, you might want to start by asking yourself these questions.

  1. What do you want to know? Then for each answer ask why is this important? If you cannot give a really good answer as to importance then this is probably not something you need to evaluate. For example, you may want to know about ethnicity but do you need to know if someone is of Puerto Rican, Mexican, Cuban, or Central American origin, or is Hispanic/Latino enough? Sometimes, this is very important and sometimes not. This has to be decided at the local level.
  2. Are there data that are required by the funder or by your agency? If so you will need to include these in your evaluation.
  3. Are there good scales for collecting the data that you want? Most evaluators do not have the time or resources to create new evaluation scales.
  4. How many questions will your participants tolerate? There are many good evaluation scales that are very long. There is always a trade off between what you want to know and what participants are willing to complete.
  5. Are the scales that you are using sensitive to the type of change you are expecting? For example a weather thermometer is a fine measuring tool but is not very useful for finding a fever in a human. One or two degrees or even a half a degree makes a big difference when measuring body temperature, but not a lot of difference when deciding whether or not to wear a sweater.
  6. Do you have adequate data collection resources? This means collecting data, coding it, getting it into some computer system, and most importantly following up immediately after receipt on missing data and data that looks funny. For example, the person who says he/she was hospitalized five times but reports only three days in the hospital, or someone who says they exercise for 180 minutes per day. Remember that you have to go through this whole process with every person for each follow-up. However, here it gets more complicated as you may have to contact a person several times to actually collect the follow-up data. We usually send the questionnaire by mail, then send a postcard reminder ten days later, call ten days after that and send a second questionnaire ten days after that. You may not want such a complex system, but if you are not collecting 75% or more of the data at post test, then the data you have is not very useful.

Suggested Outcome Measures

At Stanford, we usually think of outcomes in four categories; health behaviors, health status, health care utilization and self-efficacy.

The following are our favorite scales and you might want to consider them. They are certainly not the only scales and the bottom line is that the choice is yours. Unless otherwise noted the scales, their psychometric properties, and scaling are on our website at the following link: http://patienteducation.stanford.edu/research/ Also please note that many of the scales are also available in Spanish.

Health Behaviors

Health Status

Health Care Utilization

These questions are very important as it is this data that causes policy change. It is also the data that can be used to compare costs to effects. We suggest that you use four measures.

While it would be really nice to know about visits to nurse practitioner, chiropractors, etc., the public just cannot differentiate all these different people, and so after long hard experience we just ask about MD visits.

We ask about nights in hospital rather than days because people often confuse a day surgery with a hospitalization. If they were there overnight then they really were hospitalized.

This is certainly not a complete list of utilization but does give an idea of what is happening. These questions should always be asked based on the time between questionnaires. Thus, if you have a baseline and a six month questionnaire, you ask about visits in the past six months. If you have a baseline and 4 month questionnaire you ask about visits in the past 4 months. You need to be sure that the timing on these questionnaires is the same for all questionnaires and that the visits do not overlap. That is you cannot ask about visits in the past four months before the class and then at the end of the class again ask about visits in the past four months because only 2 months will have past since you asked the question the last time.

Self-efficacy

Chronic Disease Self-efficacy Scale (6 items). Please note that self-efficacy is a weak outcome. No one wants to go to their state legislature and say this is a great program because it increases self-efficacy. If this is the only outcome you get, then there is not much there. Self-efficacy is important as it predicts changes in health behaviors and health status. The people who tend to do best in the program as those who either have pretty good efficacy to start with or who improve their self-efficacy with the workshop.

Formatting Questionnaires

There is an art to formatting questionnaires, which comes from coding thousands of questionnaires and seeing response patterns emerge from our mistakes. Here are the suggestions that we have learned over the years.

Paper

Type Size and Style

Spacing

Organizing the Scales

Coding Bars

If you use Word to create your questionnaire, you may find yourself swearing at your computer when it comes to the coding bars. The coding bars are the columns on the outside of each page where you have the variable names and a line to record the codes for each question in preparation for data entry. We’ve created coding bars two ways: either by creating columns in Word, or text boxes. The text boxes seem to work the best. Columns are a nightmare because they have to switch from the right side on odd numbered pages to the left side on even numbered pages. If you change something in the slightest, the columns are messed up.

Text boxes can be created and the coding column created inside it. They can be resized more easily. If you size and place the text box so that three of the sides are outside the margins (and won’t print), you will have a ready-made line defining the coding column on the one side that’s left in the printable area. They tend to jump around if you make changes in your questionnaire, which is irritating, but it’s easier than columns.

Coding bars can be formatted in these ways:

Footers

 

This article is available as a PDF download: primer.pdf

 

 

Footer Links: