Introduction and Background
Baldwin Research
Institute, Inc., (BRI) contracted with Clearwater Research, Inc.,
(Clearwater) to conduct a follow-up survey of a random sample of Jude
Thaddeus Program™ graduates. Clearwater conducted the survey and provided the
required products and services including sample design, questionnaire
consultation, data collection, and analysis.
Methods
Clearwater conducted a computer-assisted telephone
interview (CATI) survey of a random sample of 204 participants in the Jude
Thaddeus Program™ and one corroborating friend or family member for each
participant. Data were collected between the dates of May 31 and June 18,
2006. We provided BRI with assistance to adapt the questionnaire for CATI
administration, a pretest of the programmed CATI questionnaire, data
collection, data set preparation, and analysis. At the end of the project,
we delivered a cleaned data set, a report documenting the survey design and
implementation, frequency tables of the survey responses with statistical
analysis, and other deliverables required by the RFP.
Survey Instrument
Clearwater collaborated
with BRI to adapt the existing interview protocol for use with our CATI
system. The questionnaire was very short, essentially asking one question
about the sobriety status of the program participant. We provided our
standard review of the questionnaire to assess possible issues with question
wording, item order, and the flow of the survey from start to finish. Our
bid assumed that the average interview length would not exceed one to two
minutes per respondent, would not include any open-ended items, and would be
conducted in English only.
We tested the
questionnaire programming thoroughly to ensure accuracy in data collection.
Clearwater data collection and research staff closely monitored the
interviewers initially to identify any issues that eluded detection during
the testing. The final questionnaire is presented in Appendix A.
The average interview
length was approximately 9½ minutes per case. This includes all of the time
spent by an interviewer getting the question answered by both the guest and
the corroborator. It exceeded our original estimate of two minutes per
interviewee due to the amount of time spent contacting the individuals being
called, the extra comments they wished to provide, and Supervisor
discussion. The actual interview was much shorter in length.
Sampling
Clearwater consulted
with BRI regarding the specific information needed for fielding. The
Sobriety Survey Protocol provided by BRI originally specified the random
sample of participants be generated manually (physically drawing cases from
a container.) However, after consultation it was mutually decided we would
use Access to randomly choose names from a list provided to Clearwater by
BRI. After the names were selected, Clearwater accessed BRI’s in-house
database containing contact information for each participant in the Jude
Thaddeus Program™ to gather specific information that was copied manually into
a spreadsheet. The contact information was loaded into CATI and the
interviewers began to contact designated individuals and complete
interviews.
Out of a total of 475 records called, 204 resulted in
completed interviews with both parties giving an overall response rate of
42.95%. Clearwater interviewers “chased” members of the sample who were not
reachable at the household or telephone number provided by BRI when someone
we contacted could provide us with an updated telephone number.
Data Collection
For the Sobriety
Follow-up project, Clearwater adhered to the instructions in the Sobriety
Survey Protocol provided by BRI with the request for proposals. Data
collection was originally scheduled for May 30–June 18, 2006, but was
extended through June 25, 2006. Clearwater collected data using our
120-station computer-aided telephone interview (CATI) system. Interviewers
were able to see and record responses to questions on a computer screen. The
software managed the telephone calling, controlled distribution of sample,
consolidated data, and tracked interviewer activity and productivity.
Interviewers were thoroughly briefed prior to data collection and rehearsed
the questionnaire before conducting actual interviews.
After the first week
of calling, interviewers were briefed a second time immediately following
changes in the programming to make the survey flow better. Monitoring staff
listened to a sampling of interviews throughout the fielding period to
maintain data quality. Hard copies of monitoring reports are included with
the final deliverables.
Clearwater used
computer-aided dialing, but not predictive dialing. Predictive
dialing has the potential to annoy respondents by introducing a delay in
connections after respondents answer the telephone. This delay leads to
higher hang-up and refusal rates and a correspondingly lower response rate
for the survey.
Interviewers attempted
each record a minimum of three times or until a final disposition (e.g.,
completed interview, refusal) was reached. All definite appointments were
attempted regardless of the attempt number. Some records were attempted up
to 14 times with an average of 5½ attempts per guest/corroborator pair to
locate the correct individuals and make contact with them. To maximize the
likelihood of reaching the sampled person, interviewers called each
telephone number at a variety of times, including weekdays, weekday
evenings, and weekends. Each attempt represents a different calling time,
but could include dialing several different phone numbers. Detailed
descriptions of each attempt can be found in the attached “Attempt File.”
(See Appendix F for a description of the layout.)
Our interviewers are
trained in techniques of refusal avoidance that are effective in maximizing
response rates. We developed an interviewer manual for the Sobriety
Follow-up study that provided the interviewer with information about the
study to be used while relating with respondents to keep them on the phone.
Answers to frequently asked questions and guidelines for dealing with
certain situations were available during the interviews on hotkeys listed in
the interviewer manual. The interviewer manual is presented in Appendix B.
The text of the scripts are in the questionnaire document in Appendix A.
Data Preparation
During data collection,
interviewers documented any errors they encountered on data change forms. At
the conclusion of data collection, these errors were corrected in preparing
the data set for analysis. The data were converted and formatted for review
in SPSS (a statistical analysis software package) and Microsoft Access.
Clearwater pays close
attention to comprehensive and routine data inspections, data cleaning, and
data set preparation. The completed survey data set was cleaned and prepared
in an electronic file format in SPSS and Excel. Any data changes were
documented on data change forms which are included with the final
deliverables. For table production and statistical analysis, variable and
value labels were added to the cleaned SPSS data set.
Calling on the RDD
sample was considered complete at the point all records had reached their
minimum attempts or a final disposition had been assigned. We exported the
data from CATI, cleaned, and compiled them into a labeled SPSS datafile.
Clearwater calculated
the final disposition for each sample record. The CATI call history tables
recorded the interim or final disposition entered by the interviewer for
each call attempt. Database programming referenced both the CATI call
history tables and the data sets to determine the correct final disposition.
Call Outcomes
Each call attempt was
given an interim disposition depending on the outcome of the call. At the
end of the field period, each record was assigned a final disposition for
the study based on the history of interim dispositions and data collected
for that case. Because there were multiple phone numbers on each record, the
interim dispositions only captured the results of the last phone number
attempted, so additional notes were made.
Table 1 shows the
calculated final dispositions. Detailed descriptions of each disposition can
be found in the interviewer manual in Appendix B, Table 2. The final
dispositions shown represent the overall outcome of a guest/corroborator
pair. Some individual refusals may be hidden within the partial complete
counts.
The
higher corroborator response rate reflects both a greater availability
and willingness to answer among the corroborators than the guests.
The formulas for the response rates
were designed specifically for this study, but were based upon the
widely accepted response rate formulas published by the American
Association of Public Opinion Research.
The following formulas were applied to each column individually:
Discussion
The success rate based on the answers
of both the guest and corroborator is 53.2%. The success rate based on
just the guest answer is 61.8%. When both the guest and the corroborator
were reached, 107 agreed the guest was sober and 94 said they were not.
Based on just the guests’ response, 144 guests said they were sober,
while 89 said they were not. According to the corroborators’ responses,
188 were sober and 116 guests were not.
Considering both guest and
corroborator answers, the true percentage of all guests who are sober is
95% likely to lie in the range of 47.1% to 59.4%. More than six out of
ten guests (61.8%) indicated they are currently sober. Based upon their
answers alone, the true percentage of all guests who are sober is 95%
likely to lie in the range of 56.4% to 67.2%, as shown in Table 3.
Considering only the responses from corroborators, 61.8% of guests are
currently sober. The true percentage of guests who are actually sober
according to the corroborator is 95% likely to lie in the range of 57.3%
to 66.4%.
