Introduction and Background: Baldwin Research Institute, Inc., (BRI) contracted with Clearwater Research, Inc., (Clearwater) to conduct a follow-up survey of a random sample of Jude Thaddeus® graduates. Clearwater provided sample design, questionnaire consultation, data collection, and analysis.
Methods: Clearwater conducted a computer-assisted telephone interview (CATI) survey of a random sample of 223 participants in the Jude Thaddeus Program® and one corroborating friend or family member for each participant. Data were collected between the dates of February 18 and April 2, 2008. We provided BRI with assistance to adapt the questionnaire for CATI administration, a pre-test of the programmed CATI questionnaire, data collection, data set preparation, and analysis. At the end of the project, we delivered a cleaned data set, a report documenting the survey design and implementation, frequency tables of the survey responses with statistical analysis, and other deliverables requires by the Request for Proposal (RFP.)
Survey Instrument: Clearwater collaborated with BRI to adapt the existing interview protocol for use with our CATI system. The questionnaire was very short, essentially asking one question about the sobriety status of the program participant. We provided our standard review of the questionnaire to assess possible issues with question wording, item order, and the flow of the survey from start to finish. Our Bid assumed that the average interview length would not exceed one to two minutes per respondent, would not include any open-ended items, and would be conducted in English only. We tested the questionnaire programming thoroughly to ensure accuracy in data collection. Clearwater data collection and research staff also closely monitored the interviewers during the fielding period to identify any issues that eluded detection during testing. The final questionnaire is presented in Appendix A*. The average interview length was approximately seven minutes per record. This includes all of the time spent by an interviewer getting the question answered by both the guest and the corroborator. This was less than our original estimate of ten minutes per record.
Sampling: Clearwater consulted with BRI regarding the specific information needed for fielding. Baldwin and Clearwater mutually agreed to have Clearwater use Microsoft Access to randomly choose names from a list provided to Clearwater by BRI. After names were selected , Clearwater accessed BRI’s in-house database containing contact information for each participant in the Jude Thaddeus Program to gather specific information that was copied manually into a spreadsheet. The contact information was loaded into our CATI system and the interviewers began to contact designated individuals and complete interviews. Out of a total of 956 records called, 232resulted in completed interviews with both parties, giving an overall response rate of 26.17%. Clearwater interviewers “chased” members of the sample who were not reachable at the household or telephone number provided by BRI when someone we contacted could provide us with an updated telephone number.
Data Collection: For the Sobriety Follow-up project, Clearwater adhered to the instructions in the Sobriety Survey Protocol provided by BRI with the RFP. Data collection was originally scheduled for February 18—March 17, 2008 but was expanded through April 2, 2008. Clearwater collected data using our 135-station computer-aided telephone interview (CATI) system. Interviewers were able to see and record responses to questions on a computer screen. The software managed the telephone calling, controlled distribution of sample, consolidated data, and tracked interviewer activity and productivity. Interviewers were thoroughly briefed prior to data collection and rehearsed the questionnaire before conducting actual interviews. After the first week of calling, interviewers were briefed a second time, immediately following changes in the programming to make the survey flow better. Monitoring staff listened to a sampling of interviews throughout the fielding period to maintain data quality. Hard copies of monitoring reports are included with the final deliverables. Clearwater used computer-aided dialing, but not predictive dialing. Predictive dialing has the potential to annoy respondents by introducing a delay in connections after respondents answer the telephone. This delay leads to higher hang-up and refusal rates and a correspondingly lower response rate for the survey. Interviewers attempted each record a minimum of three times or until a final disposition (e.g. completed interview, refusal) was reached. All definite appointments were attempted regardless of the attempt number. Some records were attempted up to 14 times with an average of 5.37 attempts per Guest/Corroborator pair to locate the correct individuals and make contact with them. To maximize the likelihood of reaching the sampled person interviewers called each telephone number at a variety of times, including weekdays, weekday evenings, and weekends. Each attempt represents a different calling time, but could include dialing several different phone numbers. Detail descriptions of each attempt can be found in the attached “Attempt File*.” (See Appendix F* for a description of the layout.) Our interviewers are trained in techniques of refusal avoidance that are effective in maximizing response rates. We developed an interviewer manual for the Sobriety Follow-up study that provided the interviewer with information about the study to be used while relating with respondents to keep them on the phone. Answers to frequently asked questions and guidelines for dealing with certain situations were available during the interviews on hotkeys listed in the interview manual. The interviewer manual is presented in Appendix B* .
Data Preparation: During data collection, interviewers documented any errors they encountered on data change forms. At the conclusion of data collection, these errors were corrected in preparing the data set for analysis. The data were converted and formatted for review in SPSS (a statistical analysis software package) and Microsoft Access. Clearwater pays close attention to comprehensive and routine data inspections, data cleaning, and data set preparation. The completed survey data set was cleaned and prepared in an electronic file format in SPSS and Excel. Any data changes were documented on data change forms which are included with the final deliverables. For table production and statistical analysis, variables and value labels were added to the cleaned SPSS data set. Calling on the sample was considered complete at a point all records had reached their maximum attempts or a final disposition had been assigned. After we exported the data from the CATI system, it was cleaned and compiled it into a labeled SPSS data file. Clearwater calculated the final disposition for each sample record. The CATI call history tables recorded the interim or final disposition entered by the interviewer for each attempt. Database programming referenced both the CATI call history table and the data sets to determine the correct final disposition.
Call Outcomes: Each call attempt was given an interim disposition depending on the outcome of the call. At the end of the field period, each record was assigned a final disposition for the study based on the history of interim dispositions and data collected for that case. Because there were multiple phone numbers on each record, the interim dispositions only captured the results of the last phone number attempted, so additional notes were made. These notes are available in the attached Attempt File*. Table 1 shows the calculated final dispositions. Detailed descriptions of each disposition can be found in the interviewer manual in Appendix B*, Table 2. The final dispositions shown represent the overall outcome of a Guest/Corroborator pair. Some individual refusals may be hidden within the partial complete counts.
These response rate calculations are based on rate calculators developed by the American Association of Public Opinion Researchers (AAPOR.) Response Rate 1 is simply the number of interviews divided by the number of total records. Response Rate 2 attempts to give a more accurate picture of the level of participation by controlling for all attempts which could not have reached a respondent.
Discussion: When both the Guest and the Corroborator were reached, 133 agreed the Guest was sober and in 99 instances at least one said they were not. Based on just the Guests’ response, 189 Guests said they were sober, while 108 said they were not. According to Corroborators’ responses, 153 Guests were sober and 79 were not. Counts of all Guests and Corroborator responses are shown in Figure 1.
Figure 1: Sober vs. Not Sober
Percentages and confidence intervals are recorded below in Table 4**. Considering both Guest and Corroborator answers, the true percentage of all Guests who are sober is 95% likely to be in the range of 51.0% to 63.7%. The observed percentage of Guests who were still sober according to both was 57.3%. More than six out of ten Guests (63.6) indicated they are currently sober. Based upon their answers alone, the true percentage of all guests who are sober is 95% likely to be in the range of 58.2% to 69.1%, as shown in Table 4** . Considering only the responses from the Corroborators, 65.9% of Guests are currently sober. The true percentage of Guests who are actually sober according to the Corroborators is 95% likely to be in the range of 61.7% to 73.7%.
Table 4** : Confidence Intervals
*A copy of the entire study, including all appendices, is available by clicking on http://www.soberforever.net/08-436%20Sobriety%20Follow-up%20Final%20Report.pdf.
**In the original final report this Table was erroneously labeled Table 3, but in fact, was supposed to be labeled “Table 4.” For the purpose of clarity the second reference to Table 3 was changed to the correct notation, Table 4.
