A comparison and evaluation of two survey data collection methodologies: cati vs. mail

In order to assess and evaluate the relat ive effectiveness of administering a traditionally mail based attribute frame survey using computer assisted telephone interview (CATI) as the primary data collection mode, EIA conducted a pilot study. The pilot study used a matched pair stratified random sample. Each sampling unit was matched on characteristics thought to be related to the ease of responding by mail versus telephone. Two separate surveys were then conducted and tracked by the primary collection mode, mail or CATI. The results of the pilot are presented and compared for two sets of measures, cost and data quality, where data quality is measured through response rates, response time, and response edit failures. The impl ica t ions and recommendations for applying the lessons learned to a full survey data collection effort are also described. Study Design Due to budget reductions for the 1998 EIA-863 frame survey of approximately 22,000 companies, alternative data collection methods were examined for potential cost savings. Because of the extensive number of follow-up phone calls required for nonresponse and edit failures, EIA considered changing the traditionally mail based survey to a computer assisted telephone interview (CATI) survey. To evaluate the feasibility of C ATI as the primary data collection method, and to determine its cost effectiveness, EIA conducted a pilot study designed to allow comparisons between the two data collection methods. The pilot study used the previous 1994 EIA-863 survey as the sampling frame. Companies who reported an active status at that time and were not reporting on the annual survey were considered to be in scope. Companies selling petroleum products in four or more states were eliminated to simplify the programming for the pilot. This resulted in 20,419 companies that could be sampled for the pilot study. Two variables from the 1994 survey were then used for stratification: 1) number of states reported and, 2)number of items reported. These variables were considered to be directly related to the respondent's initial decision to complete the survey by phone or by mail and to the cost of conducting the survey for either data collection method. The pilot study used a matched pair design with each sampling unit matching on the two stratification variables. The previous survey's respondents, as described above, were then allocated to each stratum for each of the two data collection components: CATI and Mail. A random sample of 500 permitted 95 percent confidence intervals on estimates with plus or minus 5 percent. The majority of companies in the previous survey were one state-companies. Half the sample was therefore allocated to the single statestatus-companies and half to the two and three state-status-companies. A fixed number was then allocated to each number-of-items-reported stratum within the number-of-states-reported groupings, except in the three statestatus-cells where the population was not large enough to achieve the allotment designated. This yielded a total sample size of 568 for each survey instrument mode, as shown in Table 1, sufficient for the confidence intervals stated above. Table 1. Pilot Sample Allocations by Stratum