The Survey Process: Items to Consider
Prior to engaging in a survey research project, one must define goals for the project. These goals translate into needs; and they can broadly defined and framed as a question. For instance, the following may be said to be a general research question:
"How do our customers wish to receive information about our products/services?"
When phrased as a question, a general organizational research interest can be phrased as generic research question:
"What kinds of people exhibit what kinds of attitudes/behaviors under what kinds of circumstances?"
However, before sponsoring a survey, an organization should consider the following items:
- Is there a well-defined research interest?
- Is there a population that should be understood better, such as: customers or employees?
- Is there a sense of what is to be learned from the population?
In the event that multiple objectives exist, two elements need to be considered. First, when all the objectives are defined do they target the same population, completely discreet populations, or populations which overlap. Second, can the survey be broken down into two surveys, for examples, a customer satisfaction survey and a market survey.
Surveys are methods of research, and therefore, do not define content. Prior to initiating a survey, preliminary research should be performed on the organization, in order to determine the specific context for the survey. Consultants should obtain briefings or reading materials on the organization requesting or sponsoring the survey research. At this point, an organization outlines goals and expectations of the research. The consultants, in turn, may begin conducting focus group. With the information provided by the organization and the outcome of the focus group sessions, the consultants obtain an initial understanding of the population.
Ultimately, the population is the object of research. All elements of interest to an organization are incorporated within a population. The definition of the population comes after the determination of the general research interests. No two populations are alike. Variation in populations can originate from both the research interests and the inherent complexity in the population.
When defining a population, the following questions should be considered:
- Are there subgroups in the population (i.e. female/male, current customer/former customer, public/ private)?
- Are there research driven interests in ensuring representation from certain portions of the population?
The research objectives are the specific definitions derived from the broadly defined needs. The research objectives isolate the way customer service, for instance, will be defined. The research objectives will further focus the study on elements in the population. In the research objectives, the eventual purpose of the analysis and reporting is defined.
Sampling methods are varied and numerous, and usually reflect the needs of the survey. For instance, if a study is designed to produce results that instruct local level activities for a national agency, then the sample must be statistically representative of the smallest local unit. Research objectives and the nature of the population define sampling methods.
The survey plan covers the key dates, expected delivery method, and duration of data collection for the survey process. Knowing the final sample size of the population facilitates defining a plan action. For instance a sample of 100,000 individuals creates very different logistical demands than a sample of 500 individuals.
The questionnaire can be conceived of as the interface between your study objectives and sampled individuals, or respondents. The questionnaire is a collection of questions and measurement scales, and is only as useful as the thought put into its construction. Often survey projects begin with the misplaced emphasis on finalizing the questionnaire before any other steps are taken. However, the questionnaire should actually be developed well into the project. The areas below are areas considered prior to and during questionnaire construction.
VARIABLES TO BE MEASURED
The general purpose of a questionnaire is to collect variables for analysis. A clear sense of the variables to be measured should exist prior to writing the first question, because these variables will be used in the final analysis. Consequently, questions should reflect the variable to be measured in the survey process. If there is no intent to use a certain variable in analysis, a question should not be included in the survey.
QUESTION GROUPS
GROUPS At times, question groups may characterize questionnaires. The question groups are ways of measuring the variables described above in "variables to be measured".
QUESTION WORDING AND MEASUREMENT SCALES
Only after there is a better sense of the grouping of questions to measure variables are questions written. Particular attention needs to be placed on both wording and the scales used.
Wording
Question wording should be kept simple, brief and balanced. Each question should attempt to measure one and only one principle.
Measurement Scales
Measurement scales are used to group responses into common categories. At times, scales offer the respondent the opportunity to provide a response ranging from one number (strong agreement) to the other extreme (strong disagreement). The scales should be balanced and should offer a mid-point.
ALL QUESTIONS SHOULD OFFER THE RESPONDENT THE OPTION OF STATING THAT SHE/HE DOES NOT KNOW, OR THAT SHE/HE DOES NOT WISH TO ANSWER THE QUESTION.
Delivery methods depend largely on cost factors and response rate requirements. Response rates vary depending on the delivery method adopted. A general rule of thumb holds that the more steps asked of respondent to actually respond the less likely a response. For instance mail surveys that offer no stamped return envelopes likely will receive a low response rate, unless some other form of coercion or enticement if used to ensure a better response rate. Data collection methods also depend on the types of questions being asked and on the objectives of the study. Benefits and liabilities associated with different delivery are described below.
TELEPHONE
Telephone interviews offer the advantage of a personal touch, random digit dialing, and monitoring of both interview progress and selected interviews. Computer Aided Telephone Interviewing (CATI) allows for direct data entry into a data set.
MAILED QUESTIONNAIRES
Data collection through mail offers certain advantages over telephone data collection. For instance, mail delivery offers respondents a greater sense of anonymity. Mail, likewise offers significant cost reductions over telephone and in person surveys. Mail surveys, unless particularly salient to the respondent, suffer from fairly low response rates and require incentives and/or reminders to achieve adequate response rates
IN-PERSON INTERVIEWS
For projects that require thorough coverage of remote regions, in-person interviews offer a strategy to collect data. In addition, in-person interviews can be used when offering a treatment (i.e. a taste test between different products), when engaging hard to reach respondents (i.e. executives), or when a great deal of interaction is required between the interviewer and respondent. In person interviews can be run with Computer Aided in-Person Interviewing (CAPI).
ON-LINE QUESTIONNAIRES
On-line surveys, an increasingly popular form of data collection method, make use of Common Gateway Interface (CGI) technology to collect information from respondents using an Intranet or the Internet. As with any data collection method, on-line data collection has benefits and liabilities. In an organizational setting (i.e. Intranet), on-line surveys offer organizations near real-time collection and analysis of data regarding employee attitudes to workplace events and/or policies.
Data entry with CATI, CAPI, and on-line data collection methods described above occurs in real time. In other data collection methods, such as mail, data may be entered using scanning technology, such Optical Character Recognition (OCR).
Analysis is the point in the survey process that is most akin to divining. It is through analysis that relationships are uncovered, where causes and effects are linked; and ultimately, where an understanding of the population is developed. However, analysis, takes on many different forms, ranging from the relatively rudimentary reporting of summary data to modeling of complex systems.
DESCRIPTIVE
Descriptive analysis reports certain parameter estimates, for example: variances, means, percentages, etc. All parameters, when measured from a sample, are actually estimates of the population parameter. Descriptive analysis can tell a researcher what percentage of an organization's external customers is satisfied with the organization's service.
CAUSAL
Causal analysis, as its name suggests, is focused on uncovering the causes or determinants of certain variables. In the area of job satisfaction, for instance, causal analysis may look at the relationship between satisfaction and all other variables in the study. Methods of causal analysis include hypothesis testing (i.e. difference of means), covariance and correlation (bivariate and multivariate), chi-square, and regression. At times it is instructive to perform exploratory research on all potential relationships, thus discovering anew any potential patterns in the data.
STRUCTURAL MODELS
Models, as the term suggests, are abstractions and are constructed using a series of variables called independent, or interdependent, variables and dependent, or interdependent, variables. Models tend to be a representation of the elements in a system, such as: getting a good to market, servicing a customer's needs, or solving a technical problem. Models are instructive in isolating the strongest relationships in a system, as opposed to two variables, while accounting for the properties of the system. Modeling is especially valuable in the instances where analysis aids strategic planning and resource allocation, or where a process may be optimized.
Reporting of results is tied to analysis, the uses to which the report will be placed, and the sophistication level of the readers. Reports serve as long term communication of results. Some reports need only convey the key findings, such as executive summaries; in other cases, in-depth coverage of both methodologies and findings is required. Irrespective of the reporting method used, there should exist copies of all data, models, and results referred to in reports.
TABULAR
Tabular reports usually are in the form of cross tabs, showing relationships between intervening variables and groups and, say, attitudes.
IN-DEPTH REPORTING
In-depth reporting is provided for strategic planners and/or management. In-depth reports can cover all forms of relevant analysis performed on the data. In-depth reports are nearly a necessity for planners and management.
EXECUTIVE SUMMARIES
Most frequently executive summaries are used to broadcast results of a study to all individuals in an organization or to those individuals, such customers or sponsors, involved with the mission of an organization.
HTML AND PDF PUBLISHING
The Internet, or World Wide Web, provides a cost-effective publishing avenue that provides research results to the many individuals participating in an organization. Further, executive summaries, or other reports, may be published on an Intranet or made available to the general public via the Internet or Compact Disk (CD).
Follow-up refers to the actions taken based on research findings. Good analysis should drive the actions of management teams. Results should lead to action plans where needed. In some instances, a follow-up measurement may be required. The critical point to be made with respect to survey research, particularly when customer satisfaction, job satisfaction, or organizational research is being conducted, is that the process is designed to instruct, not punish or belittle. Therefore, the survey process, when used correctly and accompanied with proper analysis techniques, can open an organization's eyes and ears to the worlds of customers, patrons and partners.