d i g i t a l a d v i s o r . . .
Five Big Decisions About Your Survey
1. Defining a Target Population and a Sampling Frame
The first thing you need to conduct a survey is a list of relevant potential respondents. Usually there are two slightly different groups of interest. The target population refers to everyone you are ultimately interested in, everyone who would be theoretically eligible to be in your study, and the sampling frame is everyone you could actually invite and would invite if you had the resources and wherewithal to do it. It is a list of all possible practically feasible people. So first you want to think about the population of interest, the target population, and then you need to think about where you can get a list, or sampling frame, that would roughly represent that population. Defining a target population is a conceptual problem; defining a sampling frame is a procedural problem.
Say you are interested in high school students. Your target population might include
You want to define your target population according to things like what age, what level of education, what demographic range, gender, ethnicity, geographic region and so forth.
Once you have a definition of a population, you need to figure out how you can find a list of all the people who are in it. This could be the list of students enrolled in some specific schools, it could be a list of young adults who just obtained drivers licenses in the past year, a list of families who have high-school age students, a list of students who have appeared in court or students who have taken their SAT exams.
Notice that none of these lists includes all young adults between the ages of 14-18. All are tilted in some way. Even a telephone directory is not a complete list of a population. Some families have no phones, some have only cell phones. No list will completely cover your target population, so you need to think about which list is closest and what types of biases each one may have.
Having a list is obviously important for procedural reasons, but it is also important to think about because any specific list you use will necessarily differ in some way from your population of interest. For instance, if you use the list of students enrolled in Williamston high school, you will miss students who have dropped out, and their views may be important to you. Or there may be errors in the records so that some are listed as sophomores when really they are juniors, or as girls when really they are boys, or as in a college prep program when really they are in a vocational program.
Any list you use will have some errors and some discrepancies and will not be an exact accurate match to the population you conceived of. Hence the importance of thinking about possible lists and what flaws each one might have. You want to know how well your specific list matches your conception of the population you want to survey, and you want to be aware of ways in which it may miss the population you are envisioning.
For more on the problem of defining who you will study, see my page on sampling
2. Persuading People on the List to Participate
One you have defined the population of interest and have figured out an acceptable list of potential study participants, you have to decide how many of them you will actually contact and how many of them are likely to respond. If your population and sampling frame are really large, you won't have the resources needed to survey all of them, so you'll need a way to sample a subset. For instance, if you have lists of students from each of 15 high schools, you might want to sample a handful from each one, rather than surveying every student in every school.
The other side of the solicitation problem is how to persuade them to participate. Here are some things to consider:
3. Figuring out What You Want to Learn From Them
This is the heart of the matter here, and it will take you some time to do this. Often, it takes many questions to really sort out an issue, yet the more questions you ask, the smaller the number of people who will reply. So you need to think hard about what you really want to learn from them and ask only about those things. There are three steps to this: (a) figure out what you want to learn, (b) figure out what you'd have to ask them in order to learn that, and (c) figure out how to ask them that.
(a) What you want to learn. Say you are interested in students views about MEAP tests in social studies. You should be able to formulate a series of questions that might look something like this:
You also need to think about whether, when you ask about people's views, you mean views about taking the test itself, views about the consequences of taking the test, views about the content, views about tests in general, etc., or do you actually mean feelings like frustration, confidence, fear etc.
Notice that, although I've written these outline topics in the form of questions, they are not as general as the questions in the original list of what you want to learn. But they are still not in the form that you would actually ask them on your survey. The purpose of this step is to force you to think through what topics you will need to cover on the survey. If the list is long, you'll need to think about tradeoffs. Once you have this settled in your mind, you can begin to think about how to phrase questions for each topic on your list.
(c) How to ask. The actual survey questions will look different from the outline. At this stage, we get into details of how to phrase things so you aren't biasing their responses, aren't confusing them, etc. When you get to this stage, you need to talk with an advisor who knows about surveys, to learn nore about question formats and their relative advantages and disadvantages, how to avoid confusion with your wording, how to be efficient and so forth. A good way to begin would be to search the literature for other studies that relied on surveys and studied topics similar to your topic.
Question construction is really important. It can increase the likelihood that people will respond and that you'll learn what you hope to learn. If your respondents don't feel that the questions allow them to say what they really feel, they will become annoyed and toss the whole thing or, if they feel obligated to comply, fill it out with nonsense.
4. Coding and Cleaning your Data
Here are the types of things that will need to be done:
- Once you get questionnaires back, you will discover that the markings are not clear. Sometimes people will check two options, sometimes none, and you will need to have some decision rules for how to handle these.
- You will need to figure out how to actually translate all of these responses to a spreadsheet. For instance, if you have yes-no questions, you might code the yes's as 1 and the no's as 0. Each question format will have some details to be sorted out as you translate pencil markings on sheets of paper into numbers on a spreadsheet
-After everything is on the spreadsheet you'll want to check it again for errors. Usually that means asking a computer program such as SPSS to present you with some histograms of each item, just to see what they look like. This helps you find, say, a erroneous score of 8 on a question that has 1-5 scale.
© Mary Kennedy, 2006