This is the second installment of a two-part Last month, this column advised you how to decide whether testing a small set of data might help focus your project, to provide a game plan for gathering documents and to hone your data analysis. That's all well and good. But how do you sample a large group?
First decide the purpose of your sample. That will often drive you toward one of the common sampling methods, though more complex methods exist.
There's really a ladder in sampling techniques. At the bottom, researchers simply pick some cases, sometimes for a reason. Then they put a fancy word on the technique- "judgmental sample." For purists, though, only a true random sample, which carries the imprimatur of science, will do.
But random samples often won't do what you want, at least not with huge samples that might be beyond your means. In the middle, samples are based on the concept of random numbers but are "assisted" or "stratified" to give a relatively large set of particular instances you know you need.
Exploratory tools
Let's start at the bottom with the "judgmental sample," or a sample of convenience. Here, you pick out documents or records you know interest you and examine them in depth. An example is the most recent 50 entries into a database compared with the original documents that generated the records. Looking at what happened lately can give you a big hint at what's to come.
These kinds of seat-of-the-pants tests are particularly useful as exploratory tools to determine whether to go further in your project. You might want to treat them the same way researchers treat "content analysis" the science of studying documents for trends. It's the closest analogy in social science for much of what we do in the newsroom.
Researchers will sometimes check whether data entry clerks categorize items the same way. This step is particularly important when asking for help in collecting as well as entering data. Using this kind of sampling would have helped one newspaper that hired clerks to collect and type demographic information on college coaches. Simple questions about the coaches, such as years of experience, were interpreted differently. One clerk counted junior college. Another counted high school. A third counted only assistantships at major universities.
When you care about specific instances for your story - people who drive red cars, people who live in your county, or airplanes that land at your airport - judgmental samples are also useful. But if you stop there, you'll miss your story's crucial "compared to what" statement. You may know what happens locally, but you won't have context of what happens elsewhere.
Planning randomness
At the other end of the spectrum is the pure random sample, usually reserved for stories in which you plan to state your findings without ever studying the "population."
The major criteria for random sample: The sample must be selected from a list of every possible person, document or company you're studying. In most polls, this is virtually impossible. Researchers use samples of every known telephone number, registered voter, or company paying unemployment insurance.
Every person or case has an equal chance of selection. No one is excluded. Picking one case has no influence on picking any other.
Getting a random sample is easy. First, make sure your data contain unique record identifiers. In Access, you'll want to create an "AutoNumber" field. If you already have one, make sure you haven't deleted any records so you can use it as a record number. Note the number of records in your database.
Then go into Excel and create random numbers. (These aren't exactly random- they won't repeat your numbers for quite some time.) Your function will look something like this: =RANDBETWEEN(1,2000000) This picks a random number between 1 and 2 million, which you'd use if you had exactly 2 million records in your original database.
Copy to as many rows as you want for your sample. Unless you want to look at sub-groups, in which you will need more records, most samples are about 350 records. Then copy and paste this list into a new Access table. You may need to change the to Integer." Join the two tables on your AutoNumber field, which only keeps records listed on both tables.
Implicitly stratified
The dirty little secret of sampling is that hardly anyone - neither the major statistical agencies nor the pollsters - conducts pure random samples. In fact, some statistics books say that only a "probability-based" sample, not a random one, is needed to make statements about the larger group.
Statisticians, then, often do something close to - but not quite - random sampling. It's often called "list-assisted" or "implicitly stratified" sampling. Instead of choosing a list of purely random numbers, these methods take a random starting point, then choose every subsequent 10th or 100th (or some other number) record. It guarantees the sample has the mix you care about: demographics, time, or even data entry clerks.
There are three steps to these samples: Sort by an important element, like date, keypunch operator's initials, or ZIP code. Determine the "sampling interval," or the number of records between each record selected, to give you the sample size you want. Divide the number of records by the sample size. For instance, a sample of 350 records out of 2 million records would have a sampling interval of 5,714.
Figure out which records to select. You want what's called a "random start." Select a random decimal between 0 and 1, then multiply by the sampling interval. If your sampling interval is 5,714 and your random decimal is .35, then you'll start with the 2,000th record, followed by the 7,714th, and so on. Again, it's easiest to create the list of numbers in Excel, then paste and join it in Access.
Explicitly stratified
Other sampling variations include what statisticians call "explicitly stratified" samples, used when comparing one group to another or when you have evidence that one group would be under-represented in more random sample. The key to this method is predicting into which group an item or a person falls.
Say, for instance, you want to compare recidivists with non-recidivists, who are rare. If you can identify which cases fall into the two groups in advance, you can use either random method to select an equal number of cases from each group. Or you can sub-sample from the larger group, keeping only one of four who returned to jail. But here you cannot discuss all prisoners-you're limited to discussing differences between the groups.
No matter which method you use, select enough cases for your story. A sample of 385 or so will let you predict about the larger population within five percentage points of your estimate. Often, we want much more precision. You have to raise the sample quite a bit to lower this range to one or two percentage points. Look in a statistics book if you need to find a sample size. But don't start down this road if you aren't willing to look at enough cases or talk with enough people to make your story's point. Sarah Cohen can be reached by e-mail at sarah@nicar.org.