Surveying readers is one of the most vital activities of any magazine publisher. Survey results help you plan the right editorial content for your readers. They also help you prove to advertisers that your magazine is the right one for them.

In this special report, you'll learn how to organize a survey, from developing the sample size to mailing the package. You'll also learn how to analyze results, and will share in the experiences of other editors and publishers who have conducted reader surveys. First, though, some basics...

Sample Size

The sample is the portion of your readership to whom you send the questionnaire. You want a large enough group so the sample can be representative of all readers. The sample size should also be great enough to allow cross-tabulations of data (i.e., for all who answered yes to question 3, what are the results for question 7?).

A sample size of 1,500 is recommended in most cases.

Sample Derivation

To be representative, the sample should be drawn randomly from your subscriber list. For instance, if your circulation is 15,000, to get an every "nth" selection of 1,500 readers, you would select every 10th name on the list.

If you entrust this selection to an outside list maintainer, be sure to check the names and addresses you are provided for randomness. (Check if the zip or state profile roughly matches the profile of your entire list, for example.)

The Questionnaire

State your questions simply and clearly. Avoid complex questions. Don't ask loaded questions. Call for multiple-choice responses.

When you've drafted your questionnaire, test it on your staff, on outsiders. Be sure the questions are understood in the way you intended.

The Package

Print the questionnaire on a card that the respondent can just drop in the mail.

If you have too many questions for that, explore some form of a folding self-mailer, or enclose a business reply envelope.

Using pre-paid business reply mail will increase your response.

Allow respondents to be anonymous if they wish.

Enclose a cover letter explaining the purpose of the questionnaire, telling the reader why his/her opinion is important.

Also use a window envelope to mail the package -- it will get opened more often than a closed one. The mailing label, affixed to your cover letter, should show through the window.

For your return address, leave off the name of your magazine or firm, unless you feel its presence will definitely increase the chances of the envelope's being opened.

Apply postage by stamp, meter, or bulk indicia, in that order of preference.

Survey Timing

Mailing times for the best response rate are generally September and January. This will vary from industry to industry, though. You may have some clues from your subscription promotion mailings as to when is best for your readers.

Whenever you mail, however, be sure to give the survey recipients a deadline for responding.

What to Ask

Getting reliable answers from a survey requires that you ask the right questions. Being clear yourself about what information you are seeking is an important first step.

Often it is more reliable to ask for readers' reaction to something rather than to pose a hypothetical question. For example, if you want to know what readers like, ask them directly. Have them rate their reaction to each article in the last issue on a scale of 1 to 5. Label the choices: 1-very favorable, 2-favorable, 3-neutral, 4-unfavorable, and 5-unfavorable.

Rating the Response

I first began such a survey on a monthly magazine more than 15 years ago. Each month 300 questionnaires were mailed to a randomly selected (every nth) list. Typically, about 100 would be returned. I've since used the same survey design with a number of clients and find that the response percentage can vary considerably.

In general, look for a return between 10% and 50%. The percentage you experience can be considered your first piece of survey data! If 50% of the recipients are returning their questionnaires, you have a highly interested readership. With a 10% response you should consider that your readership is somewhat apathetic.

Tabulating Results

This type of survey, done on an ongoing basis, will produce a surprising amount of in-depth data.

The first tabulation to make is the mean or average score of each article. To find it, count how many answered "very favorable" and multiply that number by the numerical value for that answer, 5 in this case (4 for favorable, 3 for neutral, etc.). Do this for each of the five possible answers. Add these products and then divide by 5 to get the mean score for that article.

Expect that most mean scores will fall on the high side of "neutral". After all, to be subscribers, they must have somewhat of a positive predisposition toward your editorial content!

An optional calculation for each article is the mean deviation. It will give you a measure of how unanimous is the mean. For example, an article could score 3.0 if everyone rated it "neutral". On the other hand, that same score could arise if half the respondents rated it "very favorable" and half "very unfavorable". And you might make very different editorial judgments depending on which way the 3.0 arose.

Measuring Randomness

What if one article scores 3.9 and another 3.8? Is there really a significant difference? Or can that small difference be explained as just random variation?

You can measure that. Include on your questionnaire, along with the list of articles, some boiler-plate feature of your magazine. Perhaps your index of advertisers, or staff listing, or events calendar -- something for which there will be no change in reader interest from month to month.

Over a period of time, observe the variation in scores received by this boiler-plate entry and consider that number to be a measure of how much variation will occur in scores from random occurrence.

Spotting Trends

Beyond using this survey technique to evaluate individual articles, it can be used to measure response to general areas.

Do readers prefer financial articles over industry news? Do people-oriented articles out-pull technical ones? Do picture stories score better than straight text? To get these answers, take several months of survey data (the more, the better). Group the articles by the categories in which you're interested. Then compute the mean scores for each group. And compare.

Another trend worth looking for is one that is seasonal. Look for this one by computing the averages of all the articles' scores in each issue. Plot them on a graph and observe whether the ups and downs follow a seasonal pattern.

Marketing Questions

It's not always that an editor has the luxury of conducting a readership survey for strictly editorial purposes. Sometimes a survey must additionally serve the needs of the publication’s advertising department.

In addition to providing the editor with feedback, it may also have to give the marketing people an objective measure of reader acclaim, preferences with regard to the competition and possibly some demographic data.

For the least biased results, a survey of this nature should be done without the recipients' knowing that you’re sponsoring it.

Who's on First

This is the kind of survey that compares how you're doing with regard to the competition. Ask as your first question, "Which magazines do you receive regularly?" Along with your magazine, list your top two to three competitors, all in alphabetical order. Provide a place to answer "other" where the recipient can specify the name of a magazine not listed.

Later, when performing tabulations to compare your magazine with publication "x", be sure the questionnaires you're counting are those of readers who are receiving both magazines.

Reader Satisfaction

"Which magazine do you enjoy reading the most?" is an obvious first pass at pegging reader satisfaction. Repeat the same list of magazines used above.

But enjoyable reading doesn't tell the whole story. Especially if the magazines are read for information rather than entertainment. Try asking, "Which magazine has helped your business the most?" If you're not publishing a business magazine, just change the word "business" to profession, avocation, health, or frame of mind -- whatever your magazine addresses.

Indeed, Who's Reading?

Classifying readers is quite useful for both editorial and advertising purposes. The editor gets a better perspective of whom he or she is writing to. With this same data, the ad salesperson is able to show prospective advertisers how your readers will be ready buyers of their products or their services.

For a business magazine, there are two key questions: "In what type of business are you involved?" and "What is your position?" It is often valuable to cross-tabulate answers to these questions with the reader satisfaction questions. In that way you'll know to which categories you're appealing and to whom your editorial material is most valuable.

Cross-Tabulating

When a survey cross-tabulates one question against another, sample size must be sufficiently large. Otherwise, the number of answers being counted in a particular cross-tab could become so small that the difference of plus or minus just a few answers could cause a large jump in percentage. That could lead to unreliable results.

Again a sample size of 1,500 is recommended. It should give you a sufficient number of returns for cross tabs. (Expect a return between 10% and 40% for this type of survey.) However, the more subsets you use in cross-tabulations, the larger your sample size needs to be.

Ranking Interest

If you cover, for example, technical articles, human interest stories, gossip, industry forecasts, new product news -- list these categories and ask readers to rank them starting with their number one interest.

Clearly, this will let you know if the amount of editorial space given over to each category is consistent with the amount of reader interest. It is also interesting to cross-tabulate this ranking by the various occupational or demographic categories.

The Acid Test

To really see how closely you're hitting the editorial mark, repeat the above list of categories and ask the reader to indicate which magazine does the best job in each. You'll see in which areas readers feel you're doing well. And by referring to the ranking of the categories, you'll see if those areas are important or not to the readers.

When asking for a ranking of categories, expect that many respondents will not answer this question. It takes more effort than just checking off a multiple-choice answer. In addition, some will answer incorrectly by inserting check marks rather than rank. This becomes a problem if the number of responses you are working with is small to begin with.

Freeloaders?

Pass-along readership is an indication of the total audience reached by a publication. It can be assessed by asking, "How many others read your copy?" To the ad salesperson, this may reveal a previously "hidden" audience which can be a boon to sales promotion.

It is also significant to ask if these other readers influence the decision to subscribe or renew. If they do, aren't they as valuable as your primary readers? Your concept of your reading audience may take on a new dimension!

In-the-Book Surveys

Editors also survey readers directly through their publications. Questionnaires may be printed on a page in the magazine as a "bingo" card, an insert, or on a cover wrapper.

A characteristic of the "in-the-book" surveys, however, seems to be a lower response rate compared to questionnaires mailed to readers. Responses for "book" surveys range from 10% to 0.1% (compared to a range of 80% to 10% when using the mail).

Why Do It?

Savvy's editor, Wendy Reid Crisp, says surveys have "helped immensely." She polls her readership every month. "As a result, I am very well aware of what the key issues are," explains Crisp.

Leslie Eden, publisher of Hydro-Review, finds surveys helpful in judging the success of articles. "Articles that bomb can be obvious without surveys," she says, "but articles that seem to do reasonably well are not so obvious. Surveys bring winners to the forefront." Eden uses her survey additionally to see which authors do well and to get a sense of how to organize each issue.

Donald Christiansen, publisher of the award-winning IEEE Spectrum, adds another reason to survey: "It's given so much information, we've even used some as editorial copy in the magazine."

What Kind of Survey?

The monthly surveys of Savvy and Hydro-Review seek readers' reactions to the tables of contents for particular issues. In addition, both Spectrum and Hydro-Review make occasional use of an outside research firm or consultant. They conduct more in-depth and statistically valid studies to supplement and validate the staff-generated results.

The Spectrum surveys incorporate what Christiansen calls "benchmark" questions. That is, some of the same questions are used from year to year to track how the magazine is progressing. According to Christiansen, Spectrum has received "a lot of good ideas from respondents' open-ended comments."

Planning, too, seems to be an area where surveys help. Both Crisp and Eden remark that survey results are valuable when editors look toward the future. Eden explains, "Often in choosing between options, it's hard to make decisions. The survey can give you confidence."