Planning an Online Survey

Survey Design & Best Practices

A well-designed survey produces better data while minimizing burden on respondents. Consider these foundational principles when planning your survey.

Define clearly the purpose of the survey in the introduction: 

  1. Who is administering the survey?
  2. What type of information is being collected?
  3. Essential aspects of the process (estimated time to take the survey, any necessary instructions, closing day of survey, etc.)
  4. If applicable, include a statement(s) about whether respondents’ survey responses are anonymous or confidential, mandatory reporting (see “Is your survey compliant?” section), and any necessary IRB-required language.

Ask Relevant Questions

Only ask questions that apply to your target audience. Use survey logic or branching to ensure respondents only see questions that are relevant to them. For example, questions about campus recreation facilities should only be shown to students who use those facilities.

Ask Answerable Questions

Make sure respondents can reasonably answer each question. Avoid asking them to recall information or experiences they may not remember accurately

Focus on One Topic Per Question

Each question should measure a single concept. Avoid double-barreled questions that ask about multiple topics at once, such as pay and benefits. Instead, create separate questions for each topic.

Use Clear and Neutral Language

Write questions using plain, specific language that respondents can easily understand. Avoid jargon, acronyms, technical terms, and wording that suggests a preferred answer.

Use plain language:

  • ✅How often do you use Canvas to access course materials? 
  • ❌How frequently do you leverage the LMS platform?

Avoid jargon or acronyms:

  •  ✅Did you complete an internship? 
  • ❌Did you participate in a high-impact practice?

Avoid leading questions:

  • ✅Do you agree or disagree with the following statement: The new ban on vaping was communicated effectively.
  • ❌Do you agree that the confusing new ban on vaping created problems for students?

Use Balanced Response Scales

Response options should provide an equal opportunity for positive, negative, and neutral feedback. Balanced scales improve data quality and reduce bias in survey results. For example:
 

A satisfaction scale such as:

  • Very Poor (-)
  • Poor (-)
  • Neither Poor nor Good (neutral)
  • Good (+)
  • Very Good (+)

is more balanced than:

  • Poor (-)
  • Fair (+)
  • Good (+)
  • Excellent (+)

which offers limited negative response options and no neutral option.

Keep Scales Consistent and Fully Labeled

Use the same response scales throughout the survey whenever possible. Consistent scales reduce respondent effort and make analysis easier.

Label every response option rather than only the endpoints. Fully labeled scales help ensure respondents interpret responses consistently. For example: 

A 1-to-5 satisfaction scale should include labels for all five values:

  1. Very Dissatisfied
  2. Somewhat Dissatisfied 
  3. Neither Satisfied nor Dissatisfied
  4. Somewhat Satisfied
  5. Very Satisfied

Instead of:

  1. Very Dissatisfied
  2.  
  3.  
  4.  
  5. Very Satisfied

which makes the middle options open to respondent interpretation.

Use Response Options from a Single Dimension

Answer choices should measure the same concept. If a question asks about frequency, every response option should represent a frequency category. Mixing frequency, circumstances, and satisfaction levels within the same response list can make results difficult to interpret. For example:

All options measure frequency:

  • Never
  • Once
  • 2-3 times
  • 4-5 times
  • 6+ times
  • Don’t know

Instead of:

  • Never (frequency)
  • Monthly (frequency)
  • When needed (circumstance)
  • Very satisfied (evaluation)
  • Don’t know

Use Closed-Ended Questions When Possible

Questions with predefined response options are easier for respondents to answer and easier to analyze. If additional flexibility is needed, consider including an "Other (please specify)" option.
 

Use Conditional Questions

Show follow-up questions only to respondents for whom they are relevant. This improves the survey experience and reduces frustration. If available, use your survey tool's display logic functionality. When creating response options, make sure the list is as complete as possible. Options such as "Don't Know" or "Other" can help capture responses that would otherwise be missing. For example:

Question 1:

  • Do you have an assigned academic advisor?
    • Yes
    • No
    • Unsure if I have an academic advisor

Question 2 (display if Question 1 - Yes"):

  • How often have you connected either in person or virtually with your academic advisor during the past academic year?
    • Never 
    • Once this semester 
    • A few times this semester (2-3 times) 
    • Monthly
    • More than once a month 
    • Don’t know

Make sure the list of possible answers is exhaustive enough. For those respondents who selected “Yes” in Question 1, but do not recall how often they connected, they have the option to select “Don’t know.”  The benefit is that the results might provide some helpful insights about those respondents who might have opted not to answer this question otherwise because they do not recall the frequency of their connection with their advisor.  Offering “Other” as a response might also make the list of answers exhaustive.

Limit Open-Ended Questions

Use open-ended questions only when detailed feedback is essential or when valid answer choices cannot be anticipated in advance. Keep in mind that open-ended responses may require additional review and are subject to university data handling and mandatory reporting requirements.
 

Ask Only What You Need

Every question should support a specific analysis or decision. Unnecessary questions increase survey length, contribute to respondent fatigue, and reduce completion rates.
 

Allow Respondents to Review Answers

Whenever possible, include a back button so respondents can review and revise their answers before submitting the survey. See link for setting up the back button with Qualtrics.
 

Test Before Launching

Always test the survey before distribution. A small pilot with one or two reviewers can identify confusing wording, logic errors, missing response options, and other issues that may affect data quality. Catching these problems before launch will improve the accuracy and usefulness of your results.

Survey length and field period (how long the survey is available to participants) both influence participation and data quality.

Below is a graph showing the relationship between the survey length (in minutes) and the response rate, as well as the relationship between the number of days a survey is open and the response rate.

 

 

Reference

Struminskaya, B., & Gummer, T. (2021). Risk of nonresponse bias and the length of the field period in a mixed-mode general population panel. Journal of Survey Statistics and Methodology, 10(1), 161–182.

Why It Matters?

Research suggests that extending a survey's field period does not always improve representativeness. After a certain point, additional responses may come from the same types of respondents rather than from underrepresented groups.

Additional Resources

Qualtrics: For How Long Should You Accept Survey Responses? For How Long Should You Accept Survey Responses?

✅ Keep surveys open for approximately 10–14 days.

✅ Choose a launch date strategically: with your population type in mind and without overlapping with existing institutional surveys (see KU Survey Calendar).

✅ Send one or two reminder messages.

✅ Consider additional strategically timed reminders between launch and closing (Time 1: Invitation to participate, Time 2: Reminder about the survey, Time 3: Reminder of the survey on the day it closes)

✅ Clearly communicate the survey closing date in the invitation.

✅ Monitor response patterns rather than keeping surveys open indefinitely.

✅ Help reduce survey fatigue by following the KU Survey Coordination Policy when applicable. 

Response rate alone (see section on “Survey Metrics and Reporting”) is not a sufficient measure of survey quality.

Key considerations include:

  • A high response count does not automatically mean the results are representative.
  • Nonresponse bias can occur when certain groups are less likely to participate than others. 
  • Extending a survey's closing date may simply generate more responses from participants who are already well represented.
  • Careful sampling strategies can substantially improve data quality.

Recommended Sampling Approaches

Some useful terminology of sampling terms:

  • Random sample = only a subset is invited, selected randomly {through a random sampling process).

  • Census = everyone is invited (everyone who meets the criteria for selection in the study population). 

  • Convenience sample = participants are recruited based on availability. 

  • Self-selected sample = participants volunteer to respond.

 

Census vs. Sample (subset)

A census survey invites every member of a population to participate – based on the target population or study population in mind.  This approach is useful when results are needed for many subgroups or when broad participation is an important goal, for instance if reporting participation rates as indicator of the general interest in the survey is important. However, for large populations such as 22,000 undergraduate students, a census may contribute to survey fatigue and yield relatively low response rates.

A random sample survey invites a scientifically selected subset of the population. When properly designed, a random sample can provide highly accurate estimates of student opinions and experiences while reducing respondent burden and improving fieldwork efficiency. In many cases, surveying a representative sample of students is a more effective use of institutional resources than surveying the entire population.

 

For information about sampling methods, see article “Sampling Methods for Political Polling” from the American Association for Public Opinion Research (AAPOR) [https://aapor.org/wp-content/uploads/2022/12/Sampling-Methods-for-Political-Polling-508.pdf]

Random Sampling is the process of selecting a subset of individuals from a larger population in such a way that everyone has a chance of being chosen. 

  • A well-designed random sample can provide accurate information about the population while reducing respondent burden and survey costs.
  • Increases the likelihood that results reflect the target population. 

 

Stratified Sampling involves dividing a population into meaningful groups and randomly selecting participants from each group. 

  • Helps ensure representation of smaller populations or groups that might otherwise be underrepresented.

Response Monitoring

  • Track participation across important demographic or organizational groups and adjust outreach efforts accordingly.

Improving participation starts with understanding your audience.

Reduce Survey Burden

  • Keep surveys concise.
  • Avoid unnecessary or repetitive survey requests. Don’t ask the same group of people to complete multiple surveys with similar questions. 
  • Ensure respondents can complete the survey efficiently (See section above on “Writing Effective Survey Questions”).

Increase Motivation

Consider:

  • Explaining the purpose and importance of the survey. Taking time to craft a thoughtful survey introduction helps respondents understand why the survey is being conducted and why their participation matters. Communicating the survey's purpose and significance reinforces its legitimacy and demonstrates that it is a worthwhile use of respondents' time.
  • Communicating the value of participation by explaining how the results will be used. For instance, in a survey on the cost of attendance, students may be more motivated to respond when they understand that their input will help the institution assess the full range of student expenses, improve information provided to current and prospective students, and support decisions regarding financial aid.
  • For instance, it is motivating participation to explain, in a survey on the Cost of Attendance answered by students, that the results will help better assess various student expenses that can be communicated to current and prospective students and inform decisions regarding financial aid awards. 
  • Providing appropriate incentives when feasible.
  • Sharing results and outcomes with participants when possible.

✅ Have you checked if you are a mandatory reporter for your survey? (See the “Language to include in surveys” Survey Reporting Sample Language | Office of Civil Rights & Title IX [https://civilrights.ku.edu/survey-reporting-sample-language]

✅ Does your survey project need to go through the IRB process?

✅ Does the Survey Coordination Policy apply to your survey?

✅ Have you made sure your survey complies with Digital Accessibility requirements?

 

All KU surveys should be accessible to respondents with disabilities.  Check out our KU Digital Accessibility website Digital Accessibility.

Accessibility Checklist

  • Run the Qualtrics Accessibility Checker before distributing your survey.
  • Verify compatibility with screen readers.
  • Ensure all survey elements can be completed using keyboard navigation.
  • Review question types for accessibility limitations.

Accessibility compliance is a requirement for KU surveys.

Emerging Tools

  • Qualtrics AI capabilities
  • AI-assisted survey development and review tools

(Additional content can be added here.) Link to page specifically about accessibility here.

Understanding Response Rates

Different response-rate calculations can produce different results depending on how eligibility and participation are defined.

The current version of Qualtrics provides some very useful metrics and descriptions of those metrics.

Qualtrics automatically calculates the following rates:

  1. Start rate defined as “Number of all surveys started / number of all recipients that were delivered the invite message.”
  2. Response rate defined as “Number of all surveys finished / number of all recipients that were delivered the invite message.”
  3. Completion rate defined as “Percentage of all surveys finished / all surveys started.”

AAPOR (American Association for Public Opinion Research provides a calculation of the cooperation rate, which focuses on participation.  The cooperation rate measures the percentage of eligible individuals who were contacted and agreed to participate (Number of completed surveys/number of eligible individuals contacted. 

The calculation of response rates, completion rates, collaboration rates, and other survey metrics depends heavily on how both responses and nonresponses are classified. For example, was the respondent ever successfully reached? Did the invitation email bounce? Did the respondent open the email and click the survey link but never progress beyond the landing page? The answers to these questions can significantly influence how participation is categorized and, ultimately, how survey rates are calculated and interpreted.

Check out Qualtrics website on Distribution Fields to understand the various categories of invitations sent: https://www.qualtrics.com/support/vocalize/widgets/distribution-reporting/#Fields

For more advanced survey metrics, definition, and calculations, see the resources links below: