How to Choose a Survey Tool to Conduct Original Research: 16 Questions to Ask

Time to read: 8 minutes

So you’ve decided to publish survey-based original research to build your audience? How do you know which survey tool to choose?

There are dozens of tools to choose from, which can make the decision feel overwhelming. We’ve developed a list of questions to consider as you do your due diligence. These questions are based on many years of experience using survey tools, as well as some mistakes made along the way. (And, read on to learn about our experiences with the tools we have tried.)

Questions to you ask before investing in a survey tool

Do you want to conduct a poll or a survey?

Some people use the terms poll and survey interchangeably, but they are very different.  A poll asks a single, multiple choice question. A survey asks more than one question, and is typically more wide-ranging in topics and question formats.

The distinction of one question versus >1 question is important. With a poll, you do not get the chance to ask what we call the “knock out question” (also called a “screening question”).

For example, if you need to ask marketers whether they use chatbots, you need to ask two questions: (1) Are you a marketer? (2) Do you use chatbots? You use the first question to “knock out” those people who don’t represent the category you’re trying to study.

Most polling apps don’t give you the option to knock-out survey takers, which is a big no-go for many applications. Also, polls only offer one question format: multiple choice (i.e. “choose one”).

What type of customer support does the platform offer? At what price?

Look into what types of support your license level offers–both educational support as well as last-minute-crisis support. Most of the big players have packages that include customized training, as well as priority phone and email support. Trust me: When you run into a problem at the eleventh hour, you will not want to be signed on for a plan that only guarantees a response within 24 to 48 hours.

What question formats are available?

Pay close attention to the question formats available at each price point. What question types will you use? If you’re confused by the list of question types, consult SurveyGizmo’s comprehensive guide to question types.

PRO TIP: Word to the wise: whenever possible, simple = better. While grid-style questions and heatmaps may look interesting, many of these more advanced question types do not play well on mobile. When you’re starting out with surveys, don’t experiment too extensively — and learn which survey design mistakes to avoid.

What type of survey logic is possible?

Your survey may include conditional formatting or survey logic. This is when the type of answer someone gives determines the future questions they may answer.

For example, if you’re surveying people about their pets, you don’t want cat owners to face questions about dogs. A cat owner might skip over certain questions (“skip logic”) or cat owners may be shown a series of questions no one else sees (“display logic”).

Another common feature is called “piping.” This refers to when a question/answer incorporates information gleaned earlier in the survey. For example,  if someone says they live in Boston, the next question may say, “What is your favorite restaurant in Boston?”.  Survey logic is very useful–even for relatively simple surveys–and often costs more to include as an option. It’s critical to find out what types of survey logic are available at the price point you are considering.

Does the platform let you see how your survey will look via mobile?

Many of the enterprise platforms will let you preview your survey to see what your audience sees, but be sure it also offers a mobile preview–a critical tool to alert you to problems for small-screen survey-takers. (Some question types are notorious for displaying poorly on mobile, so this feature is a need-to-have.)

How does the reporting function work?

To answer this question, sign up for a trial. Most survey tools will let you have a weeklong trial, and during the trial set up a new survey and then populate it with dummy answers (there’s usually an option to fake a few dozen responses so you can see test your survey).

Once you’ve populated the test with the dummy responses, it’s time to study the tool’s reporting function. For instance:

  • Is it easy to read?
  • How does it handle conditional formatting? We find some platforms struggle to display questions using skip/display logic … so that’s an area to investigate closely.
  • Is it easy to set up a “crosstab” report? This is a report that displays how people in different segments answer the questions differently. For example, if you ask people what industry they work in, you may generate a crosstab of the top six industries and how those respondents differ. This is a very common report view and one you should explore if you think you will use it.
When considering a survey platform for your #originalresearch project, 1) create a survey; 2) populate it with test data; and 3) check out the available reporting functions available. Click To Tweet

What type of experience will the survey taker have?

Does the survey platform convey the gravitas you’re aiming for? Some survey platforms offer a user experience that is just a small step up from “I’m working on a high school summer project” vibe. Be sure to test out formatting and customization options to make sure it conveys the right look and feel.

Can you customize branding in the survey? Get rid of the “powered by” footer? Use your own URL?

Another thing to consider as part of the user experience is if you can display branding. Of course the answer to all of these is usually “yes” … but at what price? When you test-drive surveys during a trial period, be sure to test out the branding options to be sure your survey-taker’s view suits your needs.

How does panel pricing compare to competitors?

Here’s a sneaky tip: When you buy a survey platform, you’re also hooking your wagon to that company’s panel services. It’s a cost that’s not on the website, but is worth investigating. If you plan to use panels (instead of your own lists or lists from partners), ask the company to price out the types of panels you may need during your subscription. That way you are comparing not only license costs, but the entirety of your future costs. (You’ll need to come up with a sample panel profile or two to compare prices.)

Is the platform GDPR compliant? Will it support you in this endeavor?

Most (if not all) research platforms have made changes to comply with GDPR. If you survey people affected by the GDPR ruling, find out how the feature works. Does it have an opt-in feature (most use a checkbox users must agree to in order to proceed through the survey)? Can you customize it so your own company’s policies are reflected on that page?

Are there retargeting options?

One of the biggest challenges related to hosting surveys is pulling in responses. To get the most out of advertising, explore what retargeting features your short list of research platforms offers. For example, many survey tools allow you to add tracking codes via Javascript (e.g. Facebook Pixels) to see how well your ad dollars are converting and to find out more about your audience.

Does it have multi-language capabilities?

Most won’t need this option, but if you work for a company with multiple regions and languages, investigate each tool’s capabilities in this area.

Can you compare year-over-year responses?

Do you expect to conduct more than one survey? If you aim to run an ongoing survey–such as an annual benchmarking study–you want to choose a platform that will grow with you. Be sure it can handle year-over-year reporting rather than having to create the report manually.

Does it integrate with your tech stack?

Look for integrations with your CRM, marketing automation, email marketing, social channels and analytics tools.

What is the cost of the subscription?

For those on the leanest of budgets, you will find free options to host your survey … but beware that free tools typically require you to use a very simple question format (for example, you can’t use any type of conditional logic), reporting features are minimal and customer service is non-existent. Free is serviceable if you’re starting out, but if you stick with research you’ll likely outgrow free tools quickly. Beyond free tools, you should expect to pay anywhere from $40 per month all the way up to more than $500 per month.

What is the maximum number of respondents for my subscription plan?

This can become a surprise cost to marketers getting started with surveys. Investigate whether your plan has a respondent limit (e.g. no more than 1,000 respondents in one year) and how likely you are to exceed it. What is the cost of exceeding it?

Once you choose your survey tool, your next question will likely be: What questions should I ask? Get tips on how to avoid common survey design mistakes.

A quick overview of the tools we have tried

Here is a quick look at some of the survey tools we have tried at Mantis this past year with some observations.

Google Forms

Google’s survey tool is a serviceable option for those experimenting with simple surveys. It offers a handful of question types, and the responses will automatically populate a dedicated Google sheet for analysis. It’s a good tool for very basic surveys.

What are the drawbacks? For beginners, the question formats will likely be sufficient, but we find professionals outgrow Google forms quickly. And G-Forms data reporting is ‘meh’, which adds to your workload. And finally… I just feel weird answering Google surveys because I know Google knows who I am. It makes me uncomfortable. (Call me crazy.) I’m betting others may feel the same, which means using G-forms may impact your response rate.

Typeform

Typeform is the most elegant of all survey tools and the first time I encountered it, I fell in love. Lifestyle brands tend to use Typeform because it’s the research equivalent of a trendy coffee bar. It’s an ideal tool for a hip fashion brand or a cool music festival, but may not be a good choice for beginners as the UX isn’t always straightforward to navigate.

Pollfish

Don’t let the name deceive you … Pollfish can host a poll or a survey. What makes the platform different is two-fold: (a) it specializes in mobile surveys and (b) it’s designed for those who need the survey + the audience.

Pollfish’s pricing includes hosting the survey and pulling in the right people to take the survey. For that reason the prices may seem high, but they are in line with what you would find with any of the other research tools if they bundled in the panel price.

SurveyMonkey

This industry leader is a top-choice among research DIYers for good reason. The interface is easy to use, it includes tips and tutorials for all experience levels, and it has a generous library of existing surveys to guide you as you build your own. We also like that a new user can begin with the free or low-cost version of SurveyMonkey, then “graduate” to some of the more expensive but more robust user licenses.

SurveyMonkey’s panel pricing is reasonable, and they are very responsive when you send in an inquiry. Overall, I think SurveyMonkey is the best choice for beginners given how easy it is to use and the wealth of educational assets. Once you’re ready to spend a bit more, SurveyMonkey offers excellent intermediate survey question formats as well as very attractive, easy-to-use reporting.

SurveyGizmo

At Mantis, SurveyGizmo is the most common platform we use and we’re really pleased with it to date. We find the conditional logic very easy to use. The basic reporting functions work nicely. (You can share reports with your team straight from the platform.) And SurveyGizmo’s library of explanations about any and all matters related to survey design manages to explain difficult concepts very clearly.

Our experience using SurveyGizmo support has also been positive. (We released a massive survey the week before GDPR went into effect and I reached out to SurveyGizmo explaining my concerns. They checked in with me every day to ensure I was getting all the information I needed and had access to their new GDPR features. Overall a superb customer service experience.)

Qualtrics

For social scientists, Qualtrics is the tool of choice because of its advanced features and how flexible it is–though keep in mind the price reflects those extra bells and whistles. Why pay more for a research tool?

Qualtrics has features the other players don’t. Among the features we fell for:

  • An easy-to-use scoring feature for surveys that require you to develop “personas” based on how people responded to multiple questions
  • An assessment tool that allows users to take your survey, and then immediately compare themselves to their peer group (responses you’ve collected previously).
  • Qualtrics’ user interface, which is beautifully clean and intuitive. Even though it has a massive number of features others don’t, they don’t clog up the dashboard you use to design and test your surveys.

Where to go from here

Deciding which survey tool to use takes some effort, but here are a few additional tips to help you make your final decision:

  • Many of the enterprise tools force you to sign up for a 12-month commitment, so consider what will suit your needs now — and a year from now.
  • Ask if your company has a research license, owned by HR department, customer service, or user experience/IT. Make sure you’re not double-paying before you purchase a research license for marketing.
  • We love using G2 Crowd to evaluate technology tools, and its reviews of survey tools are excellent. Consider crawling user reviews before making a final choice.

So now that you have all this info … it’s time to go shopping. Let us know which tool you choose and why!

And if you are conducting your own research, check out our comprehensive guide on how to create survey-based research.

Posted in

Clare McDermott

Clare McDermott is the co-founder and chief research officer at Mantis Research. Before founding Mantis, Clare was chief editor of CCO magazine. Follow her at @clare_mcd.

Leave a Reply

Your email address will not be published. Required fields are marked *