Survey Methods: Connecting with a Global Audience

Mobile-first users, multiple language needs, and low-internet penetration are critical factors to consider when you’re planning an international survey. Surveys are an effective way to gain insight into trends that are harder to see in qualitative research, such as focus groups and individual interviews, but there are other factors to consider when researching specifically in developing economies. The best survey method for your research depends on which population you’re trying to understand. Standard methods—online, in-person, SMS—can still be effective, but depend on your team’s goals and the populations you’re studying. For example, if you want to reach a shopkeeper who owns a feature phone, online surveys are not a good fit. But if you want insights from millennials, social media and other online surveys work well. Let’s take a closer look at survey methods, and the process of conducting UX surveys from beginning to end.

Before you begin

Regardless of the context in which you’re surveying, best practice is to conduct preliminary user interviews before developing hypotheses. During this process, your team will transform generalizations and assumptions about the market into user-focused questions. Once you have your hypotheses, you can modify research questions into a survey questionnaire to share across your team for feedback.

Before you begin the interviews, conduct a cognitive “think-aloud” by having two to three participants read through the survey and answer the questions. Ask your participants how they would answer each question, whether there is any confusion, and if they would change anything about the survey.

If a respondent re-reads a question or pauses, this may indicate that the question isn’t clear. You should also test questions with local language experts. For example, in some cultures “you” refers to the whole household rather than an individual. Local lingo for technology terms may also differ. For example, in East Africa, the term for a buffering cell phone is “scratching,” and in India, “netpack” denotes mobile data.

Choosing a survey method for your study

Online surveys

If your target survey sample has reliable access to desktop internet, or your survey is specifically for desktop internet users, an online survey is an efficient method. Online surveys are relatively fast, inexpensive, and can be administered remotely. You can collect and analyze data in real time, and you can reach many people in different locations all at once.

In-person surveys

In-person surveys are an effective way to reach communities that are more comfortable talking to an interviewer face-to-face, or groups with limited internet access. Researchers conduct in-person surveys based on the locations of each community by interviewing people in homes, restaurants, cafes, markets, public transportation platforms, and public plazas. In-person surveys are more costly and time intensive than online surveys, but they’re worth the investment to reach otherwise underrepresented populations.

If an in-person survey is the best approach for your study, it’s important to work with a local research agency that can help select a representative cross-section of cities and demographics to survey. Local partners also have the context to provide feedback on your hypotheses and questions. To avoid misunderstandings and incorrect translations, hire people that are native language speakers to conduct the surveys, keeping in mind that there may be several different languages spoken within a single country.

In-person surveys require transcription of handwritten data into digital documents for further tabulation and analysis. Consider whether electronic devices, such as tablets, are the right fit for gathering data. While devices can make note-taking easier, take into account the biases electronics may introduce, in addition to connectivity or battery life issues in the field. In these cases, a paper form might be more appropriate.

Phone and SMS surveys

Phone surveys may be a good option when questions are short, closed-ended, or targeted. Due to the growing number of phones in developing economies, deploying surveys via phone calls and SMS are increasingly viable options. Interactive Voice Response (IVR), or voice-based survey, is a friendly option for participants with lower literacy, or those who may be new to using the internet.

You can conduct a phone or SMS survey by entering a list of phone numbers and a pre-programmed questionnaire into a service like EngageSpark. The service intercepts the participant at a specific time, presenting the questionnaire via a phone call, or a URL via SMS voice call or SMS link. Some people assume incoming voice calls and SMS are spam, so weigh the possible limitations (lower response rates) and benefits before choosing this method.

Timing, bias, and other considerations

Surveys can be time and resource intensive. To ensure accurate results, your team should run a pilot day where a researcher reviews the results after the first survey hour or full day with actual users. A pilot day ensures that the questions make sense to participants and that their responses align with your expectations. For in-person surveys, researchers can ask local partners if they noticed any confusion, and if the length of the survey seemed reasonable. Do people abandon too quickly? Are open-ended questions not getting responses? Are too many respondents choosing ‘other’? Indirect translation is another common error that occurs when translators interpret the response instead of recording the words verbatim. This error can also be corrected during a pilot day. If the data looks skewed, pause the survey and fix it, then re-introduce the updated survey with actual users.

During the survey deployment and analysis, be mindful of various factors that may affect quality of data: a festival, national holiday, or other important event might influence the response rate and results. Be on the lookout for satisficing, or speeding through a survey in order to finish it rather than taking time to give the best response. Other factors to be aware of include acquiescence bias, where respondents may be likely to agree to any statement, and the cognitive load of lengthy or complicated surveys. You can offer incentives to survey participants, but choose gifts that are useful to the demographic you’re studying. For example, online gift cards may be of little use to rural communities.

Research in developing economies requires constant reevaluation of best practices. With careful planning, clear objectives, and the best survey method for your target sample, you’ll create a fuller landscape of people.

Interested in making products for the next billion users? Check out our full collection of guides.

Thanks to the Google designers and researchers who contributed to this article: Adrienne Sussman, Nithya Sambasivan, Ted McCarthy, and Mike Brzozowski. For additional articles on Google’s global methods, check out UX for the Next Billion Users.