The Airbnb Survey Teardown: Analyzing What You Can Learn By What Unicorn Startups Ask

Airbnb is one of the most iconic names for startup success in our generation, quickly becoming one of the world’s fastest growing companies with 80 million reservations booked per year through their service.

So after a recent stay in an Airbnb, I was incredibly eager to see what their post-stay survey looked like and what I could learn from it.

Airbnb has a well-known track record of using NPS to help grow their business in many different ways, but their post-stay customer survey wasn’t a NPS survey. Still there were things both good and bad that I found instructive while going through the survey.

We’ve dissected T-Mobile surveys and Apple surveys before, so today, let’s see what we can learn from Airbnb.

Dissecting The Email Subject Line

Like any good survey, this one is sent via email. We have talked before about why we believe in-app NPS is a mistake, and although this survey is not a NPS-based one, it could easily have been presented in-app if they had wanted to. But they wisely chose email as the more personal medium, creating an extra brand touchpoint that wouldn’t have existed otherwise.

The email subject line was decent as well outside of the time commitment (more on this later): “Respond now to a 10-15 minute survey”

It’s a little on the long side (remember with people mainly checking their email on mobile phones, you want to keep your subject lines ideally to 3-4 words), but the first couple words grab your interest: Respond now. It feels important and instinctively you want to open the email to see what you need to respond about.

In the email game, half the battle is in the subject line. Keeping it short and enticing is key.

Dissecting The Email Body

The email body is good as well, albeit a tad long. They provide justification and context for why it’s important to take the survey without any undue influence. That might sound like common sense, but actually very few big companies do this. And it makes a big difference.

People these days want to understand why they are being asked to do things, and also know that someone is actually listening. If you don’t give them a reason, they will probably just assume it is so that you can send them more marketing spam. After you read this email, you authentically feel like they are doing it for the right reasons.

Airbnb_email

Number of Questions

This Airbnb survey comes in the middle-high end of the pack with 15 questions. T-Mobile asks 5 and Apple asks 23 (!!). As we always say in these survey teardowns, the fewer questions, the better. Ideally, like with NPS, you should only be asking two questions to keep engagement high and focus on actionable verbatim feedback. Remember, you can always ask more questions during a closed loop conversation with an individual customer where necessary (or relevant).

For every question you ask beyond the initial two, you will empirically lose about 30-50% of your responders. With 15 questions, that means that only a tiny fraction will be completing this entire survey.

This is why the average survey response rate is around 1-3%, while Promoter.io customers often capture north of 30% response rates.

Now, if you are Airbnb with tens of millions of customers a year, maybe it’s not so bad to only expect a small fraction of them to finish your survey. That’s still thousands of completed surveys per year to sift through, although it’s not a truly representative sample set.

If you hope to have a significant number of a smaller user base give you actionable feedback, remember this rule of thumb: shorter is exponentially better.

It also drives more verbatim feedback which is the most important part of the entire process. The “score” in NPS is largely irrelevant on it’s own.

Airbnb’s Biggest Survey Mistake

Unfortunately, after starting strong on email subject and body, Airbnb quickly made some big blunders with some of their questions. For example:

“If Airbnb had not been available, what would you have done?”

“How long would you have stayed if Airbnb had not been available, given your answer in the previous question?”

Why are these questions so bad? Two reasons.

First, they are static pieces of information. Once you have asked a proper sample size of people, the answers are not likely to change in the future. When you keep asking a static question and getting a similar answer, it’s time to take it out of your survey.

Second, these questions are asked early on in the survey. Remember, every question will reduce the audience size for the next question by around 50%. So not only are you asking questions that you probably already know the answer to and are not likely to change much over time, but you are reducing the audience for more dynamic questions later in the survey.

As a general rule of thumb, with any survey, ask the most critically important questions first. These two questions break this rule twice over.

As a general rule of thumb, with any survey, ask the most critically important questions first. Click To Tweet

Overall Impression

When looked at in totality, Airbnb’s survey wasn’t quite as bad as the Apple one, but nowhere near as good as the T-Mobile experience. Overall, by the end of the survey it felt like more of an exercise in demographics than truly making Airbnb better. But as stated earlier, they are known to use NPS, so maybe that was more of the intention for this survey information.

Also of note was the lack of any open-ended feedback question, which reinforces the supposition that this is mainly for demographic information. Which is not to say there is not any value to that kind of information. Just that it misses out on a huge opportunity to turn customers into active promoting fans.

The more people come to realize the revenue growth power of a properly designed survey, the more they will start looking at these kinds of opportunities in a new light.

Here’s the survey in its entirety

2016-02-18 11.15.17 AM, airbnb

2,768 Views

Chad Keck

As a product lead and executive for numerous successful ventures (Rackspace, HP Cloud, AppFog), Chad founded Promoter.io to help bring the actionable insights provided by Net Promoter to all businesses. He is a native Texan with a passion for helping other entrepreneurs.

Website