blog_promo_old_6

Why Doing In-App NPS Surveys Is a Bad Idea

I have made myself pretty unpopular lately by saying that In-App NPS is a huge mistake. Especially because many customers that are coming to Promoter.io ask us about doing an in-app version.

It’s hard because to an extent, the customer is always right. If there is demand, then as a business, it’s natural to want to fulfill that demand. And in fact, various competitors of ours will let you do In-App NPS surveys.

But it is also my responsibility to provide the best NPS platform possible. A platform that has the best opportunity to help businesses transform and get the most from their NPS process.

So let me try to explain why in-app NPS surveys are such a bad idea.

1) In-App NPS Measures the Wrong Score

At first, in-app NPS seems to make a lot of sense. After all, when better to gather feedback about your app than when your users are actually using it, right? Seems like an easy and fast way to get a lot of data quickly.

And it can gather a lot of data quickly. The problem is that it’s the wrong kind of data and the quality is questionable.

Think about what NPS measures: the loyalty of your relationship with your customer. How your customers feel about you in the moment they are using your service is one way to measure that. But it’s a skewed look, a fairweather customer. Nobody wants to tell you the whole truth when they are right in front of you.

But how the customer feels about you a few days or weeks later, in the perceived privacy of their own email inbox might be a lot different. And a lot more honest.

An in-app NPS survey will measure how they felt about the individual transaction or workflow they just had. But an email NPS survey (a true relationship measurement) a few days or weeks later will measure how they feel about your brand in overall…not just a single interaction with it.

And measuring brand loyalty is much more important and a far stronger predictive measurement than transaction loyalty.

2) In-App NPS Focuses Too Much On The Score

As we have shown before, the real magic behind NPS isn’t the survey, it’s in how to respond to survey results. The focus on In-App NPS is on calculating a score, which alone is not very actionable information. Knowing that your score is +20% or -40% is one thing. Building an authentic fan-base of customers who are ready and willing to promote anything you do is something entirely different.

And that’s completely possible to do with NPS surveys done correctly.

In-App NPS surveys are impersonal in nature. It’s just another button to press in yet another website. Most people spend less than 15 seconds on any website (and even most apps) per day. Do you really want 5 of those seconds to be used up taking your un-actionable In-App NPS survey?

On the other hand, one might argue that email-based NPS surveys won’t be seen by half of your customers who never open the email. That’s true, and you might be missing out on some opportunities there. But if a customer isn’t opening emails from you, how likely do you think they will be to be active engaged promoters of your brand?

Email NPS shows up in one of people’s most private places: their inbox. And when you respond to their survey with a personal follow-up that shows you read what they had to say, it becomes ultra-personal very quickly. That’s how you cultivate true loyalty and action: through personal touches.

3) In-App NPS Is a Turn Off and Feels Intrusive

Think about your personal experiences: How do you feel when you’re presented with a survey that interrupts your workflow? That’s probably how your customers feel as well.

When they are using your app, they are trying to solve a problem. Why are you trying to get in their way from solving their problem? By doing so, you are shooting yourself in the foot.

This leads to high dismissal rates and low open text feedback (if any) from those who do score. Many of the scores are questionable as well as some customers/users will just click anything to dismiss the survey.

Aside from the various other benefits we have already touched on, email is a much less intrusive way to gather the feedback you are looking for. Especially when you are using a system like Promoter.io which allows your customers to send their feedback in with a single click and at their convienence.

Conclusion

Technically, we could build an In-App version of Promoter.io quickly. It’s not a big programming challenge. But it is a direct challenge to the core principles we believe in.

We believe that NPS can be a key growth-driver for organizations who put it correctly into practice.

We believe that NPS surveys should be the beginning of conversations with your customers, not the end of them.

We believe that NPS should not be about the score as much as it is about creating new growth opportunities for any business.

And in the end, in-app NPS does not align with any of these goals. If we figure out a way to create an in-app NPS survey that does align with these goals at some point in the future, we might change our mind. But for now, we will leave In-App NPS for the score chasers who haven’t yet realized the true potential of NPS.

5,478 Views

Chad Keck

As a product lead and executive for numerous successful ventures (Rackspace, HP Cloud, AppFog), Chad founded Promoter.io to help bring the actionable insights provided by Net Promoter to all businesses. He is a native Texan with a passion for helping other entrepreneurs.

Website

  • https://ramen.is Ryan Angilly

    FWIW, Ramen’s (https://ramen.is) in-app NPS sees great response rates & balanced responses. I agree whole-heartedly that _transactional_ NPS is a bad idea (ie. “You just finished buying something! Do you like us?” is so biased it’s silly) but to make the blanket statement that in-app NPS is a bad idea is a stretch.

    The post makes some claims that seem very back-up-able-by-data, but there’s no data in the post. I’d love to see that. Might affect how we do in-app NPS at Ramen.

    Thanks!

    Edit: added “& balanced response rates”

    • http://www.promoter.io/ Chad Keck

      Hey Ryan (@angilly), thanks for the comment and perspective. Glad to have an open discussion.

      As I mentioned in the article, it really comes down to the type of sentiment you are wanting to capture and how accurate/predictive it is overall at the brand level. When you prompt a customer/user inside of an app, the feedback tends to be very specific to the action (or part of the app) that customer is using, or a specific feature/workflow. Some may respond with sentiment towards the brand, but of all the data we’ve seen (first-hand and from our customers who’ve switched) it’s highly focused.

      Now that isn’t necessarily a BAD thing in general. It can certainly be good data to capture, but that is not the purpose of NPS. Net Promoter is a metric and system designed to capture sentiment towards the brand and predict customer behavior based on that data (and to drive deeper customer relationships). If you are capturing sentiment that does not line up with how they feel towards the overall brand you will end up with a skewed picture of your customers and who your promoters/passives/detractors truly are. This can severely impact the value of your efforts.

      Now that said, I think I could have done a better job of explaining one other thing. It’s not necessarily one vs the other (relationship vs transactional, email vs in-app). In-app can be very useful as I mentioned above, but it’s a different perspective and the data should be looked at and analyzed much differently and completely disconnected from your relationship NPS.

      Here is a quick example:

      I’m a huge Apple promoter (or “fanboy”, whatever). Not many people I know have owned more Apple products. That said, I had a horrid experience at the Genius Bar the other week. Not enough to sway my convictions since I’ve also had good experiences in the store, but definitely negative. Am I still a promoter? Hell yes, but if they had prompted me with a survey in the store (especially during my interaction), it would have been the lowest score possible.

      Now think about that. If that was the only point of engagement with a survey (NPS or otherwise), how would Apple consider me? Or any other brand in a similar situation? I’d be in the detractor pool, but that wouldn’t be correct. I’d of course like a proactive follow-up so a survey sometime after isn’t a bad thing, but it should never replace that routine outreach about my relationship with the brand.

      Some companies are very strategic with their Promoters as they should be to drive ACTION. If I was dropped from the promoters group and wasn’t asked to do something to benefit the brand, they would be missing out.

      Sorry for the book but I hope that makes sense. Also, I’m happy to see about sharing some data from our customers who’ve switched how they measured NPS (from in-app to email). We’ve had dozens if not more that have come to Promoter and we took a close look at their results which was why we decided to write this post. To be fair, I don’t know that we’ve had anyone switch from Ramen, but we definitely have from other services.

      End of the day, there is a use case for both, you just have to know what to expect from the data and how to use it to drive value for your company.

      • https://ramen.is Ryan Angilly

        Massively appreciate the thought and time that went into that reply. I don’t disagree with any of your anecdotes. I still disagree with how you unequivocally stated “don’t do in-app NPS.” This is too complicated of an area for black and white answers.

        BTW, you definitely haven’t had anyone switch from Ramen; we’re still pretty young :)

      • Andre

        I have to disagree with you both on the value of conducting transactional surveys. Both transactional and strategic surveys have different functions, and their usage depends on what your objectives are. Companies may well want to know how customers feel about their brand, however, they may be keenly interested in improving the performance of a specific touchpoint along the customer corridor.

        Using your Apple example, it may the manager of the specific retail outlet who is looking for information on performance improvement, and specific information of your dissatisfaction at the Genius Bar is much more useful that your overall loyalty to the product, over which he/she has much less control.

        In addition, measuring sentiment at the transaction level allows for timely service recovery. Although you may be a brand promoter, your negative experience with the Genius Bar could generate negative word-of-mouth over the incident, and had an influence of someone else’s intention to purchase at that retail outlet.

        • http://www.promoter.io/ Chad Keck

          Hey Andre,

          Thanks for the comment. To clarify, I never said that transactional surveys were bad and that a company shouldn’t use them.

          My point was that a transactional survey should never replace a brand level sentiment measurement and it should not be looked at through the same lens. The specific transactional data can definitely be useful, particularly to those on the front-line trying to improve, it’s just not “core” to the NPS process.

          NPS may not even be the best approach to transactional surveys, but some people find it to work well.

          As long as you understand that transactional data isn’t as reflective of overall brand sentiment and that it may not be as predictive either, there is still a ton of valuable data to drive from it.

          Hope that helps to clarify my position. Enjoying the discussion :)

          • Andre

            Hey Chad,

            Understood.

            My personal experience working for a global brand, is that a regional operation sees little relevance in a growing or declining brand metric and its drivers, if their processes or team behaviours have no strategic impact on the product.

            What’s more important to me is those “moments of truth” that uncover training or operational triggers, and our responsiveness to close the loop with customers. That way I can drive tactical changes from day one.

            I agree that transactional surveys aren’t the best approach for brand strength, but I also believe that strategic NPS isn’t the most useful tool for that purpose either. NPS in my opinion is about service excellence, and while that forms a part of a company’s brand world, there are many other elements to consider.

            Both transactional and strategic surveys have their place. However, if you’re in an operational leadership role, transactional surveys are way more actionable for tactical improvements and team development.

            But of course, that’s just my opinion :-)

  • Sarah

    Chad, I had to laugh at this–

    Think about your personal experiences: How do you feel when you’re presented with a survey that interrupts your workflow? That’s probably how your customers feel as well.

    because I don’t think you are following your own advice on not interrupting people! While reading this article I got prompted two times with modals to sign up for emails.

    • http://www.promoter.io/ Chad Keck

      Hi Sarah,

      I understand the point you’re trying to make but that is a bit of a reach to be fair. In-app NPS surveys and a newsletter subscription box on a blog are not one in the same.

      There is plenty of evidence to show that pop-ups for engagement on blogs works well.

      We however would never interrupt a customers workflow while they’re trying to actually use Promoter and get some work done.

      /shots fired :)

      P.S. Sorry about the double pop-up. We switched to a lower-right modal to be less distracting and forgot to disable the main one. Thanks for the tip!

      • Dale Halvorson

        Chad,

        Have you done a test of the new sign up box. Had I not read this reply I would have NEVER seen it. It’s like fuzzy orange box in my peripheral vision. Every heat map that I’ve seen of websites leaves the edges where ads typically are cold blue wasteland…

  • http://www.promoter.io/ Chad Keck

    Hi Jessica, thanks for chiming in. In response to your first comment I probably could have phrased that better and just because someone may decide to measure NPS in-app doesn’t inherently mean they are a score chaser. That said, from dozens of customers we’ve seen who switched from in-app measurements, the open-text response rate is always extremely low (both compared to Promoter and other email based solutions). That is a big part of the reason people switch so I wanted to point it out in this post. Not a blanket statement, but a consistent observation we’ve seen in practice.

    Based on a lot of direct feedback we’ve received on this post today and some of the comments here I think we’ll be doing a much more in-depth look at this topic with some more specifics to share.

    I won’t re-write everything I shared in another response, but take a look at my comment to Ryan above. It’s not that in-app NPS measurements are BAD, but there is a very distinct difference in the type of feedback and sentiment you’ll receive based on where you engage a customer. One is more transactional focused that the other and less predictive than a relationship measurement. That is the main point here.

    Can they be used together? Absolutely, and I think that is something a lot of companies should work towards (email + in-app, if applicable), but an in-app measurement alone is not a replacement for routine relationship engagement which focused on the brand/product relationship instead of what a customer happens to be doing right when you prompt the survey on them.

    One of the biggest concerns I have that is shared with many customers who’ve switched to Promoter is the high dismissal rate of in-app surveys and inconsistent scores (once they try to reach out). Basically customers clicking on something to dismiss the survey.

    Not trying to attack any specific product including Wootric, so apologies if it came off that way. Just wanted to provide some clarity to a common topic that we get asked about A LOT and explain the difference between where you choose to engage a customer.

    The results will NOT be the same. That isn’t bad, just as long as you understand why and how to act on both types of feedback, and apply those insights to your business.

    Happy to discuss more if you’d like. Please reach out anytime.

  • https://satismeter.com/ Ondra Sedlacek

    Hey Chad,

    After I saw this post and the discussion I had the urge to write down a post from a point of view of someone who provides both email and in-apps: http://blog.satismeter.com/post/132204284798/in-app-or-email-which-nps-survey-is-better.

    I just don’t agree with your main point that email surveys are somehow overall brand relationship, as opposed to in-app being strictly transactional feedback.

    I think it depends far more on the type and usage of measured service.
    Imagine a SaaS service that customers use daily in their work. There just isn’t the difference if you ask them in-app or via email, if the last moment they worked with the service was half an hour ago. As we run both email and in-app survey, our data back this up, there is no significant difference between email and in-app, neither in overall NPS nor in verbal feedback!

    Of course there are types of services that email surveys are better suited for, as I wrote in the blogpost, but you just cannot generalise it with your “in-app is bad idea” statement.

    But not to be all negative, I think it is great you have opened this discussion, and appreciate the time and energy everyone here invested in it!

  • AskNicely

    I agree with Chad. http://www.asknice.ly

  • Chris Carter

    Great post. Thanks for sharing your thoughts here. What about SAAS products that have “hidden customers”?

    An email survey could target the contact on the account, but many times that person is fairly far removed from the day to day usage of the product. Their view could be significantly skewed.

  • Pingback: The Definitive Guide to Measuring NPS - GoSquared Blog()

  • Sam

    Are their any quantitative studies out there on in-app vs email NPS? I’ve seen others say they’ve tried it both ways and didn’t notice significant differences in the data other than response rates, but that was an anecdotal perspective. It would be really helpful to see an A/B comparison of a few industries (e.g. SaaS, eCommerce, etc.) with 1,000+ responses for each approach.

  • Willian Becher

    I agree with Ondra Sedlacek. I think it depends far more on the type and usage of measured service.

    For applications that have users with high return frequency, NPS in-app is very effective because it asks for the response at the moment when the user is thinking about his brand or product, different from the email, where he is with the thought In other things.

    In-app NPS also offers additional features to work with your customers. At http://loyalnow.com you can actually get the Promoters to take action indicating to your friends or family.

    Transactional NPS is great for less frequent experiments, such as a specific online purchase. Ideally, even send the NPS search after the customer has received the product, so it will respond based on the whole experience.

  • Bert Stevens

    We’re using satismeter and I don’t see a difference in scores / number of comments / engagement after that, etc. But what we do see is a much higher response rate….