If your customers don't talk, NPS is a vanity metric

nulbyte

I would go a step further: NPS is a garbage metric.

First, you start by assuming your customers can even reasonably ascertain their likelihood to recommend. They can't; there are people who answer 10 but will never recommend, and there are people who answer 0 but already have and will again.

Next, you assume your customers are idiots and don't know how an 11-point scale works by adjusting the midpoint: Instead of 5, the middle is now 7 and 8.

Then you realize there are two many numbers, so you throw several out by reducing your 11-point scale to a 3-point scale, after which you re-interpret "unlikely to recommend" as "likely to snag some other customers on my way out the door."

Finally you calculate your 'net promoters' by subtracting the percentage of low scores from the percentage of high scores to give you a nice round number that doesn't correlate with what's actually happening in the real world.

And this is just what happens when you do it 'the right way.'

NPS is said to measure growth using loyalty as a proxy. But then, what does that have to do with recommendations? Nothing.

gyomu

Bahaha oh man this brings back memories. I worked for a startup where the CEO spent weeks working on determining our NPS and talking our ears off about it. At the end of the process, he was so happy - our NPS was higher than Apple! I didn’t really know what that meant but I figured he had an MBA and knew what he was doing.

The startup never became profitable and ran out of investor money 18 months later.

There were many things wrong with the company but this was one of the things that made me most feel like I was in Office Space.

pflenker

Not defending NPS here - I don't use it - but some of your assumptions are wrong.

> They can't; there are people who answer 10 but will never recommend, and there are people who answer 0 but already have and will again.

What matters with NPS is trend over time, and getting the numbers at a scale. Yes, there are people who randomly click on one end of the scale or the other, but the assumption is that on average the portion of these people is stable.

> Next, you assume your customers are idiots and don't know how an 11-point scale works by adjusting the midpoint: Instead of 5, the middle is now 7 and 8.

This does not come from the assumption customers are idiots, it comes from the idea to treat people who vote "in the middle" not as neutral, but as detractors. Which makes sense: If someone tells me "hey I know Product X and it's meh", then I'm less a promoter but more a detractor.

> Then you realize there are two many numbers, so you throw several out by reducing your 11-point scale to a 3-point scale

The 3 point scale was the goal all along though, it's the idea of an asymmetric scale that leads to the 11-scale to 13-scale reduction.

> after which you re-interpret "unlikely to recommend" as "likely to snag some other customers on my way out the door."

If your assumption is that promoters drive positive growth, it's fair to assume that detractors drive negative growth by recommending an alternative. If you believe in that core assumption that NPS measures word of mouth, then this interpretation of "likely to snag some other customers on my way out the door" is a sensible one.

> NPS is said to measure growth using loyalty as a proxy. But then, what does that have to do with recommendations? Nothing. I don't think the underlying assumption is bad. That's how influencers work: people are more likely to buy something that is being recommended to them by someone they trust and someone who is passionate about the product.

Does NPS work? I don't know - I'm not using it as I said above. But at least the assumptions under which NPS are designed on top of the idea of word-of-mouth as a growth diver seem solid to me.

jonahhorowitz

I occasionally answer a customer support survey, just because I know the person on the other end probably gets to keep their job or not based on some average score, but I am not wasting my time answering a NPS survey for some random app or company that I've used. It's just not worth my time and I get too many surveys to care.

pflenker

At scale, a relevant number of your customers talk. One example in TFA is that customers do not have conversations where they randomly recommend operating systems to each other. Well, I'm sorry, but in my bubble they certainly do. I have recommended more operating systems than car models to others.

The other thing the article misses IMO is that detraction is also growth, albeit negative growth (and additionally, people are in my experience much more likely to passionately recommend _against_ something they hate than recommending something they love). So the NPS tells you a thing or two about: - Your potential to utilise whichever chance you have to grow via word of mouth - Your potential to squander that same chance due to people hating your product - Your potential to have negative growth because your customers are leaving in droves.

FearNotDaniel

I’m inferring that NPS here refers to “Net Promoter Score” [0]. Presumably within his niche, the author can assume his readers already know that.

[0] https://en.wikipedia.org/wiki/Net_promoter_score

forgotoldacc

I never knew this was an actual "thing."

Like, I've seen these for years. It feels like 5 seconds after I open any given website a "rate us on 1-5" popup shows up right in the middle of the screen. I assumed it was just some thing that's automatically thrown in just to annoy users and has no practical purpose, like the cookie warnings (that are ignored regardless of what you select), the email spam requests (which nobody reads despite what people claim [if someone out there wants to get angry and claim their daily spam emails and annoying popups do good for their business, go ahead {I will laugh at you}]), and the "subscribe/follow us on (social media)" that plague every site these days.

Knowing some manager is thinking this is valuable info and this may decide their job is just hilarious. I used to randomly pick numbers just to dismiss it but now I'm motivated to actively mess with them.

smcin

Marketing executives often have it as one of their KPI.

Clearly the less fuzzy way to define it is "How many people have you recommended this product/service to in the last n months?" not "Would you, if asked?"

Also, whether customers talk to each other (directly, privately) is not the only thing, there are online reviews, resources like HN, etc. And it's always hard to tell what's compensated or not, e.g. Snowflake's "grassroots" testimonial campaign on LinkedIn, testimonials etc., with the huge budget.

floydnoel

I would have expected everyone to have heard of it by now. I first saw it mentioned on HN in 2017 or so. Good reminder that there's always newer folks around and to define the terms!

ggm

I work in a not for profit functional monopoly space. A member body distributing resources according to policy.

I can get why we do satisfaction scores, but NPS never made sense to me. Like, does the justice system do NPS for family law cases? It's not like you can get divorced in a 7-11 by the clerk, and even mediation is a process inside the model so NPS in a non competitive self-regulated monopoly.. what does it even mean to recommend to others?

xnx

NPS, Myers-Briggs Type Indicator, and Astrology are all things that seem scientific, get a lot of attention, and are junk.

shric

As an INTP Virgo who's unlikely to recommend products to your friends or colleagues, you would say that.

tbrownaw

The problem with MBTI is that reduces a continuum to a binary for each of the letters. IIRC, the basis for what each pair of letters means isn't any worse than any other personality test.

Astrology isn't about stars. It's about cold reading if done in person, and about the art of writing descriptions that sound specific but actually apply to pretty much everyone.

NPS is a way to reduce a histogram to a scalar.

NikkiA

At least astrology has some interesting historical background.

That is the only redeeming quality of any of them, however.

superjan

In stead of asking “would you reccomend”, shouldn’t you rely on “how did you learn about us?”, or “why did you buy our product”, the first one being the most reliable, as it is a neutral question about a fact.

SanjayMehta

I never understood the logic behind that asymmetric ranking scale.

Around 16 years ago our CEO was talked into using this. After 3 quarters of using it, he killed it because it wasn’t making any visible change in sales; the only metric he cared about.

smcin

Essentially it says few customers will move the needle on bringing you new business, only the ones who rave wildly about you. Hence, "Customer obsession" as Amazon's Bezos used to put it.

Since NPS and changes in NPS is measured by surveys, not attributing individual sales, it's a KPI that's gameable (especially when used on its own); near-impossible to tell whether "improving" NPS in some existing-customer segment from '5' to '7' results in anything tangible. On the other hand you can actually measure sales and attribute which channel they came from.