LRQs are a product success best practice that few use enough. Let’s change that.
Today we’re talking about “Long-Running Questions” or LRQs. LRQs are something that every single SaaS product should utilize. No exceptions.
If you’re reading this in 2015, and you just you took a break to go Googlin’ for “LRQ”, you’re probably not going to find much. That’s because we coined this term internally at Ramen and this is the first time we’re using it publicly.
An LRQ is a single question that you ask your end users, either in-app or via email, as part of the natural course of their usage of your product. LRQs all share a few characteristics:
- They are composed of a single question. “Take 5 minutes to answer this survey” is not an LRQ, it is awful.
- They are asked of logged in users. “Would you like to chat with a sales rep?” is not an LRQ.
- They are asked automatically. Users are automatically asked at predefined points in their customer life-cycle.
- They are asked continuously. Sometimes for the entire life your SaaS product.
- They are “long running” from your perspective. LRQ responses are reviewed regularly with a specific eye towards monitoring changes in how customers respond over time.
- Their purpose is strategic rather than tactical. The goal is to monitor the overall health of a customer base over time while leaving a little leeway for serendipitous discovery.
The most popular LRQ is NPS
Most of you probably knew where this was going right off the bat. Net Promoter Score, or NPS, is by far the most popular–and to my knowledge the only rigorously defined–LRQ. For those unfamiliar with NPS, click here to learn more.
How NPS works
You start off by picking two interval periods: initial and ongoing.
- After initial days, you ask your customers the following question:
- After answering, the customer is usually prompted to “Tell us more about why you chose X” in a follow up question, but that’s not required.
- Every ongoing days after that, you ask your customers the exact same question.
- Every day, take all the responses–ignoring any people who dismissed the survey without responding–and do the following math:
- If they answer 9 or 10, label them as a Promoter.
- If they answer 0 through 6, label them as a Detractor.
- If they answer 7 or 8, label them as a Passive.
- Get the % Promoters by dividing the # of Promoters by the total # of responses
- Get the % Detractors by dividing the # of Detractors by the total # of responses
- Subtract the % Detractors from the % of Promoters, remove the % sign, and you’re left with a plain old scalar number between -100 and 100. This is your Net Promoter Score.
Net Promoter Score is just that: a score. It’s not a percent. There are not passing and failing grades. 60 isn’t necessarily good and -10 isn’t necessarily bad. People who talk about “needing to bump their NPS up a few percent” are not only using the wrong terminology, but they’re probably thinking about it all wrong to begin with.
What’s really important with NPS is how it changes over time. See the NPS start to tick up recently? You’ve probably been doing something good.
A new marketing initiative that goes after a brand new persona, for example, could cause a flood of new, really excited customers into your product. But if, 45 days later, it turns out that the language in the marketing copy incorrectly led people to believe that you had a certain set of functionality that you didn’t, they’re going to churn at very high rates. In this pathological example, you could have changed nothing in your product, and still over the course of 8 weeks seen a spike in NPS followed by a spike in churn.
This makes NPS seen as misleading/dangerous/etc by some.
Opinions on NPS aside, LRQs are a vital tool for product creators
Whether you love or hate NPS, it’s important to remember that what really matters is that you have some set of LRQs to act as a proxy on the pulse of your customers’ attitude towards your product. Every SaaS product should have at least one (but we could make an easy argument for three) active LRQs that they look at every week. What matters is that you regularly look at the results and start to get a “feeling” for the data.
What makes an LRQ
Before we get into what types of LRQs you should be asking, I want to talk about two distinct “families” of LRQs: one-time and periodic.
One-time LRQs are asked transactionally. Tweet this Examples include right after a user signs up, after they’ve been a customer for a certain amount of time, or after they’ve completed a certain workflow. The idea here is that the customer is only asked the question one time.
Periodic LRQs are asked repeatedly. Tweet this They are asked initially after a certain amount of time, and then at a regular interval thereafter. A valuable characteristic of periodic LRQs is that you can monitor for customers that change their answers over time. NPS is a periodic LRQ.
Because of the repetitive nature of Periodic LRQs, there is a lot of nuance to how they are implemented.
Periodic LRQs are “rolling”
They need to be asked in a rolling fashion, so you don’t have a “survey day” where you ask everyone that logs in on a certain day the question. This opens you up to several biases. If the majority of your customers are in SF, and the Giants lose an NLCS game in extra innings late last night, your customers might not be in a super good mood, and that can really affect your numbers.
Periodic LRQs aren’t for brand new customers
Unlike one-time LRQs, periodic LRQs are rarely asked of brand new customers, as the context of their answer will dramatically change by the time they answer the second time. You want to give them time to reach a “steady state” or else you’ll always see huge changes in their answers once they start getting the question for the second time.
Periodic LRQs are asked “for no reason”
Let’s say you want to establish an LRQ to monitor the “health” of your checkout process, and you set it up so that the question will appear after your customers finishing buying something. You don’t want to survey them every single time, so you set up the question to only be asked if it’s been 60 days since the last time they were asked.
However, in this scenario, you’re only asking people the question after they’ve successfully completed a transaction. Maybe your process isn’t actually very good. Maybe your workflow is so confusing that people abandon in many cases. I’m painting a pathological picture here, but it illustrates the point that randomly asking customers who have completed a checkout process:
Hi Matt, Over the past few months, have you found our checkout process to be a satisfying experience?
Is much different than asking them right after they finishing buying something.
Ok, if you’re still reading, bravo. This is what you’ve been waiting for. Below, we’re going to go over some LRQs that you can start asking right now, and follow up with some tools to help you do it.
One-time LRQs include questions like:
“How did you find us?”
Answer: Multiple choice (which is preferred and will change based on your product) or open-ended
When to ask: As soon as they complete the signup flow
“What are your first impressions?”
When to ask: Once the customer has had a chance to get a first impression. This could be within 10 minutes of signing up, or could be after a week. It all depends on your product.
Periodic LRQs include questions like:
“How likely is it that you would recommend Ramen to a friend or colleague?” (NPS)
Follow-up question: “Tell us more about why you chose that”
Follow-up answer: Open-ended
When to ask: Depends on your product, but Ramen asks our customers after 30 days and then every 60 days thereafter
“Anything frustrating about the app you’d like us to improve?”
Answer: Yes or No
Follow–up question: “Tell us more”
Follow–up answer: Open-ended
When to ask: Up to you, but Ramen asks this question every 45 days.
Should LRQs be asked in-app or via email?
This is a tough one. For most SaaS apps, in-app gets you a lot further, as questions can be asked in context. This allows them to be shorter. It also affords much higher conversion rates. We’ve seen in-app response rates around 40%, where email response rates are ~ 15%.
However, many products have usage patterns that make in-app not super useful. These include situations where your customers don’t interact with the product in their browser frequently, but maybe simply enjoy a daily or weekly email. In this case, you probably want to lean towards email questions.
Analyzing LRQ Results
Once you start asking these questions, you have to do something with all those amazing data. You’re technically inconveniencing your customers by asking them these questions, so make that worthwhile.
Get into a regular cadence of reviewing answers to questions. Get a weekly email setup that sends you the following for all your LRQs:
- New answers last week
- Answers from last week
- Answers from all-time
Obviously you want to see the most recent answers, and you probably want to see answers from all-time, but you’re going to also want to compare the most recent answers with the precedent time period (read: last week vs. two weeks ago)
For periodic LRQs, your weekly report should include the # of people who have changed their answers since the last time they answered. This is a lot easier said than done, and in most cases will require some custom development on your end.
This week, we’re going to break up our toolbox into sections: E-mail and In-app. For earlier stage companies, or businesses without a strong technical person, email will probably be a lot easier to setup (unless of course you’ve already installed Segment.com and want to use Ramen ;)).
- SurveyGizmo & SurveyMonkey – Both offer free tiers and make it easy to send surveys via email.
- Delighted & Promoter.io – Very similar products that provide an email-only NPS solution. However, as of writing this post, their free tiers are not very generous.
- Wufoo & Google Forms – Just like a couple weeks ago, Wufoo & Google Forms can do a lot for free. Although the UI of popping up an iFrame w/ a Wufoo/Google form would leave a lot to be desired, and it would leave you to do all the NPS math on your own.
- SatisMeter – A very straight-forward tools for implementing in-app NPS. They have a free tier, but it’s not terribly generous.
- Intercom – Intercom can be shoe-horned to do this, but you’ll have to do all the math on your own.
- Ramen – Here’s the thing: the lack of a proper solution to do this is pretty much the reason I started this company. So, I’m not going to belabor the point: we have NPS and generic LRQ support, and I’ll just leave a little link here in case you want to try Ramen out. TL;DR: These LRQs are tough to get right, and as you can probably tell from this extremely long-winded post, our team has spent a lot of time thinking about it.
Product Fail Yeti says:
Those options are way too much work or too expensive! Ain’t no Yeti got time for that! Forget everything you just read and just keep building your product without any of these EL ARE QUEUE things this dude is talking about. Too much work. #Grump
Can you believe that guy? Don't listen to the Yeti. #FightTheYeti
In the wild
Here are some great examples of companies using LRQs
Mention.com has extensively used NPS to decrease their churn. Here is a presentation their head of growth, Guillaume Cabane, has done on the topic.
After reading a draft of this post, Guillaume had some great comments which he gave me permission to post here:
We discovered that you can’t handle answers from your free and paid users the same way. And you if you do, you shouldn’t. Because someone paying you hundreds of dollars a month and leaves a text comment, that needs to be handled by a human being.
Never lie to your customer. If he thinks (because you smart marketer let him believe that) there is a human being reading / replying to his survey, then you need to be able to handle the volume of answers. We generate north of 1,000 NPS surveys per month at Mention, and though it’s some work, it’s definitely worth it.
SugarWOD has a native mobile app for tracking CrossFit workouts. They also have a web UI for gym owners to schedule workouts and monitor their athletes’ progress. A question they’ve had in the wild for a while is about getting first impressions:
Among some of the most interesting answers are:
“So far an excellent program. Before looking at investing I am waiting to see what kind of following this will gain in my gym.”
Drew Larsen, founder & CEO of SugarWOD, had this to say about that prior piece of feedback: “Knowingly or not, this guy shared with me his biggest concern / objection / trial criteria. I followed up with some rollout tips, some stats on what kind of adoption to expect and created a “to do” to follow-up with him on this specific topic after a week.”
Another piece of feedback that Drew received from this LRQ:
“Hey guys – quick question. Do you see this platform expanding to emulate services as seen on WodHopper, Wodify, FrontDesk, ZenPlanner, etc?”
From Drew: “This prospect is asking if we are going to provide complementary tools from an adjacent software category. We’re not, but it was really helpful to know so I could pro-actively address his issue. He is now a customer.”
Final Thoughts on LRQs
I hope you found this post helpful. These types of questions are hugely beneficial to product creators, and I urge you to start asking them regularly through whatever means available to you.Wrong form ID