Geoff Ralston:
We are very grateful to have Suhail Doshi from Mixpanel who co-founded Mixpanel almost 10 years ago now and is going to talk about how to measure your product which as you heard from Gustaf is really the other side of the coin of growth and everything that helps you build something that people truly want and will use. With that, Suhail.
Suhail Doshi:
All right. Hello, everyone. I'm happy to be here. Awesome. I thought I would start off by just kind of like, I sometimes try to change my presentation a little bit on the fly depending on the audience. I thought I would just ask like a really quick question, just a quick show of hands, I'm going to define like a user as like, just someone that uses your product, whether it's ... Whether a B to B company, paying free, it doesn't really matter like they did something in your product. How many people have like zero users right now? Okay, cool awesome. How many people have 100 or less? Well, okay, you have greater than zero between one and 100. Okay, awesome, great and then how many of you have like more than 10,000? Okay, awesome, all right so we've got ... I'd say, majority of people have zero users right now. Cool. Awesome, great. I'm going to start off with just a quick about me for those who haven't heard of Mixpanel, I started this company called Mixpanel, I think the first line of code was written in like October 2008 technically. I was like 20 years old. I was in my parent's house in my bedroom and 10 years later, the company has about 300 employees. We've raised a bunch of money and we have about 7,000 paying customers and we're starting to close in on around 100 million dollars in annual recurring revenue, just to give you a sense of, yeah 2008 now. Cool. Awesome. If you don't know what Mixpanel is, we make pretty graphs like this. We help you measure what people are doing inside the product so this is like, someone inviting a colleague to their app. We might help you like measure a funnel and see the conversion rate of people opening your app to like viewing an article, we'll help you measure retention. I know Gustaf went over that a little bit today. I'm just showing you this just because like everything that I talk about today is basically ... Is something that you could probably do in Mixpanel or even any other tool. There are a bunch of tools that are great to measure these things but I want you to know that if you walk out the door and you wanted to apply in a bit, you totally could. There's like a premium plan and we cater very much to startups, those are all of our early customers so feel free to do that. Awesome. Great. Today, what I wanted to dive into was I thought I would take a top-down approach to thinking about analytics and metrics, instead of kind of coming at you with charts and visualizations and numbers and things like, I thought it would be better to kind of think about the problem that you're trying to actually solve and sort of ignore for a moment that you have to do complex analysis and calculations. You will figure that out. It's not hard or you'll just use tools that will make that a lot easier for you. That's one thing that I thought I would do. I picked these three things. I tried to think pretty hard about like, what the heck did I care about when I started a company, like what matter to me? What was on my mind? I thought, well, I think the first thing is just like do people even understand what I made and Mixpanel had a really hard time with that in its first 18 months. We had to change a lot of things and most people do. The second thing is, okay, great, they kind of understand what it is but do they find it relatively easy to use the product on day one? That's kind of this never ending mission that you'll end up taking when you start your company and when you ship your product. The last thing is, I kind of try to pair it down a little bit but I thought the most important thing would be the next, are people coming back and using my product? I think that you'll find that like ... I know Gustaf talked about this and I'm going to be extremely repetitive because this is one of the most important things that early founders make a mistake with. Yeah, so from there, I thought that I would try to also ... I thought that it will also be helpful for you to see, it kind of broken down in a formula. This is the formula and I like to think about things from first principles, right? What are the things that would cause you to grow? They're pretty simple, right? There's just like people that visit some page or the app, right, basic, you experience this everyday and then there's like people that sign up and then of those people, who did something ... How many of those people did something valuable like watching a video or making a recipe or taking a scooter, a scooter ride. Then, of the people, how many of those people end up coming back some period of time later and then how many people will like spread that product, like tell a friend, share it, maybe even have a sales team, who knows, right? These five things is like basically equal growth and these are your levers. The thing that I want to kind of point out and I want to stress is that, today, I'm going to cover the basics but I want to kind of impart on you that, the basics are really important. Even thousand person companies get the basics wrong. In fact, you'll find that if you don't get ... Even these thousand person companies like get it wrong, they tend to over-complicate the number of things that they should actually measure and track and it's really easy to make 25 metrics conceive of them, align them, divide them all by ... Give each team five and then really over-complicate something that is actually quite simple to do and when you over-complicate it, what happens is that companies, even large companies have immediate paralysis. Even mid-sized large companies have this like decision making problem. I used to have this funny phrase at Mixpanel where I used to say that all data is like contestable. You could like see this number go off into the right, and you go, "Well, I don't know, it was like sunny outside and like ..." and that's why people did it and people like have these arguments no matter what because causation is really hard to discover. What I find is that, just picking like three to five things, a north star and really simplifying it will do more good for you and I know because not only have I watched thousands of companies make this mistake at all sizes, I have made this mistake. I had a piece of paper that had like 25 sub-metrics and it was nearly impossible to actually keep track of it and people end up doing is in reality, what people care about is there are metrics that guide the team and there are metrics that help you monitor and assess if anything is going wrong and today, we're going to talk about things that guide the team, okay? Cool. Great. With this formula, you should be able to assess anything going wrong in the company, right? If you have a really confusing landing page, you should be able to figure that out. If your sign up process is too difficult, you should be able to figure that out. If you have a product that people don't really value, you will figure that out with just that formula. It's just that simple and you can get more complicated after that, right? You could say, okay, landing page views are down all of a sudden, why? Is it ... Let me look at that segmented by country, maybe something happened, maybe I changed something in a different country. You should be able to assess any reason why you're not growing just starting out with that. That's why I say, starting simple will last a very long time. This is what it looks like, right? Linking those two ideas together, you have this funnel of people, going from visits to eventually retaining and spreading the product and like on each one of these things, you're trying to basically solve one of these three problems. Is it easy to understand? Is it easy to get started? Are people coming back because they find it valuable, and then it loops and the funny thing about this is, even if you're like Airbnb and you have thousands and thousands of people, this process of optimizing each one of these steps is never ending, no matter what scale you're at, you'll always be changing something and you probably have competition so you have no choice. Awesome, and then once you are able to measure those five things and assess what's wrong, you have your levers and you should be able to take action and fix any of those things, put a team on it, maybe you're fixing it. It is really that simple. Awesome. We're going to start with the first thing which is, is my product easy to understand? In order to illustrate this idea, I decided that it would be somewhat sympathetic because I could show you like, really awesome landing pages of like great companies or I could just show you like what it's like to start out. I also wanted to mention that having a really bad landing page, there's this really awesome quote that PG used to have in our batch that I thought I would impart in you, which is that like, people basically just have their mouse hovering on the back button and they're just waiting to say, "I don't know what this is, it's confusing," back. That's immediately how most companies are just losing most of their uses from moving on. Just remember that your enemy is the back button. This is what Mixpanel's first front page look like in 2009. That's not what our logo looks like anymore. This is a design by me, original design by me and in all of its glory, if like actually go review the words, there's like tons of grammatical mistakes because I was horrible at writing. For some reason in my mind, I thought, "I know what the tagline for the company should be. Metrics that will make you drool." I guess, I think the reason why was because I thought these graphs were like so pretty looking compared to what was out there, that I thought like that would be the value proposition of a company. Of course, this is like totally wrong, like you'll all laugh. I wish I could have like made this landing page, presented it to you and then you'd all laugh at me and I'm like, "Okay, I should change that thing." I didn't have that. I had zero users at the time and yeah, I tried to target small startup big company. I didn't know who I was targeting, application developer, I'm like trying to target everybody. I think this is ... Every mistake that one could make on a landing page is like right here, right, like the sign up button. At least I kind of got that right. Thank you Adora for teaching me how to optimize buttons at slide. I made it like yellow but that's about maybe all that I got right, right? Just maybe roughly the sign up button. Then, after a lot of work, a lot of hard work, a lot of iteration, a lot of talking to customers, a lot of measurement, eventually a couple of years later, it kind of turned into this, right? I like, manage to get like the logo right. The sign up button is like ginormous and yellow. What we figured out though after a lot of time and a lot of energy was that we transformed from metrics that will make you drool to action speaks louder than page views and you'll notice it's like extremely prominent, the reason why was because the number one question that we would get as a company was, that's cool but like how are you different than Google Analytics, I don't get it, right? It turned out like this tagline for us at the front page, lasted like the next five years of the company and because it very much differentiated us as something where we would measure engagement, we did something different like Google was about page views, we were about like, what people did in your app and that resonated and then, from there, we tried to explain like what various ... We tried to urge people to go down the website and like look at different features and see if those things were valuable. We try to be benefits oriented in the copy, tried to show customers and case studies who maybe found it valuable. We probably made like four changes after this. I thought I'd present this one just because it was the one that maybe lasted, like the copy and the general essence of what we did and what we said we did, kind of stayed the same largely but lots of design changes. The real question is, great, so like how do I measure ... How did you measure whether people really understood ... Whether they understood what you were doing. I think there is like one tried and shared number. It's like very simple, it's just did people even bother to sign up? Bothering to sign up doesn't mean that they're going to use your product. That's a whole different matter. Bothering to sign up is like, "I don't know, I'm kind of interested, I mean, I'll kick the tires. This seems kind of interesting and cool to me." That'll tell you a lot about whether what you're doing is even something valuable for people. The second thing that you can try, these are just ideas, trying to find the ratio of people that are going from just like hitting your page, right and then doing any kind of thing after that. Instead of SAKA, instead of hitting the back button, right, like did they click on segmentation or funnels or attention in that case, did they bother to do that? I know there are a lot of B to B companies. Actually, one quick question, how many people are making like a B to B company? Yeah, great so it's about half of the room so I thought I'd give a B to B slant. One thing that I discover was that people ... A lot of your user will click on a pricing page link at the top. It's like second or third like most clicked on link if you're a B to B I think, generally speaking. I think ... This is totally contestable but I find that people clicking on your pricing page is like actually pretty decent indicator of like, "Okay, cool, you have something kind of valuable, okay, how much?" Even something like that I think can tell you a lot. Then, if you're a consumer, the benefit of being a consumer is that you generally have ... Usually will have like hundreds or thousands of users such that you could actually go do an AB test whereas in B to B it's a lot harder to achieve that. I really ... I think Gustaf is absolutely right that you should try to conduct as many experiments as you can and really AB test your copywriting and find out like what resonates with people. These are all things that you can do to basically figure out, is this easy? Cool. The second one is well, is it easy to get started with my product and the thing about this is like, a lot of people ... The thing about these first two things is that, if you don't make the first step easy, if it's easy to understand, a lot of people put so much energy, when we're trying to build a company, we try to think about like all the cool features and things that we could make that would be really valuable for customers. The problem with that is like if you don't even get step one right, you don't even have a chance at any of the subsequent steps. When we think about like, is it easy to get started with my product ... Forgot about password is kind of a useless feature. If you have like a very small percentage of people that are basically not even bothering to get started. In fact, I remember there is a YC company called Zenter which turned into like Google Slides. I remember Robbie Walker during one of our Tuesday dinner said, that they didn't even implement the save button for like their online PowerPoint because they knew that none of their users whatever ask, like, "Hey, can I save this," they just didn't even bother to do it because they wanted to see would anyone even use the PowerPoint presentation products. When they imparted that on me, I was like, "Wow, that's crazy," and for like, I don't know maybe like 18 months of Mixpanel, we didn't even have forgot about password because we were just so ruthlessly trying to prioritize like things that would really matter to customers. It turned out we didn't get many request anyway. This is really important and here's an example of easy to use, something you use every single day. That is literally one step. You just go to Google, you type it in and you immediately get to use the product. There's no sign up, there's no captchas, there's none of that, right? A more complex example is like Airbnb who might have like 10 steps or something like that. We worked with Airbnb for a while, helping them optimize initial user experience. Even though Airbnb probably has like 11 pages of ... You have to go to Airbnb and then you have to fill out this form and then you get taken to the results and then you click on those results and then you click on many different results because you're trying to price compare and you're going through this very complex check out process so there are actually many steps to booking something online. Airbnb had to kind of figure out like, how do we A, describe what we're doing, what we offer, what's useful for the world and B, how do we help people get started as quickly and as easy as possible. This is Airbnb.com right now and they have this very simple like, "Book unique homes and experiences this," you fill out this very basic things and you hit search and that's it. That's their getting started experience. It's not their entire flow but it's the first way to get started. They even try to like kind of impute something like what you might be able to find, something they manage to have on Airbnb. The thing is that, this seems simple, right? This seems like something that you could just like easily make, it doesn't take ... How hard is that, right? You can do that with Twitter bootstrap in no time. The problem, the thing that's really hard is like, figuring out that that's the right thing, that's really hard. That takes like ... they probably change that 50 times before they figured out, this is the right current optimal thing at this stage for the company. It takes a lot of work and you have to grind very hard to get to this point. The question is like, "Okay great, so how do you ... How would you measure something like that?" Well, I think the really simple answer is a funnel. How many people go from a landing page to signing up, to doing that valuable thing like watching a video, doing ... In Airbnb's case, it would have been just doing a search, just one search at least. Once you get past that, measuring your funnel for the entire initial user experience. It's like, where things were like, there's actually situations in games, games can have like 20 steps and I've seen hundred of gaming company's funnels and even though they might have like 20 or 30 steps, sometimes you can ... They actually convert very highly, like they might get like 80, 90% conversion rate through all those 20 steps. Just the first two are like actually the ones that have the greatest drop off. The last thing is just like actually speed. This is one that I think a lot of people don't actually do. It's just like how fast can someone just get started and that will tell you a lot about how complicated your experience is. Sometimes slow is okay, it depends on the business but speed is really important. Does it take them like five minutes to get something figured out or does it take a minute? In Mixpanel's case, we cared a lot about like ... We had a complicated flow, not only do you have to sign up but you had to be able to have the right code. It's just really complicated and that was the one downside, we weren't really sure, where people ... With Google Analytics, you just copy and paste the Javascript, with Mixpanel, you have to like actually go and track a line of code. We had this question in our mind, was that easy enough? Speed became really important to us. How fast can someone do that. I wanted to give you a couple tips and tricks. Things that I have seen that companies do that are generally like not great. Number one you're a startup. You don't have lots of fraud problems. You don't have ... You probably don't have a lot of people spamming your service. You probably don't have a lot of like fake actors in the system trying to sign up. If you don't have those problems, don't optimize for them until you do and I just want you to know that generally, email and text confirmations have an enormous drop off. I'm talking like, you get ... Of the 100% of the people that come like only 30% will basically go on and click on those email confirmations or 40%. You spend a lot of time trying to optimize that and then to even make matters even worst, like you have to hope to God that isn't go to spam. It's this really horrendous experience. Just be really careful, you'll lose a lot of users that way. It's really important to iterate on your initial user experience. That will be a never ending process. You will do this forever. Even as ... Especially as the product changes and it becomes really critical to like have someone on it so at least one responsible person just thinking about that all the time. Then, the last thing is, not every company, not every product is capable of doing this but to the extent that you can just let users into the product, it's always better to do that. In Mixpanel's case, it wasn't possible because like we needed to get your data so you have to sign up but we experimented with like what if someone just copied and pasted the Javascript but they didn't sign up and we just collected their data and then after that, we like asked them to sign up? We did experiments like that so, to the extent that you can achieve this, that's awesome and good examples of this are actually like Airbnb and Google where you just try it but there are always exceptions to these rules. For example, like Pinterest makes you sign up. They didn't used to but they do now and there are good reasons for that. Then, the last one, definitely not the least, are people coming back? To me this is probably one of the metrics that is greatly ignored by most startups and it's often the reason why I've seen companies even with millions of users actually die. I've seen probably 15 different companies that grew virally and then just died. The company died. I want to demonstrate this in a graph. I called this the shark fin effect. The reason why I call it the shark fin effect is because, well, one was because Mixpanel ... In Mixpanel, the default line color for your first data point is blue and it always ... To me it always look like a fin in the water and it looked a lot like this and what happens is you're slaving away and you find a way to go viral, right? Maybe you're going super viral on Facebook or Instagram or something and you're like, "Wow, I have struck gold. I am acquiring tons of users, my life is awesome, I'm going to be a billionaire, it's great," right? The problem with this is, is that if it happens too early but you didn't quite think about retention it becomes really problematic and this hurts companies. Basically, what happens is the app goes viral and then eventually, the rate at which you're losing users becomes high enough that you can't acquire the new ones. It turns out that like, even though we like to try to be extremely rational people, it's really hard because like when the app is going viral, you're just like, "Oh my God, I need to like ... Just take advantage of this moment." "What if I don't get it and what if my competitor finds it so I'm just going to optimize, optimize, optimize." It turns out that it ends up not being very valuable because if you're just losing all the users, it's really, really hard to reactivate. If you talk to people that have ever done like a reactivation campaign, reactivation campaigns generally don't go very well. You usually just like lose those users until they decide that maybe they want to give the product a second chance. It's not like, your response is like ... When this happens, a lot of people's response is like, "Oh we lost the users because our app wasn't valuable." "Let's just email them again and see if they'll come back." That's seriously not a great idea. It can be really ... it's really important to really think through retention and you really don't want to be like this guy. The shark fin effect is really, really bad. In fact LivingSocial is a great example. A lot of people don't know that like LivingSocial is basically this app that was on Facebook that had nothing to do with like the Groupon, deal cutting thing and they went super viral on ... I can't remember exactly what the product was. LivingSocial literally had to pivot the entire company to basically competing with Groupon because the app wasn't very retentive in those days. There's some pretty basic ways to measure that people are coming back. The most obvious one that I hope most people know is that ... It's just new users so making sure that you're tracking whether a brand new user, someone who just signed up, what percentage of those users will come back like a week later or 30 days later. It's really important to track a longer period of time because a week later is just not harsh enough metric and it's important to see how fast you'll end up losing users because you'll have to figure out a way to find the new ones and that becomes really, really hard at some point if the number of users you have becomes ... Starts to become quite high. The second one is just daily active users. There might be people that disagree with this. I don't think there are too many but I now think that monthly active users has become like the new BS metric, that's like very similar to like, just like number of registered users, like number of registered user is like a very silly metric. I remember seven or eight years ago, I think like ... I want to say was like Linkedin came out and said, "We have 200 million registered users." I was like, who cares how many people are using LinkedIn and I'm now starting to think that like MAU is becoming quite close to that now. Daily active users is really hard to maintain. I mean, how many products do you use every single day. If you just look at your phone and you go through all of the possible apps, how many of those do you actually use every single day. I think it's a really harsh metric and we actually found at Mixpanel ... So even if you're a B to B company and you might think, "Well, I don't know, I'm selling the businesses so like, can I really maintain like daily active users, especially on Saturday and Sunday," and we actually found that there's like a really, really strong correlation with daily use and rate of churn, rate of people like no longer paying for the product. It didn't take very many users and it was like one point, four daily actives would like reduce churn to like a dollar churn to like to sub 10%, it's incredible. I really stress measuring DAU and then the last thing is, a B to B slant again is something that I think a lot of B to B companies make an early mistake with. It's actually not just a B to B company. It's actually companies that do subscription so I don't know if there are companies that would ever think that they may offer a subscription of any sort, like if you're a meditation app or something like that. It's just dollar ... It's monthly dollar churn so if you're charging your customers monthly, many businesses charge annually but usually most startups start with monthly because annual commitments sometimes can be hard for early adopters. It's just measuring revenue churn. I want to give you a site story, if you have revenue churn and your revenue churn is say 7% per month, right, you might think, "Awesome, I'm keeping 93% of these dollars every single month. This is awesome, I'm doing really, really, really well." The problem is, it's a 7% monthly churn and if you extrapolate that out over 12 months, you're actually losing 58% of your revenue in a year. It's really harsh. A lot of people like ... For some reason it's like, this thing is like not mathematically intuitive to most people. I don't know why. It wasn't even mathematically intuitive to me early on and if you think about it, like you're spending all of this energy with marketing and improving the landing page and improving the getting starting flow. You're doing all of that and if you don't have retention, if you're losing that 7% every single month, you have to somehow make up for that 58% that you lost going into the new year. If you start thinking about it like, "Wow, we're making a million dollars and we're losing $580,000 and we're starting the year like that," you not only have to do that and grow on top of that. Once your base of revenue gets really high, it becomes really hard to keep up and you kind of have this like really long arc of like a shark fin effect basically. It's really important to look at your monthly churn and this will be the single cause of like debt but the problem, it's like the worst kind of debt. It's like debt that occurs. It's really like ... It's really flat growth but it occurs like eventually in like four or five years, if you're able to acquire users. I mentioned this, not only because ... I talked to some founders about this but this is also something that like really hurt us in the early stages of Mixpanel and we really have to figure out a way to fix this and there's all kinds of ways you can use product, you can use pricing. There's all kinds of ways to experiment with this but this thing probably gave me the greatest number of nightmares and lack of sleep I think ever. I really want to stress that on all B to B companies here. Really, think through this number and think about like, almost graph like, "Okay, what would our rate of growth be if churn stayed constant?" It's really important. Cool. I wanted to go over two things. One, I recognized that there are a lot of people here that have zero users so they're like, what the hell do I measure? What if I have less than 50 users? Well, the truth is I couldn't really think of a better idea than just basically, you just have to talk to your users. I thought I'd give you a mini story about, at least at Mixpanel what we did. When we had like no users and I have that page, that just said, metrics you'll drool over, one trick that I use is that I had like maybe 10 or 11 or 12 customers and honestly, I just put all of them on IM, like on G Chat or something. I don't know what the kids use these days maybe like WhatsApp. I honestly just like would just badger them on like WhatsApp or G Chat or whatever and then just ask them. I remember one time we were trying to redesign our funnel UI and we used to have a vertical funnel but we had a competitor that had a horizontal funnel and it seem like that was a better idea. It seem like that was more intuitive for customers. I didn't really know how to like get the data because like how was I really going to find out whether they liked one of those two UIs. It was just me and a co-founder so how the heck were we going to like do an AB test on that feature? There was no realistic way to do that. All I did was I just made like a really crappy version of the UI that I thought would be better, the horizontal version of a funnel. It was kind of like gray scale. I made like a really beautiful colorful version of like the vertical one which was kind of our control. Then, I just asked like 11 people which one is better? Then I want to impress this thing because Gustaf mentioned that it's the delta, it's the difference that matters in AB test, not that this is like a true perfect AB test but like, I asked 11 people on IM, got feedback in a day and 10 out of the 11 said, horizontal funnels and then we just made it and we built it and we never looked back and it was totally the right decision in retrospect. I don't think there's anyway that you can kind of skirt by, talking to customers but here's the benefit of talking to customer, even though it feel maybe as more tedious than looking at a graph. You're going to get way more information and depth from talking to customers than you ever will looking at a data point in a graph. No matter how much you slice and dice and segment the data, no matter how mathematical you are about it, you will never get as much information and in building your own intuition than just talking to customers and it's really critical that you do it. The one thing that I wish I had done back then was I wish I just wrote it down. I wish I wrote down tons of the feedback because I think it would have helped ... My co-founder would have helped employees in the company down the road, really go on the journey. Make them feel like they went on the journey with me rather than having a role, kind of feeling that they were so split up. Then the last thing that I thought I would bring up is to impart what I kind of said at the beginning which is that, I think one of the biggest mistakes that people make with analytics is being really overcomplicated, thinking that they need to be supersophisticated, thinking that they need to track all kinds of crazy cohorts and they need to have like a dashboard with like 30 pains on it that all like loads, we have like mission control. I don't think that is necessary. I had that, it didn't work, it was really hard to run the company that way. What also happens is that when you build your team, it will be really ... It will be confusing for your team, of those 30 pains like which one do I need to care about? I think that we need things that ... Humans just need to simplify things so I'd pick one north star metric. The north star metric I would choose in this case is like, what is a number that you're willing to bet the company on, right? If that number goes south, you deserve to die. If that number goes up, you will like ... You will have like made a huge dent in the universe. What is that metric? I'm not saying, you need to choose that metric forever but choose it for six months. Commit some time to it and if you find out that it's actually like the wrong metric which it will be probably the first time you chooses, usually your metrics are wrong. Then, you can change it but commit again for another six months but choose that one number and then like if what you have to do at the beginning is basically just like print it out and like put it all around the office, do it, because people will start to be maniacally focused on it, they'll show up to meetings and go, "Yeah, but that number is like down, what are we going to do?" Where you can get really complex is discovering why that number is down or why it's going up. That's when you can get really sophisticated. That's when you can slice and dice it and figure out what the retention is and things like and measure it over a funnel but keep things really simple. Then, the second thing is like, don't boil the ocean. Pick three or five other things, I think less is more and just stay there, telling even large companies ... Most large companies don't do this and it usually is very discombobulating for the work force. It's totally fine to have numbers that you wish to monitor but don't focus on them. Awesome, I think that's it and then, for the people that were like slightly bored, because I went over the basics, I thought I would help you out by just kind of having some advanced topics for later that you can read about. Really these are things that like I found really useful. My favorite one is this one, the Next Feature Fallacy by Andrew Chen. It's just this idea that we always think that like, the next thing that we make will be the thing that will change the trajectory of the company, i.e. Like the next feature is right around the corner and we're going to be huge and I would really recommend that you read that article. It's really awesome. One of my favorite one so for advanced stuff, these are just people worth following, reading about and can give you more ideas than what I've given you today. Awesome. Thank you. Jeff, Q and A. Okay. Cool. All right.
Geoff Ralston:
You want me to pick? Do you want me to pick?
Suhail Doshi:
Yeah, I want to be fair. I don't be like, if like every speaker chose someone in the front.
Female:
What would you suggest the conversion, some people conversion or trying conversion from visitors to register, considers registering versus subscribers, obtaining subscribers.
Suhail Doshi:
Yeah, the problem with that is that, it's so ... Yeah, yes, I think someone asks, what are like good benchmarks for ... Given those five steps, visiting, signing up, using the product, what would be good conversion rates for anyone of those steps. The chart part with benchmarks like that, is that it's so dependent on the business in B to B, like going from like a visit to a sign up, I think like a four to 5% conversion rate is pretty good. I think I like swap notes with Stewart at Slack because we kind of had similar landing pages. I think I ask them, what was your conversion rate at Slack and I think we both ended up being somewhere around like four to 5% from visit to sign up. That's said, like, I don't know what that would be for something like Airbnb where it's like getting to a landing page and then searching for your first place that you'd want to book. I think it's really dependent. I know that Mixpanel publish on its blog, publishes lots of benchmarks. I know the marketing team there that does that and they like publish all kinds of benchmarks for like, what should your retention be if you're a gaming company, if you're an E-commerce company, if you're a B to B company, things like that, social company, a video company, things like that. I think that you could probably go there and find various benchmarks reports but it's really dependent on the business, yeah.
Male:
When you go into the sign up process, how do you see numbers vary, if you're like created an account name, and password, doing all that kind of sign up versus a single like log in with Google or LinkedIn, yeah, how does that transfer, do more people sign up if they have that like single click option, do you see more people drop off because they have to use one of those?
Suhail Doshi:
I think the question is what converts better like those social buttons where you can just like quickly authenticate like Facebook Connect versus just like standard sign up, is that right? Yeah, all of these questions, they require like a lot of specificity because it's actually like really unclear. In that case, I actually don't know. I truly don't know and so I think the way to answer this question is to find friends in tangential industries that like don't compete with you but basically do that and then just be like, "Hey, what is your conversion rate? Have you tried something?" I don't actually think a lot of those numbers are secrets in reality. I mean, you can't go to your competitor and ask that but most of these things are not like super duper secret type stuff and like they could either give you the information and save you the six months or they'll just like say ... Or you try it out but it's sort of strange to keep it a secret so I'd rather give you ways that you can discover that. I would just ask friends that have a start up or even if they work at a big company, sometimes they're willing to share it. Yup. Cool.
Male:
You talked about making it easy for user, start using your product and then there's a slide there [inaudible] . How do you balance simplicity with informing your visitor, like what it is you do, why they should use your product? You don't have that brand recognition already where the visitor already had-
Suhail Doshi:
Yeah, right.
Male:
How do you balance those two?
Suhail Doshi:
Right, yeah. The question is, Airbnb's landing page was super simple and how do you balance like, I mean what if a lot of people just know about Airbnb and they just know what to use Airbnb for and then that's why they can keep it kind of simple versus like having something more complex. A good example of this is like Craigslist, it's crazy. If you go to their front page of Craigslist, it's like absolutely nuts. Even Amazon, if you go to Amazon.com, right? Even though Amazon has like huge brand recognition, you go to Amazon.com. It's like, "Whoa," there's like million things going on at once, right? How have these companies found the balance, how have they found how to deal with this complexity and the truth is, is that, I find that it's important to have like a guiding principle. What is it that you're trying to actually optimize for, right? In the case of YouTube, let's take YouTube as a good example, right? YouTube could just like be Google, and just like, what do you look, what do you want to watch? They could do that but they don't. I think the question really is, is like what is the team at YouTube trying to optimize for? Are they just trying to get you to like watch a video? Is Airbnb just trying to get you to search for any kind of hotel, or not hotel but Airbnb? I think like the team has to develop first a hypothesis of like what matter ... What do we think matters and then from there, it's important to experiment and try to figure that out. It turns out like it's not too ... Again, it's about the delta of like what people will actually gravitate more towards. If you make a really simple experience but like people are like really confused, because they don't even bother to do it or you ask them, then you kind of have like new information and I think the key is to find the minimum number of things you need to add to get someone started. I don't think there's anything ... I'm sure Amazon has tested that front page, like crazy and they found that it's like quite optimal. Yeah.
Male:
In your personal experience, do you think that short term videos that's more than two minutes [inaudible] .
Suhail Doshi:
They can. It depends on the product. I'll give you an example in Mixpanel's case. Sorry. Yeah. The question was are video ... Are one to two minute video tutorials effective, fair? Yeah, so I think that it's complex. We did a video tutorial video in our flow, for example and like, I would say that, a lot of people didn't watch it, truthfully and this is what I mean by experimentation. A lot of people didn't watch that video tutorial. I mean, I actually I think eliminated it in the first three steps but what we did find was this interesting thing that occurred, as a result of looking and measuring things which was that, we have a feature in Mixpanel where you can like see every action that a user takes. I would literally just like go to every user and then just like watch like a creepy person just like sort of watching like everything that like someone would do in the product. Eventually when you do that like ... When you do hundreds of those and you just watched hundreds of those, you start to develop some sense of like, okay, what you developed is an idea, a hypothesis and all of these things require art and science and I think the companies that take it too far, they like take it too far to science, end up getting it wrong, they end up optimizing for like weird things. You can always make a button more yellow and bigger. That doesn't make users happy eventually. Anyway, to answer your question, what I found was that lots of people were just clicking randomly on like all the side menu options and my hypothesis was, well, they hadn't integrated Mixpanel yet so there's nothing to look at so what the heck were you doing just like clicking on like funnels and retention, why would you do that? My hypothesis was, they were just curious like what the product did, what is it that they did and so we did this AB test, where we like had a control with like nothing, just the regular clicking around, seeing like some image and then we had another one that had like videos. Then we tried to see what cause people to like do their integration, their first integration, what was the conversion rate and turned out like the video, like super won. Actually we did even a more simple test than that. I think we just emailed people a video and said, "Watch this video, super basic," I didn't even bother to implement it on the website and I had an amazing conversion rate. Then, we decided to add videos all ... To all of those tabs where they wouldn't have been anything anyway. That's why I want to say like, I want to stress this, it really depends and what matters is that you run the experiment to develop a hypothesis, i.e. The art not the science and then measure it, which is the science to find out whether the thing that you thought of was in fact right. Yeah.
Geoff Ralston:
All the way to the back.
Male:
I have a question for you, it's the churn, the revenue churn. How does that relate to [inaudible]
Suhail Doshi:
Fifty eight percent loss in revenue, over a year. Yup.
Male:
You confirmed that we look at ... Where we look at retention [inaudible] .
Suhail Doshi:
Yeah, so there's slightly two different things. One is like user retention and then one is like dollar churn like. In the case of dollar churn, the reason why I bring up dollar churn is that, even ... Like you could totally have a company where nobody uses the product and the company continues to keep paying for it, right? What I find that like most companies are like pretty incentivized to cut their spending and employees are incentivized because they lose it in their budget. With respect to dollar churn, you could have lot ... In a B to B company for example, you can have even one user using the product but the company could continue to spend money on it and like, you might see like a 40% retention rate. It might be okay, as long as the company is continuing to spend but losing 40% of your dollars in a year is really hard and not good, retention. Does that make sense? Those things are not exactly linked. They're not directly linked. Yeah. Yeah, to give you kind of a benchmark like a chat app that's like often used a lot will have like 60, 70% retention. Those are some of the most retentive things. Something like Slack would probably have like 80% retention. It's kind of like unbelievable. Whereas like our product, like Mixpanel, when we first started out like the first year, I think we had like 30, 40% retention and that's not that great, these are our first few years. Yeah.
Male:
Are these whole metrics, I think you have all four, is that hardware startups, is the same like you build that we need to go for it?
Suhail Doshi:
Yeah, the question is, are these metrics, can they be used for hardware startup. Yeah, I don't see any reason why. We worked with Jawbone for a long time and we work with Fitbit. Most hardware ... Does your hardware company have a software component to it?
Male:
Actually, we do subscription model like [inaudible] .
Suhail Doshi:
Okay, so it's a hardware company with subscription model. Yeah, absolutely. It maybe more challenging as a hardware company to track but it's like, sometimes but it's totally doable. Most hardware is like connected to the internet somehow so I don't see any reason why, like a lot of the scooter companies for example, they're like all connected to hardware and they have a software component. There's really no difference. Yeah.
Male:
[inaudible] .
Suhail Doshi:
Yeah, so the question is, the first page was trying to target like various kinds of sizes of companies and then like the second landing of Mixpanel like in 2012 basically didn't kind of dispose of that and had something very different, it was marketing features and stuff and how do you figure out like what to do basically from that point to the next point. Yeah, so I think the basic gist is that ... So you can clearly see like I had no idea who our customers were and then after like a few years, you talk to users, you ask them like what size of company they are, if they sign up, if you want to, we didn't do that but we just spent a lot of time ... It was so obvious that like everyone that was signing up for Mixpanel at that time was a startup. In our case, it would be, we call that a small business.
Male:
[inaudible]
Suhail Doshi:
I think it took like maybe no longer than six months to figure out that we were not going to sign up a large company. Yeah. We just didn't have the product, it just wasn't there, we hadn't worked on it long enough so the only people that were willing to basically try were companies that could take the risk to try it and most of these companies didn't have anything anyway so it was even easier. It's easier to target companies that had nothing and had ... Were willing to bear risk and then most of the time, I would always employ this trick where I would just ... I would charge the money for it and then if they said no, I would just give it to them for free because the feedback was more important. I always wanted to see if they would pay for it and so the only people that are willing to do that are also small businesses, it turns out. For us it was just like ... It kind of hit us, it slapped us in the face, it wasn't like this hard thing to figure out. Yeah.
Male:
[inaudible] .
Suhail Doshi:
Yeah, so to be clear, we weren't necessarily building things to bring new users on. Sorry, the question is you guys built, forgot about ... You decided to not build forgot about password, how do you know when to build quality of life features versus features that just acquire new users? Yeah, so I think that we didn't ... It wasn't so much that only built features to acquire new users. We built features to acquire new users and to keep existing users. We were very focused on like parts of the product that would actually make a difference, right, that would be clearly differentiating compared to all of the other possible options out there. One of the things that's like not different is whether you have forgot about password, right? That was like in the bucket things that like maybe don't matter. I think the question is really like, when does it become a problem enough for you that it's worth building, right? If you're finding that like it turns out that the number of request that you're getting for forgot about password is like increasing and then manually doing this process, it's like annoying. Then, you should just build that feature because you know that if you don't build a feature, they can't log in then you can't retain them and then you lose them. It becomes this flywheel effect where like the only way to retain them is to do that. In fact, even though Mixpanel didn't build forgot about password, for like maybe 12 to 18 months, we went eventually ... We found that people would try like multiple times, when I was looking at that stream of every action that a user would take, I found that like people were like going to the log in screen like five times. I was just so confused. I was like, "Why are they going to the log in screen five times?" I was like, "I think it's because they can't log in." Some people don't even hit forgot about password, they just like try, try, try. I'm sure you've all done this like with a bank or something, you're like, "I'll just do it tomorrow," you don't even bother. It turns out that even that happens and so we went so far, even though we didn't do that, I like to think that like we really made up for this later on. We went to ... So far, we found this clever trick that Facebook use which was that, when you failed to log in to Mixpanel like three times, on your third time, we just send you an email that says, "We just sent you an email. Just go to your email and click the button and you'll log in right now. Don't worry about forgetting it, don't worry about changing your password, just log in." It turned out that like that had this great retentive effect because we were really focused on retention at that time. Yeah.
Geoff Ralston:
Okay, a couple more questions, over here.
Male:
[inaudible]
Suhail Doshi:
Yeah. How do you gauge product market fit? How do you measure product market fit? I think there's a lot of people that probably have already ... Will talk about or have already talked about like people ... And I think they're all right so I'll give you a quantitative slant versus like a qualitative slant. Some qualitative slants were things like when people are willing to spread you to their friends, that kind of thing. The quantitative version of this is really just measuring your retention, like how many people ... It's really measuring like DAU, it's really measuring the percentage of people that come back a week later, 30 days later. It's a number of people that like become almost they need your product or they feel like life will be a lot worse. I think that you can totally measure that by how they use it and so I think the quantitative measurement is basically finding like some benchmark, some cents for your industry, maybe that's like 30% and just remember like that benchmark is probably like average and your desires is to basically be above average. If you can like significantly beat that benchmark, then I think that you've like really ... You've done better than probably product market fit. I think that's one thing. I think another thing is like, you can try to measure like overall like, repetitive use, like frequent use in a day. You can make pretty harsh metrics. You can be like, for example with DAU, you could say, "I don't want it to just be daily active user like they did one thing. I want it to be ... That user has to have watched five videos and then and only then do they count as a daily active user," so you can be particularly harsh about that metric, right? Someone coming and like ... That's why I say MAU is kind of a BS metric now because you come and watch one video and then never come back again and you like count and that's not like a very healthy business so I think that find ways to be harsh about your north star and I think that you'll find that you reach product market fit, yeah. Yeah, I think with Mixpanel it was like, 30, 40% retention, it was like really horrendous and we totally didn't have product market fit yet. Yeah.
Geoff Ralston:
Okay. Last question.
Male:
What is one piece of conventional startup wisdom that you think is usually wrong?
Suhail Doshi:
It's like a Peter Thiel question, what's one piece of conventional wisdom that you think is generally wrong?
Male:
About startups.
Suhail Doshi:
About startups in general.
Geoff Ralston:
It's a hard last question.
Suhail Doshi:
It's a hard last question. It's like everything else is fine. I think ... I don't know, I need to think about it but I think that one thing that has like become pretty clear to me is that I think after having done Mixpanel for about a decade, I kind of figured out that ... I tried to impart it on you in the talk today, which is just that, it's much better to just like ... I think that we tend to complicate things as humans and that comes through in like everything. For example like we tend to make a version one of a product and then we get to version two and we're like, "Okay, we're done." We're like, we did it, we solve the problem and then often a found were like, kind of get distracted and then go and like work on the next product, the next thing and they haven't yet figured out like their retention is probably not great or they're operating at negative margins or like all these things. That same thing even happens with numbers like we pick our three metrics and then we continue to get more complex as a business and then we add the next 20 things and we do all these things to like ... In hopes of like, feeling like we control the situation better and I just think that those are all distractions and mistakes and I think it's really hard for the average founder to actually just be focused on something boring. We tend to have like 80 or something and like we just decide to get very easily distracted and I think it's hard to do something ... Do the same thing for four or five years, because we don't like feel challenged anymore and I think that's a huge mistake that founders often make and I don't know that anyone say, "Oh that's like totally wrong Suhail." For whatever reason like every founder repeats it. There must be something, I don't know ... There must be some kind of wisdom that we all think this is the right thing and I made those mistakes. I made new products that I shouldn't have. I've measured too many things, over-optimized things that didn't need to be optimized so yeah, that's it. Cool. Thanks.
Geoff Ralston:
Thank you Suhail.
用户评论