Geoff Ralston:
I would like to introduce Jared Frieda, my partner, and his esteemed panel, who he will introduce, to talk about technology. Thank you.
Jared Friedman:
Thank you, Jeff. Okay well, I am super lucky to have a very esteemed group of guests with me here today. Everyone on this panel is a technical founder of a really successful company. Everyone here has built a really cool company and a really amazing technology organization. We've changed the name of the event for today. It was originally called "CTO Advice" and, on the advice of Lillian, thank you so much, we have changed it to "Technical Founder Advice", which I think is a better description. And I would also like to thank everyone on the startup school form who wrote in with questions for the panel. We posted a few weeks ago and solicited questions, we got over 150 responses so, thank you everyone who wrote in with those questions. We're gonna do our best to cover as many of them as we can and then, at the end, we'll open it up to the in-person audience for some questions as well. Okay so, let's get started. Could we start by having everyone introduce themselves and tell us about your company and about your technology like what's your technology stack, what's some of the interesting technology that you've built, how's your technology organization look like today. Ralph you want to start.
Ralph Gootee:
Hello everyone I'm Ralph Gootee. I'm CTO and co-founder of PlainGrid. PlainGrid we're 350 people based out of Mission, San Francisco. We write beautiful, easy to use software for the 17 trillion dollar construction industry. So what that looks like to you, the analogy off the top of the news is we're like GitHub for construction. Construction has blueprints, blueprints change rapidly, version control is extremely important. If you have changes that means there're issues that are happening, issues need to be tracked and then we build collaboration tools on top of that too. As well, as a lot of other tools for the construction industry that go into some deep jargon. Our stack, based on AWS we used to be based on a variety of other things, we had to move everything to AWS overtime. On our back end, mostly Python on the backend and then we've got some go in other things. And one of our challenges is that we actually write native for every platform so, we've got IOS, Swift, Subjectivcy, Android, All Javlin, Kotlin, Web React and then Windows. We have a full windows app as well which is .net.
Jared Friedman:
Cool.
Calvin...h-Owen:
Hi everyone, I'm Calvin French-Owen and I'm CTO and co-founder of Segment. Segment is a single API to collect, organize and adapt all of your customer data into hundreds of down streaming tools you might be using. Whether that's a analytics tool like Google Analytics or Mixpanel, maybe a customer success tool like Gainsight, maybe an e-mail tool like Customer.io. Segment helps you collect data from where it lives and get it where it needs to be. In terms of the company overall we're a little over 300 people right now. We have our headquarters in San Francisco but, we also have offices in Vancouver, New York and Dublin. In the engineering product and design team which is kind of how we build product here at Segment, is a little over 80 or 90 right now. In terms of our technology stack we're also built entirely on top of AWS. In terms of how we run our infrastructure we run on top of ECS, Amazon's Container Service and pretty much all of our different services are containerized. Today we're running about 300 different microservices which are kind of piping together very escophic topics and reading and writing, transforming this data, getting it where it needs to go. And our back end is primarily written in GO.
Diana Hu:
Good morning everyone, so my name is Diana Hu. I was the founder and CTO for Escher Reality and we're building the back end for augmented reality. And I say I was because my company just got acquired by Niantic, the makers of Pokemon Go. So, what we were building at Escher and now, we actually continue doing it in Niantic, is building this backend technology for augmented reality to enable developers to build AR experiences as easy as possible. So we handle all the complexity with the computer vision algorithms, with all the interaction with the backend, all the hard parts of getting things to render so that developers could build easy AR apps in minutes, so that's what we do. We provide a lot of advanced AR features like multi-player, persistence and cross platform so that it is easy for developers to just develop let's say unity and it just works. In terms of technology stack it has been a bit of adventure, back at Escher we had a service that we hosted in AWS but, that didn't matter too much because we had everything in Docker containers. Now, Niantic is heavy user of Google Clouds so move everything through Google Cloud. But, we have a lot of the bulk of the code for us native means more C++ literally, because it's a lot more efficient for us to write the code and all of the algorithms once and then cross compile it across all the architectures for Android or IOS and event he back end. So that we run some of the [inaudible] we can prototype it in the phone and then we can easily move them into the server because our server is actually also written in C++.
Lilian Chou:
Hi everyone, I'm Lilian I am COO not CTO of Second Measure. Second Measure we're about 50 people we got started back in 2015 and we're based in San Mateo. What we do is we analyze credit card data. So, basically we take billions of credit card transactions and try to build a daily view into public and private company performance. We analyze these billions of transactions automatically everyday and Justin will clean them, enrich, normalize them and then service that in a front end application for our clients which include VC firms, hedge funds and big brands like Blue Apron or Spotify to check trends, any sort of competitive intelligence or customer intelligence. So we actually don't have a CTO that's a fun fact about us. That's why I'm here on the panel today. Myself and my co-founder actually are both technical's that's been pretty fortunate. We don't have to deal with as many of the challenges around starting with non-technical founders. I mentioned our team is 50 people, primarily technical so, are technical organization is about 30 people and that's actually split evenly between data scientists and engineers. And that's probably something that makes us unique is that, our core product is actually the data itself. So our data scientist and engineers have to collaborate super closely on a day to day basis. And probably one of the thing that has technically been interesting is a lot of our users want to explore and dive deep into our data and building a front end that allows that flexibility of exploration while still creating a good user experience basically requires us to figure out how to rapidly query data and run really complex queries on our back end. You asked about technology stack, we're also primarily on AWS so for our pipeline we leverage services like Lambda and Spark. And then for our front end we are React based and leverage columnar data storage on the back end to service queries, so think Redshift.
Jared Friedman:
Awesome guys, that was super cool. It's so cool the diversity of different kinds of technology organizations and products that are here today. Okay so, most of the companies in start up school are really early. So I want to take everyone here back to their V1. The very first version before you had actually shipped anything. Tell us the story of how you built your V1, who built it, how did you build it, how long did it take you to build it and what things went wrong in the process.
Ralph Gootee:
Okay so I'll start. This is a great story. I've actually got my CEO and CO-founder over there watching me right now. So that's kind of-
Jared Friedman:
Say Hi Tracey.
Ralph Gootee:
Yeah, Hey Tracey, how's it going? So, that's where we started Tracey told me about her idea of putting blueprints on an iPad. This was in 2011, right at the launch of the iPad so it was very early for the technology. And I said "Yeah no problem. I can put blueprints on iPad it's like a PDF, that's no problem at all." So I was attempting to impress her really quickly by putting blueprints on an iPad, it turns out there was actually some hardcore challenges with putting these giant images on a really low computer availability on the iPad, there was no graphics chip at the time so basically these were 20,000 pixel images and they would just crash or overrun the video memory. And they were in PDF which is a kind of painful format to deal with so, the first proto type sucked it was really slow, and that actually told me that there's something real meaty here, there's something that's actually a challenge. And it took me about a month to write the second proto type based off some... My background actually I used to work at Pixar so I had some graphics experience there. Didn't use any proprietary stuff but, used some off the shelf computer graphics knowledge to write the first blueprint viewer that ever ran on mobile. So that was our first prototype back then and I think the best thing I remember back on it is the only thing we really focused on the first prototype was what technically impossible. There's no way you could load blueprints onto an iPad at the time and that was our prototype which, meant that all the data got there by side loading which meant there was no web interface and the little crappy web interface we had, had a delete button right next to a publish button that was not a pixel part and there was no confirmation on the delete button. So, we actually as a team we manage everything on the backend, we kind of like behind the curtains, loaded all the documents for our first 30 or 40 customers. But, all the saw was the prototype that was fast, and did what they thought it would. They never saw the man behind the curtain so to speak. So, that's a little story of our first prototype.
Jared Friedman:
Awesome.
Calvin...h-Owen:
Cool and I'll share a little bit of a story of our V1 and then maybe our V1 next... I think for those of you who have seen the startup school talk, my co-founder Peter talks about finding product market fit. He goes into the story in way more detail than I could today but, I can share a little bit of our journey. So, when we first started we were in the YC 2011 class, which sounds like a dinosaur now in terms of YC years. But, at the time we were actually building something completely different from Segment today. We were building this classroom lecture tool, which would help professors get feedback about what their students were thinking in the middle of class. We were students at the time, and we figured "Hey, this seems like a great idea. Professors will sometimes go on for five minutes, and they'll lose the entire class, the class gets confused, and you just waste everyone's time. Let's solve that problem." We built it out over the course of the summer and we deployed it back in Boston in the fall. And the whole thing just kind of crashed and burned like students would go to Facebook, YouTube, Wikipedia. In retrospect all the things you would assume college students would do if they have a laptop in front of them in a lecture but, we didn't quite see it that way. So, we started looking around for a new idea because we just raised a seed round. And we say "Okay what are the problems that we have with this college lecture tool." And one of them was that we couldn't answer questions about how people were using our tool. We couldn't tell you hoe college students at Harvard were using the tool differently than students at MIT or we couldn't tell you how anthropology classes use it differently than biology classes. And so we switched again to something which, also we don't do today but, effectively it was a competitor to Mixpanel or Amplitude. It was this tool that would allow you too cluster your users by different rules and break them up into segments, which is where the name came from, and from there you could then reach out to them or figure out what your most engaged users were doing. And so we built out that system for about 15 months. The whole time we we're doing Revs on the backend infrastructure, we were building it out and we were trying to get users and no one wanted our tool. And so we arrived 15 months later, we actually moved back to the bay area after living in Boston for a year and we go and talk to PG and we tell them about our whole saga that's been running for the past year and a half. And he listens to our story and he listens to everything that we've done and when we were finished, he just sort of pauses, I think it was maybe 400 ft out there walking around the round about, and he says "Wow so you just burned half a million dollars and you're still at square one." We were just like "Yes." He said "Well on the plus side you still have some runway left so try again, like launch something new." At the time we had this internal tool that we had been using as kind of a growth hack to get ourselves more users. And the whole idea was that you would use the same API to send us data as you would send to Mixpanel, Kissmetrics, Google Analytics, all these other tools. And people actually like that single API and they were contributing to this opensource library on GitHub and starting and using it and so my co-founder Ian said, "Hey why don't we turn this into a product. Why don't we launch this and see what happens." And my co-founder Peter said "Oh, that's the worst idea I've ever heard, it will ever work." He's CEO now but, he's seen the light. And so he said "okay, I've got to figure out a way that we can test this idea incredibly quickly." So we cleaned up the library and we launched it on Hackernews and to our surprise it actually got about a thousand stars that first day on GitHub. And so, our real V1 was that first week taking that library and basically just had a landing page up which said, "Hey if you're interested in a hosted version of this leave your e-mail." And about 500 people left their e-mail. And we built out that V1 in about a week, we're just everyone was in mad hackathon mode and building out the product which we launched a week later. And so, today I think there is actually no parts of that product which still live on today. But, it was enough to get us the seed of product market fit and then start getting us a user base.
Jared Friedman:
So one week is the short, one week, one week, everybody hear that one week. Okay.
Diana Hu:
That's kind of amazing one week. So our story is definitely more than one week. so me and my co-founder had been thinking of building different things and one of the technology trends that we really saw... He comes from a robotics background with a lot of experience with Slam. Slam is this algorithm called spontaneous localization of mapping, and it's one of the core algorithms that runs today in AR, which helps tell where your camera position is while you also build this map of the world. So what happened is there's this category of algorithms that have been used mostly for robotics now, AR had then kind of tried and many times but never really been picked up, there's AR with markers and that never really took off. And then we thought about why do we bring these algorithms and run them on the phones and we got to the point were we thought we could do it because the computation at that point with phones for example iPhone seven has the same computation actually as a MacBook Pro from 2013, if you think about that that's kind of amazing in terms of all the trends with [inaudible] and some of the one's with power efficiency... So we decided to go do it. At that time I was actually still working and leading a data science team back then, and I said "Okay I'll do that." Because I was thinking of doing some transition. I was working part-time with another engineer on this, one of the engineers that we kind of got excited to work with us. So, we struggled a lot to try to get a version working. The first one was very duct tape, and it wasn't very encouraging. It was only working five frames per second, which is terrible because if you want a good AR experience the whole thing is that you need it to render nicely, and be at least 30 frames per second. So that was very discouraging in a sense because it was all kind of put together fast. Then we started kind of really removing all the research code out and then it started working and seeing a lot of promise, and it was running then at least 30 frames per second. And note that when we were working on this it was at least about a year and a half before AR could have launched. And then we were excited about this so were both very more from the technology side, so we didn't know the product market fit, or where to go with this. So, we had to find a market. [inaudible] side of that is we went in and interviewed a bunch of people that we thought would use it and we found a really good fit with gaming. So we decided to focus on that and that was our YC application. And the thing that really took off and this is actually a good story, is the first day of YC, Apple announced ARkit which is basically what we had built and that was sort of a moment of truth for us, a sort of go flight or fight. So we decided why not let's just take on Apple. So we decided to just double down, and we had this idea that this wasn't just on the device, which we had all our demos and technology was just working on the phone, we didn't have anything on the backend yet so, backend YC we got it working with a backend and also with Android and basically accelerating the roadmap that we thought was bat a year, shrink it and got it done in three months. And that was the thing that once we put it out there and let people sign up if they were interested, a thousand developers signed up in about a week. So "Okay, I think we've found something." And then after that we got a little interest and among those Niantic was one, and the story is we got acquired later, that's our little summary.
Jared Friedman:
So how long was it from the time that you started working on like the original Slam algorithms until you had like a working product in users hands?
Diana Hu:
So we were not really working full time, it was probably about a yr of part-time. And part-time I wasn't really working so I went full-time and took another six months of me and another engineer to do it. And then during the summer, let's see we hired two more engineers and that's when we were able to really accelerate it and get it working well.
Lilian Chou:
Cool. So for us our V1 probably took assuming full-time, two or three months to build. And I actually would say that most of that time was just figuring out what our product was going to be. We knew that the problem we were working out was going to be really interesting, initially we were focused on investors basically trying to get them an information edge on sort of any investment decisions that their making whether that be hedge funds, focus on public markets or VC's focused on private markets. We ran through a few ideas, we were like "Hey do these investors want predictions, do they just want to know how this company, public companies quarterly sales are gonna hit?" And the short answer is yes but, no. That didn't really seem like a really sort of sustainable business model so then we shifted a little bit. It was like "Okay maybe these investors really, really want to go deep and they just want to cut data anyway, they want." So, we put something together really quickly, put that in front of some of our friends who are investors and found out very quickly through that somewhat user testing that, that was way too overwhelming. Our users wanted some guidance. So what we ultimately landed on is we needed to build our own self-service platform where we can apply our opinions around how our customers should be viewing this data and getting quick access to insights. So once we figured that part out, it was actually pretty quick. So my co-founder focused on building the analytics product piece so the front end application. And together we built the data pipeline and the transformations on the data. But, I think some important choices that we made early on, is maybe something that you guys are facing right now, is what technology do you want to use like what do you want to build this is? Ultimately we decided to build our first application in Groovy using the Grails framework which is very not shiny, not that many people are doing that but, that was actually the code that we had the most experience in. So my co-founder Mike had built large production scale systems servicing hundreds of thousands of concurrent users and pretty much within a week had built the bones of the application. And kind of like some of these other stories here like none of us are running our V1 anymore. So it's more important in the early days to be able to iterate quickly and usually that comes from using the technology that you know the best, building things in a modular way such that once you actually find that product market fit and know what your audience needs then you can focus on that area and actually select the technologies that are a best fit for that problem.
Jared Friedman:
Awesome. Okay so I'm gonna go next to the very top voted question from the startup school from, which is about the trade off between engineering best practices like good test coverage and security, and scalability, and redundancy versus writing code as quickly as possible and shipping something. So, can everyone talk about how you made that trade off for your V1 and then how it's evolved since then and sort of like the time table of its evolution as your company grew and those things became presumably more important. And just to mix it up let's trying flipping the order, we'll start with Lilian this time.
Lilian Chou:
Sure. So speed is paramount. Nobody's going to pay you for having excellent test coverage so, in the early days you definitely want sort of what you need, right? So access the risk to your business around sort of security and not having things working, and obviously you want whatever you build to run every day. You don't want to be fixing or it breaking all the time and not actually building. But, really it is a balance. I do believe that initially it is more important to have that speed of development and constantly testing and incorporating the learnings into your product than it is to have the most robust or most scalable product, right? You're trying to find something that people will pay you money for or solves a real problem and that takes time and a lot of iteration. That has definitely evolved since we got started. You know, once you find product market fit a lot of these problems change, your system gets more complex, your team grows, and you have a lot more people contributing code, a lot more ways for your system to break, a lot more users who are relying on your product. So for us, the way that manifested itself is we started writing uni-test to verify, each engineer would be verifying their code works. Then we introduced CI and CD to make sure that that code would work with the whole system. Our director of engineering who leads our engineer organization, started developing and formalizing our best practices around like code reviews, testing etc. And to finding those processes and then finally building some more controls directly into our system to make it harder to break things.
Jared Friedman:
And specifically what was the rough time line for those things? Like are we talking about two months after launch, two years after launch?
Lilian Chou:
So I'd say most of that stuff I was just talking about happened probably in the last year to year and a half.
Jared Friedman:
And that's how many years after launch?
Lilian Chou:
About a year and a half after launch. And the main reasons for that again were the growth, and those complexities of our systems and our technical staff tripled in the last yr so, just that along introduces a lot more need for process.
Diana Hu:
In our case it was aways about trying to get to MBP that was reasonably working well. And all that was definitely duct tape code, is nothing I'd be proud of. It's just... I put together and barely working as much as possible, so we were there even when we did the kind of private launch in a sense when we did the private data and got customers who signed up. And we had private data that we had some number of our game developers working with us and that was also the duct tape code. The only time we started actually having all the best practices was once we joined Niantic because now, we have to integrate into the games which Pokemon GO has hundreds of millions of users. So we were burying ourselves to get a lot of the... We pretty much rewrote the whole code base, none of what we had before is there it's all re-written to prepare for that expected number of users.
Calvin...h-Owen:
Cool. I think for us at Segment there are kind of three distinct phases of our development. I'd say the first one was when we were in full hackathon mode, right? Were definitely didn't have any test, no CI. I'd say there was an 80% chance that we were just going to throw it out two weeks later because it wasn't going to stick and that we were going to move on to the next thing. I think because we had been burned so much by building out all this infrastructure and investing in a lot of these ways to provision your infrastructure, and write test and spin up different new services, over the past year and a half, it really had gotten us nowhere. We had no users. It just created this like pretty aligning homing instinct for us where we just said "No matter what the only thing that matters is getting users at this point in the game." So, that's where we started and I think that lasted us probably for about the first nine or 10 months where we were just focused on rapid iteration, getting more users, making sure that we didn't run out of money and then whole thing wouldn't of even mattered. I think from there the first phase that then shifted is when we brought on our first engineer TJ, rad for those of you who know or are involved with the note community, TJ is one of the best engineers I think I've ever worked with, I think 90% of note jobs run on some part of his code. He basically came into Segment and he looked around at well, this mess that we created, the product that we created and he said "Well, there's no way for me to work in this. I have a horrible time onboarding."I think it took him a week to get his entire laptop set up. And so we kind of moved from this point where the entire development team shares the same tube of toothpaste to now, starting to have more and more engineers of [inaudible] for the project. And when that happened it really stepped up our game in terms of testing and CI, and just reproducibility when it came to running the builds and running the stack, because suddenly we had to expand outside of just the four of us. And then I think that period lasted another two or three years. Probably in the past two, three, or four years from now or from this point in time, we've gone through another shift were we started thinking a lot more about end to end testings, security, kind of best practices around handling all of our customers data. And the real reason for that is now we actually have a pretty significant amount of both like revenue and customers, and reputation to lose. From day one we had no users so it didn't matter if the whole thing went down for a day or hours or whatever it was, the calculus didn't make sense there. But, today we have thousands of customers who are relying on Segment to polish their data, to not lose it, to treat it and handle it securely. No so for us the investment makes way more sense.
Jared Friedman:
And that last period roughly what scale did you have to reach before you started thinking seriously about those things?
Calvin...h-Owen:
Yeah, that's a good question. I think when we started having enterprise customers who are paying north of six figures per year, that's when it started to shift and we started thinking "Oh, we really have to take care in how we're handling this data because these customers are really depending on this to power their business."
Ralph Gootee:
So I think my stories a little different than the other panelists. This might be interesting. I would say all three of those different facets. I think I heard security, I heard scalability and I heard engineering best practices. We really have to treat them independently at PlainGrid. PlainGrid's used to build all... In California most hospitals, most jails, most heavy civil road work were used on all the tech campuses all of the different government buildings that go up. So obviously security from day one, I mean that's key to us. So we had no choice but, to always be super conscious of security and scalability for that matter because our customer's the only way we're useful is if we're used to build the project. Like we're the replacement for blueprints which means in construction if we're not working properly no one builds. And not building for a day in construction can seriously impact the bottom line in these businesses. For us the first two security and scalability were always key. Scalability is a challenging one because it's I don't know... I'm not sure there's a way to do it without it being kind of reactionary. You either over engineer everything and then you maybe never get a product to market, or you eventually have to deal with scaling issues. And for us we tried to architect it in such a way that we last through the foreseeable future, the foreseeable future would come, and some customer would find the first kind of flaw in the scalability. To give you an idea of the scale we've probably about... We have customers with projects that are like 500,000 blueprints with a ton of changes and maybe over five million annotations on one project. And we work offline as well so this is all downloaded onto a cache on the device. Our physical size on the iPad can take up like a 100 gigabytes for certain projects. So, we've had all kinds of scaling issues we've had to approach and we always try to balance it between engineering a year ahead of time but, not maybe three years ahead of time which is a trade off. So, you know find that trade off yourself. I remember when we were in NYC we had all these stories of companies that had built the most iron clad architecture and just never had a product to market so, that was on our mind while we were building that. For engineering best practices our V1 so of that code still exists in the product today. So, some of the code I wrote for V1 is still there, I would say it's probably our measure of technical debt as an engineering organization so that's humbling. But, at the same time I've seen some developers... We do everything cool now, CI, CD we run integration tests, we got a UI testing team. So, this is feedback I'm getting from the past. But, if you're a experienced developer it's pretty hard to write spaghetti crap code, you know, it's actually challenging most experience developers... Some people are just really good off the get go it's normally something through time and writing a lot of production products but, you just generally learned the tricks of not writing spaghetti code pretty quickly. And some of that code runs without heavy testing. The other thing to mention to is to properly scale test our product it would take days during integration test to download and upload all the data so, we really have a lot of functional testing to help us double check. But, the thing I've noticed with UI and unit tests is, we're a big fan of unit tests, we're a big fan of tester and development but, I've definitely seen developers write the most spaghetti test frameworks and the most spaghetti tests where they're just testing their mocks and their doing this in a middle of a production environment, you know, like a production need for release. And they spent so much time writing these unit tests that actually weren't testing anything rather than getting the product out the door. Engineering best practices is great, there's a trade off between writing a lot of tests and writing a good code from the get go.
Jared Friedman:
So speaking of test driven development a lot of questions were about the various engineering methodologies. Agile development, lean start up, test driven development, extreme programming. How do you guys thing about those? Do you adhere to any particular methodology in your company?
Lilian Chou:
I would describe us as agileish. So, we do not fully adopt any of those methodologies. Basically we find what works more for our team and aren't super dogmatic about is this agile or is this not? We run sprints, we have daily stand ups so, we have some elements incorporated in our development but, I think with anything you're going to be solving a lot of problems for your organization and you have to find what works best for you. I think it makes sense to take bits and pieces that work for you and adapt it to your organization. And then I guess its just the one other thing I'd mention on this is as you grow a big challenge is just constantly evolving how you work, right? So it's very different to work on a small team of four where everyone kind of knows everything, and everything is in everyone's heads versus a team who have 30 or 50 or for some of us 100's, that is very different. So I kind of constantly need to be assessing is this style of working or this methodology right for this stage of the company?
Diana Hu:
Yeah, I think the point of evolving the process is something that we've been doing a lot. Last year we were only with four engineers and building the product. So back then it was very messy, we were just trying to move as fast as possible really. So, 0 documentation, everything was kind of whiteboard and everyone could handle everything in their heads and build it out. And in terms of we really didn't do sprints because I think some of the engineers didn't quite like it as much and their were just too many big tasks and we just trusted different individuals to go and tackle kind of one area and then we would integrate it. But, now things are very different. Our new team is triple ready in the past eight months, for the AR platform product. We have a bigger team we definitely need a lot more process to be able to be efficient and communicate because you don't want duplicate, and not everyone knows everything and the system grows in a lot of complexities. So went from 0 documentation to a lot more. We used to have a literacy I but, now is very much [inaudible] that builds to all the architectural flavors to Lennox, Mac OS, Android, X86, [inaudible] , IOS etc. And it actually runs al the test in all of them and catches a lot of the bugs. We have coverage and all those things but, we also hae to train and get the team to be okay with having more process so, now we do not quite like a sprint but we do planning for the week. And then we reconvene and I think the teams are broken down into sub teams that work. People kind of flow in and out of different focusing areas. We split in three main focuses one is, sort of the backend, other work on the clients to make it cross platform, the other do API's, and the other, the big one is bringing a lot of the computer vision algorithms to correction. So, the funny thing is that everyone in the team has ended up being and trained up to be a computer visional engineer because that's sort of the core. We have CV algorithms in the server, we have CV algorithms in the client and the core algorithms that run. So, that's how it kind of shaped up to be but, we really don't have a process per se but, we do... Right now, Niantic does OKR so now, we have OKR's as well per quarter. So now, we're starting to plan longer and longer and have ideas... Before we used to have like a rough idea of where things would go but now, it's this, this and this and this.
Calvin...h-Owen:
Yeah for us we don't have any set process that teams have to follow at Segment. We have individual engineering teams sort of like what Spotify does. Their able to self organize, their able to run how every they want. If they want to do sprints that's cool, if they want to use Chera they can do that, if they want to use Asana, whatever it is their allowed to run however they like to run. The one thing that we've introduced over the past two years is similarly the OKR model. For those of you that aren't familiar, this was a model that was developed at Google. It's the idea of O's are objectives and then key results so, you have your one objective where you're trying to go and then key results that are suppose to be objective measure of how you get there. Such if you do every KR then it adds up to the full objective. So, those are something that we do on a company wide basis, every single team in the company gets together and puts together their OKR's basically, one week before each quarter and then for that three month period there just executing on that plan. And some teams grade those on a weekly basis and check in and say "How am I doing against these goals?" Some teams grade them on a monthly basis, it kinda doesn't matter. But, at the end of the quarter every team is saying "Hey, here is how well I did against my stated goals." Separately for the teams that I work with in particular we wend up running a weekly meeting where at the beginning of the week we end up planning out what we want to get done, then we have a daily stand up. I honestly don't care too much about the content for those meetings so as long as we're always discussing the most important problems. We're pretty big believers that the tools and process should be there to serve us not, the other way around.
Ralph Gootee:
My experience at PlainGrid echoes all the other panelists, its agileish, self organizing. Every team can choose the tools that they want. Maybe some things that would be helpful to you, from the beginning some things that we've always found that's been very useful in every engineering team are the daily stand ups. You've got to communicate on a daily basis. As engineers sometimes it can be a little difficult to want to communicate every day and not just program when you wake up but, that fifteen minutes, and you can do it remotely as well over slack or something, has always been really helpful in just kind of unblocking people. Time boxing, I think that a lot of these management practice's all rolled up into some abstract philosophies. And sprints are time boxed methods where you just not going to work on something infinitely, that's very helpful to kind of keep people's realities in check because often estimates can be off, and you know sometimes what we thought was going to take us a week can take us a month, myself included. So, time boxing is a good way to keep categories of that. I'd also say some other lightweight management techniques, you can probably employ right now, and your team is probably, one to one's, weekly one to one's. Such make sure if you're talking to people in a group and you're having stand ups, make sure you have some time to actually connect with people individually and get to learn more about their wants, their needs, and their emotions, and their careers.
Jared Friedman:
That's great guys. Okay, so now I want to change topics to the right way to work with non technical co-founders. And I think the topic that came up most commonly, in the start up school farm was the one that Ralph alluded to, which is how to deal with deadlines and time frames, particularly with non-technical co-founders? And I think particularly in the early days, so does anyone have thoughts on that? Any order is good.
Ralph Gootee:
I'll volunteer for this since my non-technical co-founders is staring at me over there. A few ideas here, if you got someone that's really non-technical, and I'm not saying my co-founder's really non-technical, I'm sure she can use a computer... But, if it's really non-technical this is an amazing testing opportunity for you. I mean, I remember I would write this software, and I'd be so happy about it and I'd hand it to her and as soon as she touched it would break. I mean I don't know she shook the iPad, she rotated it three times but, she had a way to break the stuff I wrote and that was amazing for testing in the beginning. So, that's one way to engage your non-technical co-founder. You eventually have to learn how to start pounding your deadlines, that's really hard to do. I never got really good at this, I'm always optimistic. Even to this day I'm like "Oh, yeah it's gonna take me a week." So, I never really got good at this I'm just aware that I'm optimistic about it and that I'm always kind of off by a week and she's aware of that as well. So, eventually I talked to other engineering leaders that keep two books, the keep like a separate set of books they talk to other non-technical co-founders about. Say "okay the deadlines here" and then you have another set of books that's actually when you think you're going to get the job done. I think one thing that can be really helpful is having a process, a very lightweight process of like "Here's were we're like testing and here's release." Because often some people might not realize that IOS has a release cycle that's not controlled by the developers, it's controlled by Apple. So, you've got to pod that into your plan, you have to plan a little bit. So I found that a little bit of project management goes a long way when you're dealing with a non-technical co-founder and even better if their good at project management that's a great place to employ them as well.
Calvin...h-Owen:
Yeah so all of my co-founders are technical, so we didn't really run into this problem in the early days, every one was kind of like "Oh, yeah I know what's happening here" if it slips we know exactly why it wasn't as big of an issue. I found more recently with deadlines the trick that I've started using is one on one's, actually. Asking each person what they think will happen over the next three or four weeks. And what I find asking one to one is that, one people don't anchor against each others answers so, you get a very unbiased view and it forces each person to think about the near term future what's going to happen. And second, each person is much more weary of the parts that they didn't work on and so, you get kind of a sense of where there might be trouble spots and typically, the end results is sort of like the wisdom of the crowds where it's kind of the average of all three or four answers. And so, I've started using that as a way to get a better sense of deadlines where each person has their prediction or mental model of what's going to happen. And then you kind of average them all out, to where it should be.
Diana Hu:
Well I guess in my case my co-founder didn't work in engineering or build products or ship products, it's mostly just in research. So in doing mostly a PhD so in that sense is kind of little bit non-technical in a sense of not having the experience of building and shipping products it's mostly kind of the algorithm side. So there was a bit of that work to kind of understand why certain things take long and why they need to be build a certain way but at some point, it was more he's letting me panel all of that. And he got engaged mostly on the external communications business, trying to find partners and all that. So, that's what happened sometimes that could be displayed whereas a technical founder you end up owning the product and the development. And having clear communications and at least trying to set expectations that last, that's the hard part. There's always the trade off between engineering and product and business, right? The trade off on when to get things and be able to communicate that clearly.
Lilian Chou:
So my co-founders technical so I don't have much to offer here. Not much to add either on the deadlines and time estimates to do your best guess, do some padding. I'd say that's a one thing kind of the non-technical front though, that I do think it's worth mentioning is so today, a lot of you are probably building your products and writing code especially if you're one of the technical co-founders, at least in my experience that changes. So, I pretty much stop contributing to our code base about a year into the company, and it's for us leaders while I can't speak for all of us but at least for me, there's a pretty big shift that happens once you start gaining traction. Where it's less about building a product for a market and finding that fit, and it shifts more into building an organization and company, and a system of humans who are going to build something long lasting and great without a lot of your involvement. So, I guess what I'm trying to say is you know, that at some point you may end up doing a lot more non-technical things so, get ready for that.
Jared Friedman:
That's great. Okay so, a lot of the companies and start ups school are at the phase where they are trying to figure out the right way to structure their early engineering team. If they want to hire people locally, if they want to hire people remotely, if they want to hire only employees who work full time or if they want to hand off pieces of their product to contractors or to third party development firms. Can you guys talk about how you would think about that if you were starting a company now? Advice that you'd give them. And I think this is gonna be the last question and then we're gonna open it up to the audience.
Lilian Chou:
I can start. So we did not start with contractors, we started by building like hiring full time employees. I think the main guidance I would give here is, you are building a product and a company, and your developing these core competencies it's a for the most part like if you end up contracting that out, you're actually not just contracting out through that technology expertise but, you're learning a lot and these early days of what's going to work for your clients, what the product actually is. And so, sort of having that external to your company I think is, really big loss in the early days just in term of all that knowledge. I could see an argument if your just getting off the ground and your doing contracting kind of as a way to evaluate potential new hires. I could see that definitely being a path. But again, that's sort of with the intention of building your team. On the outsourcing front I think it's a similar thing of you really need to look at what is your go competency, that's where your investing in what's your IP. You don't want to outsource that at some point if you do, your pretty much just a marketing company. And so, I'd say if you are... And different business models might have different requirements so, for some of you it may make sense to outsource a large part of what your building. But, for us we have experimented a little bit with outsourcing. I'd say our approach has really been to make sure those projects are very well defined and not on the critical path. So, that we can experiment, it is a different type project to manage sort of outsource talent and that way you can sort of learn and see if that works for your business. And if it does great, and if it doesn't it's fine. Was there also a question about remote?
Ralph Gootee:
Yeah, local versus remote employees.
Lilian Chou:
Yeah I think that depends on what type of culture you want to build. So for us it was really important for us to have our team together so, we hired up a local team. And then in sort of the last year or so, we have actually started tapping into remote engineers. There's definitely a little bit of a culture shift with that. We're still predominantly local but there are a lot of advantages including, being able to tap into other talent pools. And it has actually turned out to be a really good forcing function for documentation without explaining your code, explaining your decisions which is good for a number of reasons including, just like as you grow communication as one of the harder problems you have to solve. So, yeah.
Diana Hu:
So I think for us because so much of the core of the technology is what our company is. We didn't quite outsource any of that. We did contract people as more of an interview process. So we would work with and contract someone for a month before we gave them an offer. So that's what we did. But, we did outsource more things that we really thought they were not essential to our company. For example, building 3D models, some of the design, some of the writing technological documentation, websites. Things that we could do it if we had the time or wanted to, we knew we could do it for stuff like I could build a website but, our time was better spent then the cortex. So those were things we did outsource. And in terms of remote versus local because everything was so complex with what we build we choose as a culture to be more all on site. And to this day we are still building a team locally. But, I think you could handle... There's a lot of stories of companies that do remote but, you kind of have to start remote first.
Calvin...h-Owen:
Yeah, I can share a little bit from our story. So in the very early days we basically, only hired full time folks. We didn't experiment with contractors. Kind of the only case where we would outsource is when AWS could take some piece of infrastructure that we're building or like some new product feature and we could build on top of that. I think that was still probably the right move, especially in the early days it feels like a start up, is really fragile and you want a bunch of people who all pointed in the same direction and really kind of in it for the long haul. On the local versus remote piece we actually started hiring only remote people. We were really involved with a lot of different open source communities on GitHub, particularly there were some in Node in the very early days, some Java Script ones, Component etc. And so, these were people who we had been working with and collaborating with for years at this point who then, when we were ready to start hiring we just said "Hey, would you like to work more closely together?" I think it came with it's own set of trade offs. On the plus side we had access to these people who were insanely talented, not in the San Francisco Bay area but, who were used to working with us already. And who we very clearly built an insane amount of value for Segment. I think the trickier part for us to get right was around the communication piece and all kind of being pointed in the same direction from a product perspective. So, while we grew remote from about the first 10 or 15 hires we then only grew locally through about the next 70. And then only recently started opening remote back up. I think now, it's because we actually have enough bandwidth to create a really good remote culture where we put an emphasis on that documentation, that communication piece and we were actually able to really hire folks who have been remotely for years as well.
Ralph Gootee:
Some really good advice given on this panel so far. I think the trade offs are the thing for you to keep in mind the most. I was thinking back towards contractors and I'll tell you a quick story of a contractor. Our first two hires actually, as I was thinking about it contractors. One of them helped us organize our bills and our kind of like office stuff and the other one helped us develop some technology. Someone I worked with when I used to work at John Hopkins and even though he was a contractor at first, he actually... You know we didn't have money to pay him because he was my friend so, we paid him in equity. And he joined us as out first engineer eventually and helped us build some really core technology for our company and really was a great addition to PlainGrid. So, I think that even though I wouldn't suggest hiring a contractor immediately, we just did what was right for us at our company size and what we needed right then was like "Hell yeah, I need this guy to write some stuff for me and he'll take equity as payment. Wonderful because I don't got anything else to pay him in." So, I think just be open hearted with what you think is needed for your business right then. Know there's a lot of trade offs with contractors. The other thing to note is, this was my friend. We've had contractors I've hired as well. There's this ignorant feeling... I can't even now I feel it, when you hire a contractor that's like "Oh, great I don't have to manage this person," and that is so not true. Management of contractors is significant time for you as founder. That is not a free hire. It's sometimes is worse actually, it is more management time for a contractor than a full time employee. So keep that in mind as well, as the remote things also an interesting topic. When we started our company we tried to aim to have people in San Francisco but, yeah I realized actually as co-founders we split off for months at a time. We'd be like "Oh, yeah I'm gonna go back to Chicago for a month." And then one of my co-founders who work out of Chicago for a month or two and that was totally cool as well. We've gone back and forth on this over our years over, and over again between allowing remote, not allowing remote. Allowing remote when it's a friend or a really good best in class IC that can really develop a lot of things. Now, we're completely open as a company it doesn't matter. We hire managers remote, we hire local, designers, products, everyone, it doesn't really matter we're just looking for the best talent we can get. So, I think the best advice you've gotten already is just do what's right for you at the size of your company and know that there's trade offs with all these decisions but, there's no easy answer for any of these questions.
Jared Friedman:
Great okay, I think we've got time for one or two questions from the audience. Questions? Questions? One over there. You, you yeah.
Speaker 7:
So if you were to start another company again, how would you accelerate development process? What would you do different?
Jared Friedman:
Okay, I'm just going to repeat the question so everyone can hear it. If you were going to start another company now, what would you do to accelerate the development process? Any takers?
Diana Hu:
Yeah, sure. I mean, I think we spent too much time building technology on my side. So try to really get that product market fit as soon as possible. That is a challenge in terms of the areas that you pick because if you're in a kind of frontier technology space where like AR, VR or [inaudible] those lead times could be really long. Or think about what kind of company you want to build.
Lilian Chou:
I'd say get in front of users all the time. Find your user base, it's actually a lot harder to build a product in this vacuum where you're just like in your head and you're ideating with your co-founder and you're like "What if we do this, what if we do this. These all sound great. No, this idea sounds bad." But, it's like way easier to just put it in front of somebody and have them say "This is great or this sucks." That feedback loop is invaluable.
Calvin...h-Owen:
Yeah, I'd echo the sentiment on users 100%. Get out in front of them, start showing things to them. Make sure you're digging deep to make sure that you're really solving their problem. I think if I were to start again today, actually I'd build two or three versions which, I just intended to throw away. Maybe this is biased on prior experience but, I think that fact that you can get reps in a very safe manner where then, when you're really ready to start you can, just kind of go nuts building out the V1, I think is really powerful.
Ralph Gootee:
I love mobile development, I love Swift, I love Kotlin but, I don't think I would choose to have write four different platforms. If I was gonna write again I'd probably would've picked one of the cross platform development libraries. I don't know which is best, you'd have to tell me. If anyone has like... After this give me your strong opinions of which one of those is best. They all seem to have trade offs. Probably the best advice I could give though is, I'd stop coding a little bit earlier and I would start hiring a little bit earlier because I think there's a lot of me that wanted to passionately write those futures and if I had instead been passionately hiring other developers we would've built faster I feel.
Jared Friedman:
Interesting. Okay, one more question. Over there.
Speaker 8:
As a non-technical founder what do you suggest to communicate with the CTO's regarding the implementation of new features where the CTO refers to the technical founders insist on stability, and scalability and your driving features given by the users, what would be the communication style or?
Ralph Gootee:
Get them in front of the users, bringing them into the meetings, get them over the phone. While then, give the questions you know, kind of set it up a little bit where you're trying to bait the users into giving the feedback that you're looking for or you won't get the feedback that you're looking for in maybe there's some adjustment there too. So, get them in front of users as much as you can that's the most powerful tool you have in my opinion.
Jared Friedman:
Okay, thank you all so much. This was great.
用户评论