Marko Anastasov, Co-Founder at Semaphore, talks to Jonan Scheffler about how Semaphore optimizes a happy path for developers in the Continuous Integration/Continuous Delivery (CI/CD) space, having much love for Ruby on Rails, its community members, and its impact on software development, and if he could go back in time to give young Marko advice, it would be that if you're in the position that you want to build some kind of a product that you want other people to use and benefit from, just do it!
Should you find a burning need to share your thoughts or rants about the show, please spray them at email@example.com. While you’re going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you’d like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.
Semaphores and Shipping – Simplicity and Purpose in Software with Marko Anastasov
Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's developer relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry. And we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on developer.newrelic.com/podcasts. We're so pleased to have you here this week. Enjoy the show.
Hello and welcome to Observy McObservface. I am Jonan, and I'm joined today by my guest, Marko. How are you, Marko?
Marko: I'm excellent. How are you?
Jonan: Hanging in there. I have had a rough year. I don't know if you've heard there's a pandemic that’s going on. It's not a great time. But I think considering all of that, I’m doing well. And I still open every episode with “How are you?” And then very shortly after, I answer my guest who says something kind and this sort of thing that you say to someone who asks how you are in life in general, “I'm excellent. I'm good. I'm fine,” because having the actual real conversation that is like, “I don't know. I'm living in a state of constant existential dread, but I still wake up, and some days I put on clothe; that’s how I am.” But I quite enjoy podcasting, and I quite enjoy talking to you, so I'm excited to have you here. You work at this one company, right?
Marko: Yeah, I work at this little company I actually happened to have started. So we're called Semaphore. We are doing cloud-based continuous integration and delivery as a service. So to all developers out there, we’re at your service.
Jonan: Yes, you are. You're actually one of the CI/CD companies that I hear brought up very often in Ruby communities. I think maybe because you've been around for a while but also because I feel like you adopt the Ruby approach to things like CI/CD. Your tool is very quick to set up like a lot of things. I think it optimizes for this happy path for developers where I love things that I can configure, sure. Is it my top priority to spend my days configuring a CI/CD tool? [chuckles] No, not so much. What I love more than that is shipping. I feel like your product embodies that.
Marko: Well, thanks for that because you’ve described exactly our philosophy and motivation for creating Semaphore. So we did come from actually the Ruby and Ruby on Rails community prior to building Semaphore. And I don't know how polite it is to say this, but I think there's a difference between communities. Not every community has been as forward-thinking as the Ruby community, in my opinion. So it's where I caught the TDD and BDD bug and progressed a lot as a software developer. And there's this really nice balance between caring for the craft but not to the point where it becomes some twisted, self-sufficient hobby but at the same time, optimizing for shipping, optimizing for creating things. And I think at the root, I guess the people who created, for example, Rails and Ruby they wanted to optimize for developer happiness, and that’s where it grew from. Eventually, that led us to create Semaphore.
Back in the day, there were not many products or services for developers, but there were some pretty fantastic ones. I’m talking about maybe ten years ago or so. Definitely, GitHub was already big. But there was the user experience of Heroku, for example, for deploying code; it’s fantastic. New Relic was also amazing for monitoring what's happening in your application in production. And then as you build your app, write tests as a good citizen, we obviously came to a point where okay, let's implement actually a formal, continuous integration process in our company, which was back then just a tiny dev consultancy. The only way of doing that was to install Jenkins. I remember spinning up a Linode instance, and it took me about two or three days to set up like CI that worked for one Ruby project.
Marko: It was so different from the experience of everything else, all the tooling and products that we use. So we thought, why don't we try to make something which is totally opposite of that? In the beginning, we just thought about Ruby and the stuff we use. So what if any Rails developer could set up CI in a few clicks because we know where the code is, we know how to gem install, how to run tests, and so on? So why should it take so much time from building stuff? And so that's how it took off. And in the beginning, Semaphore basically only worked for Rails. Of course, later, we expanded. We now support pretty much every stack out there, but the Ruby community always has a special place in our hearts, and we used it to build Semaphore. So we also contributed I'd like to think through, hopefully, some helpful content around writing good, maintainable code, writing tests, getting better at design of your system, and so on.
Jonan: I appreciate it very much that you are here in this space. I think you make a good point about the Ruby community. I don't know that I would necessarily describe Ruby on mass as forward-thinking, but I do think that they are quick to adopt tools that lower the barrier to delivery. They want to do the thing that software developers enjoy, which is shipping.
Working in DevRel, I spend a lot of time talking with marketers about things like personas and user stories. And I hear a lot of things that are “As a director of engineering, I want to innovate, and I want to optimize.” And I'm like, just replace all those with, “I want to ship.” As a developer, I want to ship. At any level of management, I want to ship things. That's what makes me happy. As a developer, I want people to use the things that I build and to feel like I'm contributing value to the world, and Semaphore is a great example of that.
You had mentioned that Ruby was different, you think, because of the people who founded it, and I agree with you. I also think it is relatively small still. After a long time and a huge amount of impact on startups and on software generally, Ruby is still kind of a small community. And I think that maybe contributes to the fact that it is so tight-knit, and maybe that helps preserve those community values that make Ruby the special place that it is. Because if you're looking at a language like Java, there are a lot of humans who use Java and in small pockets within the Java community around a particular language or framework, you can have I mean, like a particular framework in this case because the language is all Java, but I guess I'm looking more at JVM languages then you have a lot of different communities. And each of those has a unique set of values. The Ruby community is a popular target, though when companies are starting out, and I think it might be that it has that uniformity is a stretch. But the uniformity of wanting to ship and build and create, I'm glad that you chose Ruby to start out. I can see the influence in your product. It is very [inaudible 8:25]
Marko: There is a lot of influence because the first -- well, I programmed in many languages in my formative years. But specifically, before Ruby, I was thinking C++ is the way to program. And so there came a time when I became really interested in building for the web because this was in the times of web 2.0. And I recognized if I wanted to create something that connects with other people, the web is just an easier way to reach more people, even if you're just a student somewhere in the backyard of the world. But I remember when I was doing market research, and I was like, okay, what's out there that I can use to build a web app? And at that time, I was pretty fluent with Perl. So there were one or two Perl frameworks that kind of looked cute. There was Django, of course, and there was Rails. And there was this, in my opinion, massive difference because, for example -- I actually don't remember what the Perl framework was about, but I detected it didn't have much traction. And I remember the Django website was describing the advantages of using Django being you could use this or that, or you could choose this templating language or that templating language. And there would be three or four options listed for basically every PC in the MVC stack. And I was like, I have no idea what I want to use. Why are you telling me this? And on the other hand, there was the Rails, which was like, “We got it all figured out for you; you can build a blog in 15 minutes.” And it was a pretty clear choice to me. I think that spirit is I prefer that way of designing software and then just building things.
Jonan: There are things that I want to be an expert in, and most of them line up with creating and not so much digging into which of the swappable ORMs is going to best suit my needs or set me up for success in the future or last as an open-source project or any of those things. But I know that if I'm using a framework, if you and I make a new framework today called SemaRails, and we make a choice on an ORM, the weight behind SemaRails drives the success of that ORM as well. It helps to perpetuate the existence of the entire stack when you have an opinionated framework wrapping it all.
So I think it would be easy for us to just prattle on about Ruby and Rails because we both like it so very much. But we are ostensibly here to talk about things that are changing in the industry generally and particularly how they pertain to observability, I suppose. And one of those topics I think that is most relevant right now to everyone is Kubernetes. You said web 2.0 earlier. And I was reminded of how I felt about web 2.0 as a term when it came out. And I've known many such hype terms since then. And Kubernetes, while it's the specific name of a product, in this case, I think the hype is perhaps outpacing the learning. How do you feel about that?
Marko: I think so. There's a difference when, for example, I'm not going to reminisce on the past and much longer but just for comparison, for example, Ruby on Rails had a big impact on software development, I would say in general. And it was just like one guy created it in the beginning and then created an open-source community. And on the other end, when you fast forward on what's the hottest thing today? Well, it's Kubernetes. How that came to be well, it was created at Google, and it's a copy of how Google runs its software, one of the biggest software entities in the world. Of course, the learning curve and the complexity is just going to be on a whole other level. But the goal, of course, is noble. The goal is to make cloud essentially, yes, easier to use, but generally to make it easier for developers to be productive in building in the cloud. So what I mean is before the cloud, you had VPS. And you would get a VPS. It's like you have to attend to it, and if it goes down, you're dead. And the cloud is like, okay, you can spin up a resource in the beginning in a couple of minutes, and you can shut it down at any time, but that was the first wave of you could get resources on-demand, but essentially it was like VPS on demand. And now we're having cloud resources that, of course, it's on-demand, but it's instantaneous. Maybe there's a free tier, but regardless, you pay only what you consume; whatever that is scaling happens automatically, the idea behind serverless. So if it's runtime, okay, the cloud provider takes care of scaling up the resource that's processing the requests to just the right amount, so it doesn't break. And it scales it back down as soon as the traffic goes down. And that's all great, but our applications to really take advantage of that had to adapt. So we got to build some stuff that's composed of pieces that are ideally all disposable at any time, horizontally scalable from zero to infinity. And for that to happen, the critical step was to standardize what is an app? What is a unit of deployment? That's what made Docker the big thing. Even though containers are not exactly new, there were technologies like LXC Linux containers before that.
Jonan: Yeah, the original Docker was based on LXC, and they diverged at some point. I think you're absolutely right, though, that it gives us a concrete way of thinking about a unit of an application because it's not an instance. It's just a very difficult thing. Having worked at Heroku, it was a difficult concept to discuss at the time. We talked about a dyno. What is that? Is it an instance? It's hard to describe. Well, the dyno is a container, really. And the way that we think about things now has leveled up to a point where we can have that horizontal scaling that we need and want to operate at the massive scale that we have. Complexity of our systems overall is going up. But I remember in the early days of Heroku, a zero-downtime deploy was a legitimate value proposition over alternatives. And it's absurd to think today that deploying your website would bring it down.
I went to my bank the other day in the middle of the night for some reason or another, and they had “We will be right back. We're under maintenance.” And I was like, “Wow, that's terrifying.” In 2021, my bank is under maintenance. For hours, they'd take the site offline. That's just scary to me. They would be so far behind on the pace of software development. Kubernetes is absolutely about getting us closer to a world where we ship, but I think of it more like Rack in the Rails world, Rack enabling rails, but Rack by itself, not being a thing that you would want to build everything on top of.
I think that there will be...well, there are currently many attempts at building a developer-centric product on top of Kubernetes that makes it as easy to use as it needs to be to gain wide adoption. But then the wide adoption piece is already here. Like 70%, I think I read 70% or 75% of companies say they're running Kubernetes in production. I suspect in many cases that is yes; our engineers are using Kubernetes because our CEO came and said, “Well, we should definitely catch up with Kubernetes. We want to be on top of the bleeding edge and innovate. So we're moving faster than our feet in some ways.” You've got KubeCon doubling in size year over year or more…well, up until the pandemic. And those abstractions have not really appeared yet. However, it's still having this tremendous impact. I imagine that it has changed how you're doing things at Semaphore to some degree.
Marko: A lot, yeah. Well, maybe not Kubernetes itself or that much as Docker. Docker was actually one of the few reasons why we had to rebuild Semaphore from scratch and launch a 2.0 product that was in late 2018, the change in the way we take care of the delivery pipeline. So what happened with the pipeline in abstract terms, not necessarily in CI, is whoever decides to use Docker exposes a new abstraction layer in all phases of development and deployment. You expose it to the developer who's writing code. You exposed it to the CI system. Obviously, you got to find a way to run it in production. For us, in the CI/CD space, prior to that, Semaphore was relatively by today's standards, I would say simple. It was able to run a build which was either a series of steps or some number of parallel jobs like you would maybe run a lot of cucumber in our spec test in parallel to speed up. And then, we had deployment, and you would define a server, which could be like production or staging with some configuration. And then you could say, “If the master is green, automatically deploy to production,” for example. And then, we execute that in a separate process and keep everybody updated about what's happening. Now, when you introduce a Docker container, you have this big thing like you have to typically build at the beginning of the pipeline. For example, if you just happened to run some parallel tests, you don't really want to rebuild the Docker container in 16 of your Cucumber parallel jobs, right?
Marko: And just that very simple use case basically completely invalidated Semaphore specifically. Like, people would ask, “How can we do this?” Well, you got to rebuild. And so this kind of led us -- Like, there were a couple of other maybe use cases that made us aware how this was needed. But basically, we needed to build enough flexibility so that you can design pipelines of arbitrary complexity just in terms of execution flow but also in terms of like okay, if you build a Docker container, we can offer you to maybe not have a lot of overhead communicating with a cloud provider but actually store it in a private registry in Semaphore and then pass it over very quickly to the next stages in your pipeline.
Jonan: You pass along that artifact. You don't have to rebuild it everywhere. It's part of what enables this horizontal scaling. When you have to rebuild, I mean, building a container takes a bit, and you want that ephemeral nature, so you can spin up a new Pod to run your container in, and then they go away. And this whole process being automated is valuable. I’m seeing a lot of alternatives around products in the CNCF ecosystem. I feel like we're getting to a place now where there are just so many options to accomplish any given task that we're back to that trap that some frameworks suffered from a decade ago and, to a large degree, have very much outgrown. You had mentioned specifically Django, and Django is a very different world today, of course, then it was back then. Like Python in the community there as well, they have a different approach to the world. But where I'm going with it is actually to talk about Docker.
I think that Docker did drive container adoption forward. And then, at some point, other container runtimes started to pop up. And I think it's in part because Docker scared people. They twisted the thumbscrews a little too tight. It became clear that they intended to squeeze the ecosystem for profit in ways that were uncomfortable for people. And I wasn't down. I think they started to display Evernote quite-type issues where Evernote was this product, and it was for taking notes. And then the notes were everywhere, and it was fast, and it was easy. And then it wanted to be my scheduling system and include every feature. It wanted to be a platform. My note-taking app wanted to become a platform, and Docker wanted to become a platform. And Docker, in the sense of a deployment platform, is a deployment platform, but it doesn't mean that every software company is destined to offer every feature that exists in software. So we had this pushback where we started to see other runtimes, and I see the same thing happening in Kubernetes now. You have alternatives. Rancher, which just got acquired recently, if I remember correctly, had K3s, the smaller one; when you lower the number, that's faster. I think that there are going to be a lot of those pieces going on right now because it's such a huge land grab in Kubernetes. Every company is trying to scoop up this fresh market share where if you say Kubernetes in the right meeting, you just sold a million-dollar deal because “Well, do you support Kubernetes? Our CEO is very excited about Kubernetes right now.” And then it's done. I wonder if you have any predictions for where we're headed from here that we can hold you to in a year when I have you back on this show. What do you think is going to happen over the next year that you maybe have seen before or anticipate?
Marko: Well, over the next year, I hope we get to visit more than a grocery store for a start.
Marko: And if that happens, Kubernetes will take care of itself.
Marko: I'm an Evernote user in exit, so I know what you mean. You mentioned a couple of things, a couple of great points. The trajectory of Docker as a company is interesting. I think it's a cautionary tale. I think if I remember correctly, they raised hundreds of millions of VC, and it didn't work out. That's the conclusion. It didn’t meet those expectations. And I remember when it was happening, I was thinking, how can this possibly work? Like, they're building very important, very innovative piece of software infrastructure for building stuff, but it's open-source. How exactly is this going to -- It is ending with things like Docker Hub introducing these pretty aggressive limits. And externally, this may not be the case, but from an external standpoint, it looks like okay; how can we make any money whatsoever? I hope there's more to it.
I think it’s also a story of basically the whole software development community depending on a VC-backed, for example, registry of software, like that, was kind of -- saw it as a public space, like a public parking lot for taking dependencies and all of the open source stuff. And then one day somebody pulls the rug. For example, at Semaphore, we quickly built some internal infrastructure so that our users are not actually depending on Docker Hub, at least for this Semaphore stuff because, for example, previously, Semaphore was using some stuff for Docker Hub without users explicitly asking for it.
We’re talking about these things, but one question is, why are we even talking about containers? We started talking about the love for shipping and making stuff. I always return to that. There's a threshold where you're working more for the thing than that thing is really helping you, maybe. So I think it's a big question for every team to assess. There is really a lot of really cleverly disguised marketing these days in the software development industry. The biggest corporations are basically imitating the language and behavior of what ten years ago was coming naturally from small and typically bootstrapped companies. But the motivations and the end goals are quite different, and actually, the target market is actually quite different. So it may happen that a company with maybe a few thousand engineers or even a smaller team just basically falls for PR for which was actually intended for your bank because your bank completely ignored this cloud thing for the last 15 years. And for them, Kubernetes is maybe the first thing where they see Kubernetes is our ticket to the cloud. They're just about to start catching up to what you were doing ten years ago with Rails and pushing stuff to Heroku. But for them, it has to be more serious. It has to be more complex. It has to support every stack and so on. And when you want to have this kind of portability and not depending on any cloud provider, you end up with something where the user experience has a bit less to offer than a platform which took care of everything. So it's a trade-off.
Jonan: And I think it's because that's not the priority. You tend to follow the market that exists for your product. And a lot of companies struggle with this when they are VC-backed, and it comes time to pay that bill. Like the idea of Ruby gems becoming a venture-backed company is terrifying to me. Like, where's the money going to come from? Someday, there's going to be a switch that flips. And VCs on that board want their money back. It's a return on investment game. It's not a charity game. They're in business. So when you, as a company, experience success early on in the enterprise space, why wouldn't you prioritize enterprise customers instead of individual developers? It makes financial sense until that Amex stops swiping because the CTO asked the devs on the team, “Hey, what do you think about this?” And they were like, “Nope, Nope, Nope, Nope. Here are a bunch of open-source alternatives,” that only exist because that company twisted the thumbscrews too hard, too early trying to get the money from the enterprise customers who see that as such an innovative opportunity, an opportunity for innovation, and they want to finally get onto the cloud. That's a good perspective to have, I think. If you were to travel back in time, we only have a couple of minutes left, but if you went and met young Marko and you were going to give yourself some advice, what would it be?
Marko: You mean the containers or in general?
Jonan: [chuckles] I’d be like, “Hey Marko, invent Kubernetes. Buy Apple stock.”
Jonan: What would it be, like with regard to your career? People listening today maybe they're just starting out, and they aspire to be in the position where you are. They want to co-found a company and bootstrap it and grow it to be as successful as Semaphore is today. What would you tell them?
Marko: If you're in the, for example, the position that you want to build some kind of a product that you want other people to use and benefit from, just do it. [laughs] There's no other possible advice than just go do it. But always keep in mind who you're building it for, who is your user or a customer. Once you have clarified who that is, once you know these people, once you have some of them that you can talk to, never lose sight of them and never stop being in tune with their needs because they're just going to guide you to what you need to do next. And as you're building, build the thing and avoid complexity.
Jonan: [laughs] Complexity comes back to haunt us as software engineers because it's so much fun to know the thing, but it's more fun to ship the thing. I agree with you on all points, at least the ones you made most recently. It's been a pleasure having you here, Marko. Thank you so much for joining us today. You want to come back in a year, and we can evaluate your prediction ability?
Marko: I would love to, and it's been a great pleasure. So thanks a lot for inviting me.
Jonan: You're very welcome. Have a wonderful day.
Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on therelicans.com. In fact, most anything The Relicans get up to online will be on that site. You'll also find news there of FutureStack, our upcoming conference here at New Relic. We would love to have you join us. We'll see you next week. Take care.