The Relicans

Cover image for Shuffling Complexity – The Yellow Bricks are Made of Pain with Jeff Smith
Mandy Moore
Mandy Moore

Posted on

Shuffling Complexity – The Yellow Bricks are Made of Pain with Jeff Smith

Jeff Smith and Jonan Scheffler hang out and talk about complex issues, pragmatism, and being happy in the Ops.

Should you find a burning need to share your thoughts or rants about the show, please spray them at While you’re going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you’d like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.

play pause Observy McObservface

Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's developer relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry. And we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on We're so pleased to have you here this week. Enjoy the show.

I'm here today with my guest, Jeff Smith. How are you, Jeff?

Jeff: I'm doing good. I got to tell you, this is probably my favorite name for a podcast, ever. [laughs]

Jonan: We did it. It was a collective decision. When I was naming the podcast -- I started Observy the first week I was here at New Relic, and they came to me, and they were like, “Hey, we want to go on a podcast. Or we want the Chief Product Officer to go on a podcast this week.” And I was like, “I've done podcasts. I can do that.” And then it came into the weekend when no one is around on my first week, and I have to come up with a name because you got to start syndicating the show because we're going to ship next Wednesday. I was like, well, let me ask in Slack. I asked in Slack, and we had three or four options. And then I took the vote to Twitter, which of course is how we ended up with Observy McObservface.

Jeff: You know, now that you say that, I think I remember that poll. I think I remember seeing that coming about and someone saying, “This is why you never do this on Twitter because now I'm stuck.” [laughs]

Jonan: Yeah, that's what we're going to do. But I stand by our decision. You got to give the people what they want. So we are here ostensibly to talk about observability, and DevOps is your specialty. So I imagine that you have some things to say about observability. I want to actually give people some background first on who you are. Tell us who you are, Jeff.

Jeff: Sure. Like you said, my name is Jeff Smith; currently, Director of Production Operations at a company called Centro located in Chicago. We're a digital advertising software company, so we do a lot around campaign management, workflow management. If you're not in advertising, the best way to think of it would be like an RFP system for ads and advertising. Before that, I was with GrubHub. I helped build out their SRE team there as well. And it was just a year or so before that that I got introduced into this DevOps community and DevOps culture through DevOpsDays Chicago, and that kicked off my love affair with the whole DevOps movement. The first time I got exposed to that, I was like, yes, this is exactly what I've been seeing throughout my career over and over again, this tension between developer and operations and the fuzzy line of responsibilities. Since then, it's just been a fun ride. I wrote a book, do a fair amount of speaking at different events and conferences; not so much these days with everything being virtual, but it’s still a pretty fun ride.

Jonan: What is this book that you speak of? Well, I know the title, and that's all I know about it. But the title of the book is Operations Anti-Patterns, DevOps Solutions. Tell us a little bit more about that.

Jeff: So one of the things that I noticed in the DevOps space was as I started to get more and more involved, I started hearing all of these awesome solutions for companies that most people I know don't work for. So learning about Netflix, learning about Google and how they do things, that's great, but they've got more PhDs than my entire company has employees. So I started to think, well, it would be nice to have a book that people could use that weren't the center of Silicon Valley that had to worry about things like profitability. I love it when people are saying, “Oh, Uber does it this way.” And I'm like, “Yes, but Uber doesn't have all the constraints that a lot of other people have, like making money.”

So I wanted to write this book as a guide for those people that may not be at the bleeding edge of technology, that may be in more legacy organizations, and give them tools that they could use to build a DevOps culture from the ground up. Because in Silicon Valley, it's easy to think, or even in tech generally, if you're a super tech-focused company, it's easy to think that your leadership is reading these DevOps books and reading it in CIO magazine and pushing DevOps down from the top. But in a lot of places, it's more of a groundswell, and you have to figure out ways to push this agenda from the bottom up, and that's what the book tries to lay out and say, “Here are the reasons you're experiencing these pain points, and these are the things that you can do to alleviate them without necessarily having buy-in from the top of the organization.”

Jonan: Which is, as you very eloquently put it, a much easier solution when you're in Silicon Valley. You have the Googles of the world where money is just raining from the sky all the time, and everyone can do whatever they want. I'm sure Google has roadmaps and things, but it's not obvious to the outside world because they dabble a little bit in Balloons to send WiFi and also self-driving cars and Kubernetes. It's just like, “Well, we don't have to worry about making money this decade so let's just see what we can come up with.” And then there's the rest of us we're like, “Oh, we got a couple of months here before it's going to be real bad if we don't keep shipping.” So it's interesting.

Jeff: And I don't begrudge those companies for that; that's the beauty of success. But at the same time, I think there is a bit of a disservice that happens when we're constantly being fed blog posts from Netflix, Google, and Apple because people see that, and they start to cargo cult everything that they see without stopping to think is this actually a problem that I have? Is this really my pain point? Is Kubernetes really going to solve everything that's going wrong in my life, or is it something more basic like, hey, I need to learn to communicate better with developers about infrastructure changes, or I need to make sure that developers are more empowered to actually deliver and execute on their job? So I think we have sort of a worshiping culture specifically in America, like celebrity worship. I think the same thing happens in technology. We worship these companies because they're so successful without critically examining all of the components that go into that and what's replicable and what's not.

Jonan: Extraordinarily well-stated, Jeff. I really like your opinions and you. [laughter] You have a good perspective on the world. I think it's really common. We get caught up in things in tech, especially because you look around, and there are all of these people who are waxing philosophical in the blog posts. I work in DevRel, so it's easy to hear people talk about thought leadership, and I try to shut it down when I hear it. [laughter] We could use another word like community leaders, literally anything else but a picture of someone who is leading my thoughts around with a little hook. That's not really I think where most people end up or certainly not where they think they are, but it is very easy to look at blog posts from big companies and see that as experience that you can apply directly to your life no matter where your organization is, and it's just not true, Kubernetes being an excellent example, very popular for people to talk about lately. I, myself, I’m a Kubernetes thought leader. I have been installing it on Raspberry Pis, and I've been installing those Raspberry Pis into Yoda dolls, robotic Yoda dolls. So I'm going to have a Kubernetes Yoda cluster very shortly. I'm doing important work, Jeff.

Jeff: That’s amazing.

Jonan: It's amazing, right? I think it's really important that the world have these. But why wouldn't I want Kubernetes? I think that a lot of people see Kubernetes as a path for themselves towards their own paths, their own Platform as a service. Everyone's used Heroku. They love Heroku. Also, it costs money, which is very disappointing for people when they find a product that both is awesome and costs money. So they try and build their own a lot of the time. And a lot of people are seeing Kubernetes as the best option in that direction. I think something like 80% of companies now have Kubernetes somewhere inside the house, whether they're just experimenting with it or they're actually using it in production. But your claim, your wild claim, and I've found many people who agree with you: Kubernetes is not always the right answer.

Jeff: It's absolutely not. In fact, I'll go a step further, and when I talk to companies of a small size that are like, “We're thinking about hiring an Ops team,” The first thing I say is, “Don't. Stay on Heroku as long as you can.”

Jonan: [chuckles]

Jeff: When you leave Heroku, it should be so painfully obvious that that's the right choice, that that's the time to do it because I don't remember who said it, “Not everything has a price, but everything has a cost.” And the cost of Kubernetes is complexity. And it is this sort of unwieldy thing that you have to bring into your organization that you're now responsible for all of these components. And at the end of the day, all you really want is customers to be able to access your application. You don't want to be worried about Cube DNS. You don't want to worry about etcd. You don't have to worry about all of these things that are essential to running Kubernetes unless it's giving you a very particular advantage, whether it be through your operational tasks or whether it be through some scaling event or anything like that. But you should be able to definitively express what it is that your pain point is and how Kubernetes is addressing that.

People think of things like, “Oh, well, we want to be able to deploy faster.” Okay, but why? Do you really need to be able to deploy faster? I work at a company that we are looking at Kubernetes, full disclosure. But when people say, “You'll be able to deploy multiple times a day,” and it's like, well, I don't have a customer base that wants deploys multiple times a day. We want the ability to deploy anytime we want. We don't want a deployment to be an event. But shipping out code every time it's merged, there is no real business advantage to that for us. Some other organizations there is this idea of rapid feedback and being able to get this but for us, it just doesn't apply. So I think Kubernetes is just something that really should be looked at carefully. You should really examine all of the complexity that you're bringing in and if it's going to pay off because it's not going to be some part-time job for someone managing the Kubernetes cluster. You're going to have a team behind that. And with that team comes a ton of responsibility, just the management overhead, healthcare benefits, all of that. If you can outsource that to someone like Heroku or hell, even Elastic Beanstalk or something like that, just punt on those responsibilities altogether for as long as you can.

Jonan: So you can focus on building things for your users.

Jeff: Right. Because guess what? No one has ever been like, “You know what? I was thinking about switching to this other product because they use Kubernetes.” No, people are switching to products because it solves a very specific problem. What's underneath the hood they don't care about. And soon, we're going to get to a place where the same is true of developers. Like, I don't really care what's running my stuff; just tell me what button to push to get it out.

Jonan: Yeah. I think getting to a place where we all have that button is a priority for most people. I'm in this business because I enjoy delivering, I enjoy shipping things. That's what I do. I'm in a unique segment of this business where what I ship is mostly nonsense that I don't have to maintain in production. [chuckles] But I very much enjoy that process of building. I'm a creator; I'm here to build the thing. I also get, because I'm kind of a nerd, why it's fun to dive deep into those technologies. It's actually really interesting to me to try and understand Kubernetes. I don't know that if I was running a whole company, I would make the choice to take on that burden for the organization to learn this new technology when I have an opportunity to just focus on the things that we're uniquely good at. Nobody knows our customers like us. I'm really good at shipping features for my customers. I'm not going to be the best at managing Kubernetes, maybe ever. Kelsey Hightower is a person I'm probably not ever going to outrace at Kubernetes knowledge or the rest of the Google team, but I may someday become an expert. The point is that I'm bending a lot of cycles. Why not focus on your competencies as an organization instead of taking on these other pieces?

So this is a common thing for engineers, developers, software people that we have this layer of abstraction, and we will go too deep, too quickly. And while Kubernetes offers a lot of advantages, it's much less complex than trying to do the things that Kubernetes does at scale. It's not yet to a level where it's a button I click, and it's not really close yet. There are several more layers to get there. But why is it you think that engineers try to be experts at the deep parts of those? Is it just because it's fun? Is it just because we're inclined to go down that nerdy rabbit hole and tinker with all the bits? What is it that inspires us to believe that we can build the whole house from scratch?

Jeff: So, I'm glad you asked that question because it actually dovetails with something that I've been noodling on in a blog post that I've been working on, trying to collect my thoughts around this. And it's a half-baked idea, but, you know, it’s a safe space; it’s just me, you, and the internet right now. So, I'm sure no one's going to misconstrue my views.

Jonan: [laughs]

Jeff: So this blog post idea came from this thought iteration I was going through of like, what is it that developers want from Ops? And what I realized is in a lot of scenarios, what developers want is understanding, is the ability to abstract the things that they don't deal with into an organization or team that they can hand off their issue to. And I want to separate this from throwing it over the fence like it's a non-caring thing. But to your point, developers work with abstractions all day long, and they don't always want to dig deep into each of the abstractions. You have different developer personality types. And I'm trying to come up with constructs for this. Right now, this is a working model, but I think of the fundamentalist engineer and the essentialist engineer. The essentialist engineer is a guy or girl that's saying, “All I want to do is ship features. That's what excites me. I don't really need to know or understand all of the underlying mechanisms until it actually surfaces at my level, and I don't care particularly to know. I'm more interested in solving this problem.” And for so long, in an Ops perspective, at least, we discounted that developer as not being senior or not being skilled enough because they didn't want to dig into that layer of abstraction. But truth be told, everyone has a particular layer of abstraction that they're comfortable with working in, and that doesn't make them any more or less effective as an engineer. It just means this is the lane that they've chosen. You've got the fundamentalist engineer that wants to go deep, wants to understand everything. And that becomes the go-to person that eventually ends up building the internal tools team because there's always one of those guys that's like, “We need a team that's going to do XYZ.” And everyone's like, “Yeah, sure. If you want to do that, that's fine. I just want to ship features.”

So I think that when you choose Kubernetes, you end up assuming that everyone is more of this fundamentalist engineer. But one thing that we're learning at Centro as we're going through our Kubernetes evaluation processes as we're interviewing and talking with the stakeholders, we realized, you know what? Kubernetes isn't just the stakeholders for engineers; it’s for QA as well. And QA is like, “I don't care about what's underlying. The only thing I care about is I click a button, and I get an environment that's capable of supporting end-to-end testing. That's all I care about.” If I turn around and say, “Here, now you've got complete control of your own environment. Here's a bunch of YAML files you can modify.” They're like, “Eh, that's not what I want.” And when I survey most of my engineers, that's not what they want. There are a select few of my fundamentalist engineers that are like, “I need total control over this thing because that's what excites me.”

So I think Kubernetes is going to exist in this weird space where it is a boon to some and a pain in the butt for others. And I think Ops’ job is going to be figuring out that delta and how to bridge those gaps so that we can have these fundamentalists engineers getting in the weeds as much as possible but at the same time, providing a nice, thick layer of abstraction for people that just don't care; they just want to deploy their code or test their code or demo their code because as infrastructure becomes easier to deploy, more and more people are interested in those environments. So now product is like, “I want to be able to create an environment so I can show off a thing.” Sales teams are like, “I need to build this environment so that I can demo this very specific feature with this very specific use case,” and that's just going to continue to grow.

Jonan: I agree with you. I think we need to add one more, the fundessentialist, which is I think where I lie, where I want the layers of abstraction, and I also want all the knobs. It's a very confusing personality type. [laughs] This is part of what inspired me to start off in Rails. When I started getting really into software about a decade ago, it was on Rails. And it was because I had the excitement of watching that DHH video that everyone goes on about where you ship a blog in 15 minutes. And seeing how long it had taken me to try and do that with Java and PHP and other things that I'd used, and I wasn't even really close still with all the nonsense. I hope no one finds the kinds of websites I made back in the day.

Jeff: [laughs]

Jonan: But there's some nonsense that I got up to on the internet as opposed to today where I make the Kubernetes Yoda cluster, the very important Kubernetes Yoda cluster. So Rails has this high level of abstraction. I can just say, “Rails new,” and I can deploy that thing, and it'll run. And I've got a “Hello, World!” page up on the internet, but a whole lot of moving parts under the hood made that happen. And I have the ability in Rails to dig into those pieces and twiddle with them when I want to. But as people in the Rails community often say, “When you go off the happy path, you're going to have a bad time.” They call it the happy path for a reason. The Rails’ way is probably the way you want to stick to. Do you think that in Kubernetes or similar technologies that exist in this space, there is a happy path? Because I feel like it isn't actually that opinionated yet. Rails is an opinionated framework; they make a lot of decisions. I don't want to choose how you're going to handle the Rack::Request or whatever in my application. I just want you to make a good decision because you are a framework designer, and I'll just trust that that's the best one until I find a reason to believe otherwise. But Kubernetes seems to be taking more of the toolbox approach that people take, “We're not a framework, we’re a toolbox. Here are all of the things; here are just a pile of knobs, go nuts.” Instead of “This is probably the way you should build this thing.”

Jeff: I'll answer that in two parts. I think Kubernetes will remain a toolbox. They're just going to give you a bunch of tools. So Kubernetes, at its core, will be that. I believe the community is going to spring around that, and you'll find these frameworks that say, “Hey, if you want a basic microservices implementation, here's what you need to do. We've wrapped all the Kubernetes magic for you. And this is our very opinionated way that we think a microservice should run.” So I don't think it's going to come from Kubernetes’ core. I think it's going to come from the Kubernetes community as we settle in and what these patterns look like. And then you'll probably be able to choose from those frameworks, too, where each framework has a different set of trade-offs depending on your needs, and you pick the appropriate framework for your problem. So I see that community growing. And I think organizations, too, will continue to implement their own internal happy path where they say, “Hey, if you want to go what we call the yellow brick road path, we say, here's the yellow brick road. You do that; you get all of this stuff for free. If you want to go on your own, that's great, that's fine. You can do that.” But you still need to meet these requirements. I still need metrics. You still need a health check endpoint. You still need to be able to have your containers migrated at a drop of a dime. You still need to be able to handle storage migrations. You put all those requirements on the other path. If, for whatever reason, the deviation from the norm is important enough to your solution that it's worth the effort and the heavy lift of implementing all those things, have at it. But in most cases, people are going to be like, “That's a lot of stuff that I just don't want to deal with so let me just continue on the yellow brick road.”

Jonan: I like yellow brick roads. I'm a fan. I just want to be able to take them apart when I want. I think there's a way to build those things intelligently. You have a framework that is presenting some high-level abstraction for a lot of concepts, but you do want to somewhere give people the ability to swap those pieces out, and I think that Kubernetes at least offers that. If you go and look at the CNCF homepage, there is or there used to be, at least I think they've changed it, a huge list of potential options for everything from log management to metrics. You've got all of these pieces that you can swap in and out of this system. I think from a high-level DevOps perspective when you are looking at that broad array of technologies and also watching the proliferation of DevOps as a methodology of managing these systems, which is, I guess in itself kind of an abstraction, those two playing together sometimes seem like they're at odds to me. We have this explosion of complexity at the same time that we're trying to simplify the way that we manage these systems. I wonder if you have any thoughts on that.

Jeff: So, again, I wish I could remember who to attribute this to, but complexity is never destroyed; it’s just moved around. So I think at the core of things, we're never really making things simpler; we're just choosing different trade-offs. And that's something that really came home for us during our V1 implementation of our infrastructure platform is that we made some trade-offs with our eyes wide open. But other trade-offs, we didn't realize we were making, and they silently came back to bite us in the butt like that flexibility that you're talking about. We said, “People are going to want to do it this way. And this is going to be the easiest way to do it.” So there are certain assumptions that we just made. And then, when those assumptions didn't come true, it was like, okay, now it's like moving heaven and earth to be able to make this thing configurable. And you try to convince yourself that just this little tweak will make it usable. Oh, just this other tweak will make it usable. And then a year and a half later, you're like, dear God, what did we do? We've got so many tweaks. So we've got to burn the thing down and start over. So I think a healthy dose of reality is necessary when we’re building these things. So when we say that something is simple, that we understand that it's simple at this point, but that just means now, further down the change somewhere else, it's complex.

So it is simple to create an end-to-end environment with a single command. The complexity happens on the back end when it's like, okay, based on this command, I've got to make 20 different decisions based on what you told me. It's complex to ask for all those individual parameters upfront in the command, but then it becomes really simple to figure out what it is I need to do to make that happen because it's all been specified on the front end. So you're really just moving these pieces around, and you have to understand who is best suited in this particular opportunity to bear the brunt of that pain. And we're seeing that a lot in the cloud, whereas Dev and Ops are starting to be measured by the same things, an engineer comes to me and says, “Hey, listen, the database has been running really slow.” You look, and you say, “Wow, this query that you guys got is total garbage, and it's running pretty bad, and the frequency is increasing.” “Oh, well, it's going to take us two to three months to rewrite that query.” I'm like, “Or I can click this check box, double your instance size, and we can all walk away happy. “I could put the complexity on them, and I could make an argument from a technical purist perspective that it's better that we do it that way. But in reality, I'm sucking away valuable developer time, which is like an unreproducible resource to prevent myself from spending what? An extra seven or eight cents an hour? [chuckles] So even though you got to hold your nose and do it, as an organization, it's better to make that trade-off. So as you're building these things out, it's really about being honest about yourself about where this complexity exists and evaluating your decisions appropriately.

Jonan: This is a very pragmatic approach. I appreciate it very much. I especially appreciate having permission to write all the bad queries now not being –

Jeff: [laughs]

Jonan: Now I can just YOLO with my queries like I've always wanted to.

Jeff: It's got to be a feature that makes money though if it's not a feature, it doesn't count.

Jonan: Yeah, [chuckles] get money from it, like the Yoda dolls, for example, yeah. So this proliferation of DevOps throughout the Ops community, how do you think that's going? Do you think this is being pretty widely adopted? I know according to a Google trends search for DevOps, it is wildly popular, and everyone is a DevOps engineer now, which I personally feel is not a job title per se. It's being an agile engineer because you do stand-ups. But what do you think? Do you think that people are adopting this, or is this just like a stairstep on the road to the next thing? Some of these things grow and fade pretty quickly.

Jeff: I think the adoption rate amongst tech companies is going well. And I'm using the traditional definition of a tech company. But do I think that all of these HMOs, these regional insurance carriers, are doing DevOps in their shops? Probably not. Are these companies that have a very real-world product that just happens to leverage technology to get that out to the masses doing DevOps? Probably not. And I don't have stats, but I would reckon that they are probably a larger share of the business community than we would care to admit. So I would make the argument that certain sectors are seeing explosive growth in DevOps. Then you have to ask the question, like, are they actually doing DevOps? If they have DevOps engineers, part of me says, “Probably not.” But I think we've still got a long way to go in a lot of areas that we tend to shun or set aside. It's just like how we call the platform at work legacy, and this is the new shiny. But the legacy platform is the one bringing in all the money. [laughs] So it's not really legacy. I think we have a similar situation with the economy where it's like, okay, yeah, we're calling these the legacy companies. But altogether, they're making up a pretty large share of the GDP. So maybe we should include them in our analysis as opposed to just carving out a very specific swath of companies.

Jonan: Legacy means you survived; you made it. If a piece of code lasts for five years, wow, good piece of code. It kept doing its job until someone came along; it’s frustrating to go into these systems. I understand in the banking industry -- I was talking to a British banking software engineer one time, and they were telling me that in the enterprise space, they prefer to call them historical systems instead of legacy systems. So maybe they have it right that these are actually systems that have stood the test of time. And while they may be difficult to understand now, just because we're lacking the tribal knowledge that was there when they were made, they have survived because they function, and they continue to make us money. So maybe don't muck around in the legacy code too much.

Jeff: I wonder, and this is me again with half-baked ideas in a safe space, but I wonder if this idea of legacy has really come about also with our nimbleness with technology and engineering. Because to me, legacy really just means it wasn't maintained. It wasn't maintained to the point where now it's painful whereas, in today's world, you might rewrite a service every two years. It's not insane to say, “Oh, we're going to take this service, and we're just going to start it from the ground up and rewrite it.” So if we were putting that level of energy and focus into these legacy systems at the time, I wonder if they would have a different profile in the industry. But for various reasons, fears, really heavy change management processes, understaffing, there's probably a litany of reasons, but for some reason, these things were finished. So they're done and put on a shelf, but software is never finished; it's just abandoned.

Jonan: That's actually, I think, a really valuable perspective to have. [chuckles] I am shocked by the number of things that I have finished in my career because really software's never done. And part of understanding that context we were talking about earlier where you're taking on complexity all the time when you think you're avoiding complexity, you're just pushing it back under the bed much like I've cleaned my room my entire life, even as an adult now. [chuckles] And the complexity never really goes away; it's just about deciding where you're going to keep it. Where's the easiest place to keep it? Who's best suited for that pain? I'm definitely using that in the episode title: [laughter] Who is best suited to bear the brunt of that pain? This has been a really insightful conversation for me. Thank you so much for coming along, Jeff. Do you have any predictions about the next year? One of my favorite things to do on Observy is to force people into a position where they have to predict what's going to happen in the next year in tech because that's impossible. And I want you to do it so in a year I can come back and shame you for your inevitably flawed prediction. What do you think is going to happen over the next year?

Jeff: I think over the next year, there's going to be a lot of organizations that find themselves in an even deeper talent crunch as companies have adopted this work remote policy, and you're now competing with salary offers in your local market that you just can't compete with. You've got these well-funded startups from Silicon Valley offering your engineers twice your market rate in Des Moines or something like that. I think companies are going to have a real issue with talent. And the other problem is the pandemic; I believe the pandemic has changed our working patterns or, I should say, accelerated our working patterns with working remote distributed teams. But the catch is it is a lot harder to sell your company culture in a remote environment because at the end of the day, no matter how many Zoom taco parties we have, I'm still in my house. [laughs] I'm still in my office space; however, that's configured. There's just a certain level of connectivity that just can't be replicated. You always have different cultures in different teams, but this idea of this overarching culture in an organization is going to be harder to sell. Even if you execute it, it's going to be harder to sell because it’s hard to sell now. It's like, yeah, I've never had an interview where someone's like, “Oh, well, the culture kind of sucks, but the pay is great,” they never do that.

Jonan: [laughs]

Jeff: So you're taking a leap of faith, or you're asking very particularly structured questions to get into the heart of some of those culture definitions. My favorite is when you've got a company that spouts values in your interview, ask how they actually demonstrate those values through ritual. It's like, “Oh, we embrace diversity,” that's a common one we're seeing now. “Okay, how do you embrace diversity? What exactly are you doing?” “Oh, you know, we value the opinions of all employees.” “Okay, what does that look like? Is it just the open door policy?” I read a great blog post where someone was saying, “The open-door policy is really just a reframing of establishing how much power the boss has over you.” [laughs]

Jonan: Yeah, it is.

Jeff: It’s like, “Yeah, my door is always open if you want to talk about how much power I have over you. Come on in.” [laughs]

Jonan: Yeah, just come on in here, and we'll have a level ground conversation, just two peers having a chat. One of them can fire the other if they don't like the way they look at them. [laughter] That's good. The downfall of this is; generally, my guests make extraordinarily good predictions. I agree with you on all points. I'd said earlier that I intend to have you back in a year, and shame; I think friendly tease is another nicer way to say it.

Jeff: [laughs]

Jonan: I was trying to get something I could jab at you about a year from now, but that's pretty solid. I think that moving to remote work is a very difficult thing. Companies that start there end up doing far better. In my experience, if you start with a remote work environment from the very beginning, you're going to grow naturally into a place where you maintain all of those pieces. The culture that you have, even if it's different than what you would have had if you had started off in a physical workspace, you develop cultures in Slack, you develop like little tribalisms throughout the company that becomes your culture. Those companies that were remote early on have a huge advantage now, I think, in this new world. But then so many companies have been forced to make this transition that there was inevitably going to be these pieces lopped off. It's like when a company has some major event, they go public or something. People start talking about the old days, and they miss the old days. Inevitably, some of those people leave. Well, the old days are probably not coming back for software, the old days where remote work isn't the primary path.

And I look back two or three years ago and how many people I would love to say, “I told you so,” to about this argument, because I've always worked for companies that were comfortable with remote work. And I was like, look; you are missing out. You can hire the best developers in the world in the middle of nowhere for 20% more than what they're making where they are right now, which is still far less than you hire someone down the street in San Francisco for. You've made a decision to scope all of the employees in your company within a 30-mile radius of this building for no reason. It just doesn't make any sense. I will find you people who stopped by my booth to hear me rant at conferences, and I will tell you, “I told you so.” I absolutely agree with you. If someone was starting out in their career today aiming to be where you are, what would you tell that person? What advice do you have for a younger Jeff?

Jeff: I'm very happy in Ops. I really do enjoy operations, but I will be the first to admit that Ops happened to me. I did not take control of my career the way I think some people do, and deciding...for me, I had an unconventional path. I went to college after I was in the field. But my first job was in Ops, so my second job was in Ops, so my third job was in Ops. And then you wake up, and you're like, wow, I've been in Ops for 20 years. I would probably suggest that people make sure they're evaluating that they're in the area that they want to do and make the switch earlier. Because once you've built up a number of years of experience, switching becomes more painful because there's usually a pay cut associated with it because you're no longer a senior person at that level. You're coming in as an entry-level JavaScript engineer. And as you get older, you start to collect all of these obligations that make those career switches difficult, not impossible, but definitely more difficult. And definitely, you want to make sure that you're super passionate about it. So having that experimentation early in your life when you can sustain those sorts of setbacks, I would say that's probably best.

And I think for folks in the Ops landscape, Charity Majors had a blog post that talked about the future of the Ops career, and I totally agree with her on so many points. But the one point is that I think the idea of this generalist Ops person will go away because so many SaaS solutions are starting to fill that void. So becoming a specialist in one particular area is probably going to be beneficial. It's amazing how many Ops people don't really have a lot of database experience, engineers as well, like software engineers. So becoming extremely proficient in a particular type of database system could be good. I would say focus on the underlying concepts more so than a very specific stack. So if you understand relational databases, you can go to MySQL, you can go to Postgres, you can go to Oracle, you can go to any of these things, and you'll quickly realize the difference between them are just the trade-offs that they make, but they all have to solve the same problem. They all have to solve how do we write data efficiently? Well, we write to a transaction log, and then we put it in permanent storage later on. They all have to deal with buffer pools and memory pools and things like that. The implementations are really just about how they solve those very specific issues. But if you understand the problems that an rDNS has to deal with, learning another particular flavor of that becomes much easier, so same thing with NoSQL databases stuff like that.

Jonan: Excellent advice. I think particularly the part where you said always use Postgres. Postgres is the one true database.

Jeff: [laughs]

Jonan: I agree with that, especially. Thank you again for coming on the show. It's been an absolute pleasure. And I look forward very much to having you back in a year where you get to say, “I told you so,” [chuckles] to me because clearly, all the things you mentioned here are happening. Thank you for coming, Jeff.

Jeff: Thanks for having me. I had a good time.

Jonan: Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on In fact, most anything The Relicans get up to online will be on that site. You'll also find news there of FutureStack, our upcoming conference here at New Relic. We would love to have you join us. We'll see you next week. Take care.

Discussion (1)

chaelcodes profile image

This title is amazing.