The Relicans

Cover image for The Future is Bright – Open Source, OpenTelemetry and Observability with Buddy Brewer
Mandy Moore
Mandy Moore

Posted on

The Future is Bright – Open Source, OpenTelemetry and Observability with Buddy Brewer

Jonan Scheffler interviews (New Relic)(’s own Buddy Brewer about the importance of monitoring and performance and how it’s an opportunity to take a wide lens to look at all of the different ways that software gets used by people every day.

Jonan and Buddy also talk about how if we really want to make great software, we really need to focus on end-user experience, and that one of the things that is incredibly important today that we weren’t focusing on ten years ago, is the importance of interoperability between tools as toolmakers, community project managers, and open-source project maintainers. We should be oriented around how we help the people who are actually creating experiences compose tools together in ways that help people get their jobs done better.

Should you find a burning need to share your thoughts or rants about the show, please spray them at While you're going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you'd like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.

Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's Developer Relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry, and we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on We're so pleased to have you here this week. Enjoy the show.

Hello and welcome back to Observy. I'm Jonan, and I am joined by my guest today, Buddy. How are you doing, Buddy?

Buddy Brewer: I'm doing great. Thanks for having me on, Jonan.

Jonan: Thank you for coming. I have had, through the course of my lifetime, a few friends named Buddy, but it's been a while. And I realized, as I said that, that I could say that to just about anyone. How are you doing, buddy?

Buddy: [laughs]

Jonan: Do you sometimes, in conversation, perk up when someone is not at all talking to you?

Buddy: Context is everything. I got used to it at an early age, I think, to figure out based on the context, the situation, the inflection in someone's voice, which part of the word they use to filter out all of the other versions of it and to filter in the ones when people were actually talking to me because it's been my nickname since birth.

Jonan: Of course. It's interesting. I think in some contexts, it could be a rude thing like saying chief to someone. Like, yeah, you got it, chief.

Buddy: [laughs]

Jonan: Yeah, you got it, buddy. Anyway, interesting. So, Buddy, tell us a little bit about yourself, how it is that you ended up being here. Do you work at any interesting companies, for example, right now?

Buddy: Yeah. I work at a company called New Relic where we're working on observability and trying to make that a daily practice for all the engineers in the world, and that's really my background is in engineering. I started my career 20 years ago. I knew two things that I wanted to do coming out of college. I knew I wanted to work at a startup, and I knew that I wanted to be a software engineer, which is what I had gone to school for. And so the job was available in North Carolina, in Durham. Actually, just right up the street from where Red Hat was at the time was at a startup that happened to be doing front-end performance monitoring. They had actually just pivoted from a different thing that they were trying to do, but it's a long story. But they ended up getting into the monitoring business, and fast forward 20 years later, and that's what I'm doing here.

It's the same thing although things have changed in the 20 years. Things have gotten more complicated in a lot of ways, probably simpler in some others but definitely a whole lot different than at that time. But everything I've done since then really set the course. I've done engineering and product management, and I started a small company, and I worked in and around sales. I'm currently actually in our corp strategy group here at New Relic. But the common thread in all of that has been I'm an engineer by trade, and it's all been around monitoring. And of course, now we've expanded and shifted the aperture a little bit to the notion of observability.

Jonan: This is very interesting to me. So you walked out of college. You went to school in Durham. You walked out having graduated and knew only that you wanted to join a startup. But then you joined a startup that happens to be just post-pivot in this space, and that turned into a 20-year long career.

Buddy: Yeah, it's a fun thing to do. I recommend it. [laughs] It's a lot of fun.

Jonan: Monitoring seems to be a popular choice these days and getting more popular. I spoke to someone the other day who had made a change in their career to focus more on the observability space intentionally because they felt like this was a new frontier for the industry. The things that are being unlocked now are making it a pretty dynamic space to be in. So they had shifted over from doing a lot of infrastructure work. So you went off to this startup. You mentioned you started a company. Was that directly afterwards?

Buddy: It was after the company that I worked for was acquired and then acquired again. And in the course of doing that, I moved from North Carolina to Boston for a while and then out to the Bay Area, California, where I am now. I had an opportunity to, and especially being out with the heritage that Silicon Valley has around entrepreneurism and all of that, an opportunity to try starting a company with a colleague who had created actually the first open-source library for doing real user monitoring. And so he and I started a company together that was focused on real user monitoring because you're going back to the attractiveness of the space for engineers out there thinking about career paths or really anyone thinking about career and technology.

One of the things I really like about monitoring and observability is that while there is a lot of deep, technical stuff to sink your teeth into, especially with the complex software architectures and everything these days, and all the facets that all need to be observed, the other thing is that ultimately, all that software the whole reason all that software exists is to create a user experience for someone who's just trying to do something on their computer or on their phone or on their tablet, like buy a gift online or share a picture with family or something like that. And so, working at a monitoring company, we work with companies who create those experiences for their users. It really, I think at least for me personally, has afforded me an opportunity to take a pretty wide lens at all of the different ways that software gets used by people every day. And that's as interesting to me now at this point in my career as all of the technical details, which also continue to be interesting.

But back to the startup thing, that was why we wanted to do real user monitoring because you had this opportunity to look at what's happening in the software. And at this time, (Gosh, time flies. This was about a decade ago.) browsers were just starting to provide APIs that would let you actually ask them how things were going and not just in certain elements and do all kinds of things to render content, but actually observe what's happening inside the browser. Thanks to this web performance working group that was spun up in the World Wide Web Consortium and thanks to the browser vendors like Microsoft, and Chrome, and Mozilla who implemented these APIs. And so you had an opportunity to inspect the browser, see how long people are waiting on things.

The way you used to do front-end monitoring is you would simulate user experiences using synthetic monitoring, which people still do today; it’s still useful. But that was the only way for a long time. And now that browsers were sophisticated enough where you could actually observe at a technical level what was happening, you could also see how that was actually affecting all of those human beings who are trying to use all that software. How long are people prepared to wait before they're so infuriated that they just close the browser tab and go off and do something else? And so, the ability to try and understand that relationship between technology and people was fascinating to me. And that was what really led us to looking at that piece specifically.

When you look, there's so much stuff in observability that you have to understand about what's happening in your infrastructure, what's happening at the network layer, what's happening in the application layer, what's going on in the cloud, all these other things. We just got really laser-focused on that aspect of the connection between how long people are waiting, and how it was impacting people, and what companies could do to fix that. And now I have the opportunity to do all sorts of stuff beyond that today.

Jonan: This is exactly the lever you want to pull as a business owner to increase revenue. You know that people are getting frustrated and walking away from the experience. One, you hopefully care about your customers, but there's also this whole money thing. As you were mentioning that, I was remembering this morning walking down (I'm in Vegas right now for a conference.) to the Starbucks this morning to get a coffee and seeing a line of 40 people and leaving right away. The experience of waiting around there is zero-tolerance for me when I've yet to have my coffee. But on a website, imagine we're going to buy a gift for someone, as you were saying, or we're going to share something with a friend, and the link just hesitates for a moment. I'm onto the next product. I'm going to try another way to achieve that thing. I'm used to this instant gratification world.

So you came into it from the technical side, and now you're much more focused on these larger perspectives around people using products and getting them the things that they need. When you were talking about your transition from that company, that company was acquired as well, right? The RUM company.

Buddy: Yeah. We sold it. We actually sold it to a company and then joined the leadership team at that company and then sold it to an even bigger fish a few years later. There are just so many different products and tools, and companies, and projects, and open-source communities all oriented around helping people reason about what's going on in their software, which is great. Of course, it also creates its own set of complexities.

Jonan: It is a very complex world today, much more complex than it was in 2010 and even more than ten years before that. We didn't have the APIs at the time, as you were saying, to really monitor people in real-time. I saw a product a couple of years ago that someone was demoing at a conference that would actually record and replay the DOM events and recreate what the customer was actually saying. So it was as though you're doing a customer session, and you're watching them actually use the site. You can see their cursor move. You can see the button take too long to go to the next page, backout, refresh, keep clicking that checkout over and over again, very interesting technology. And gaining that level of insight and then tying it into all of these other pieces like we do with these massive platforms like New Relic, where you have all of these different facets of your system health running into the same place and trying to draw these lines through them and connect the dots, is a very, very powerful thing. We've never had the ability to get that level of observability before, but it's also complex to get started with.

If you were to start out again and look at observability today, do you think that you would have headed in the same direction, like with that company? Do you think that that piece with the people and their user experience real-time and then helping people understand what the users are going through right there on the site, do you think you would do the same thing? If you were going to make another startup today, where would you head?

Buddy: I think yes and no. I think that the enduring truth is that there are humans at the end of all of this software. And if we really want to make great software, then we really need to focus on that end-user experience. It's funny, in the industry, as someone who spent 20 years on the vendor side of the equation building tools for people, there's this phrase, it's a cliché that everybody always talks about like wanting to have a unified view, a unified view of what's happening across the stack. And it's only gotten more complex with observability and trying to understand what's happening at the Kubernetes layer and at the service layer, all this stuff.

But you know who's had a unified view since day one, who's always had a unified view, who can't have anything other than a unified view? It's the end-user. The end-user always has a unified view. That unified view, the composition, and the confluence of all of those different parts of the technology stack and everything are all in service of creating that one experience that the customer looks at in a holistic way. And that is where the truth lies of whether you're doing a good job of delivering software or not. Now the really hard part is if the user's not having a great experience in that unified view, how do you unpack that? Which is where all of the complexity comes from.

And I think like in terms of if you were to look back and do what we did ten years ago but do it in 2021, the aspect of always prioritizing, understanding the end-user experience, and making sure that we're all focused on the ultimate goal of helping people create those experiences I think would still be true today. And I think it'll still be true ten years from now. One of the things that I think is incredibly important today that we didn't, and I don't think a whole lot of people were focusing on ten years ago, is the importance of interoperability between all those tools.

I think like here at New Relic; I remind myself and others around me often that I have the luxury of coming into work every day. And I show up at work, and from the moment I start working until the moment I end working, 100% of my professional cycles are spent thinking about New Relic where I work. But we're just a tiny piece of the day of all of the people who use our tool because they've got a lot of other things on their mind besides observability and monitoring for one thing. But within the observability space, there are a lot of adjacencies there and a lot of different tools that people use in order to get their job done. And a lot of that is even more driven by the complexity that we've seen with the way that software architectures have evolved over the past handful of years.

And the importance of being interoperable and making it easy for people not just to pick up your tool but to be able to use it and compose it with all of the other things that people do to get their job done I think is more important than it's ever been. If we really are all about that end goal of creating better software and creating better user experiences for those end users, then as vendors, and as toolmakers, or community projects, open-source projects, I think we should all be oriented around how do we help the people who are actually creating those experiences compose those tools together in ways that help them get their jobs done better?

Jonan: So with regard to user experience and interoperability, I have maybe a counterexample, maybe not. But this company, Apple, I think you've probably heard of them. They have a walled garden, and they enforce those walls well. They're really, really good at keeping people inside. And they're also really good at user experience and maintaining that focus on the user. I don't know what's going on with Apple's open-source very well. I imagine that they're in open-source technologies as much as anyone else. And they've certainly contributed a lot of software back to the world. But there is something to be said for having a very tightly controlled user experience. Do you think that this interoperability discussion flies in the face of that? Do you think that they are at odds?

Buddy: I would submit to you that even Apple has an operating system that's based on top of…what is it? BSD, I think.

Jonan: Yeah.

Buddy: And I'm not saying that Apple's rocket-ship growth has been solely on the basis of their operating system kernel choice. But I would note that the launch of OS 10 and all of the product ecosystem and everything that surrounded that I think correlates in time to an inflection point that happened in Apple's trajectory as a company. And so even a company like that, like you said, who's probably the standard-bearer for how to get walled gardens and all of that kind of stuff right at least in terms of the business results, and they do, they have a remarkable user experience. I'm talking to you on an Apple product right now.

But even Apple, at their core, the operating system is based on open source. And the Chrome web browser is also based on WebKit. That came from the K Desktop Environment. It was the KHTML component; I think that was at the genesis. The conqueror web browser was the proto Safari, which is now, if you factor in tablets and phones, the most popular...well, Safari and Chrome, but certainly, on mobile, Safari is hugely popular as a web browser also owing to open source. And the way that a lot of these components connect together are done using open standards. I think it's something that you just can't get away from today. And it's one of the reasons why technologies closer to the observability space are so important. Like, OpenTelemetry is a mechanism for data interchange.

Jonan: Yes, finally. As long as we have had open-source in these systems, we haven't had that piece as long. The protocols that people have been creating have been proprietary for so long. New Relic's original method of reporting from the agents up to the mother ship was our own little device that we put together. And now we have OpenTelemetry to solve that problem and across all of these other standards, even like the web APIs. I think that's based more on the users again. The users demand that, and, in this case, the users for B2B being those buying and using our products. Why would I, if I'm looking for an observability solution, consider your platform knowing that I'm effectively trapped and other tools I'm going to have to build handcrafted adapters with? I think that's what's driven the change in our industry, and I see it across the board.

But it does, in somewhat of a permanent way, break down those walled gardens. For example, you have multi-substrate infrastructure. Everyone wants multi-substrate. They want to be able to swap out AWS for their GCP and then back again or whatever new cloud. They don't know what's coming next, but that has commoditized the cloud infrastructure space. Do you think that we are headed towards commoditization as well? Do you think that observability is just going to be a race to the bottom on pricing that you can pick it up anywhere, and all the pieces are swappable? What prevents that?

Buddy: I think it shifts where you have to focus on differentiation as a company. And what I mean by that is observability, or like in the historical context here, we'll just call it the monitoring companies. When you look at the evolution, there are some massive changes that have been happening for a long time but have now come to rest in this particular part of the software space, so the first one is the emergence of open source as the default way of constructing software.

It started with the operating system with Linux. That played out in a very competitive way about 20 years ago between the incumbent at the time, Microsoft, who was very anti-open source 20 years ago and is a completely different posture 20 years later today, which I think tells you something. But spread from the operating system kernel through the entire developer toolchain, through web servers like Apache, and then Nginx. And then language interpreters for the web like PHP, or even just any programming language just in general like Java which I think started out as a proprietary licensed language from Sun, and then was open-sourced and has just swept through.

But for many of those years, as that was playing out, the companies that produce tools to actually monitor software who primarily target…they're all B2B companies. They sell to companies who, if someone's going to spend money to make sure their software works, they must also be making money. So there's plenty of revenue that's washing around here. There were largely untouched as a business by this open-source trend until relatively recently in the scheme of things where you've had technologies like OpenTelemetry, and Zipkin, and Jaeger, and all this stuff.

Jonan: The CNCF ecosystem.

Buddy: The whole CNCF ecosystem that's enabled all of this to be done in an open-source way. So there are open source alternatives to the commercial providers. And, at the same time, the other thing that's happened is largely, the former has a lot to do with the latter. But it's this inversion of the buying process from being a sales-led process to a buyer-led process. And so not that long ago, selling software to another company meant putting somebody on an airplane, flying them to a conference room somewhere before you showed them the software, having a discovery call, asking people a bunch of questions. Then before you show them the software, show them a bunch of slides with pictures of the software. And only after all of that and then writing down success criteria, what would it take to get you in this car? Type of stuff would you then set about to do a proof of concept and helping people set up the software and walking them through this whole process. And people don't do that anymore.

Jonan: I don't.

Buddy: And you don't have to. What they want to do is discover the software online, learn everything that you want to learn about it. I'll get the stat wrong, but there have been studies that say that people are 80% done with their decision by the time they even reach out to a person anymore because the default is that there's so much information that's out there. And so this inversion of the buying process combined with the democratization of the underlying technologies, thanks in large part to the work of the CNCF and other open-source communities has taken all of that table stakes stuff out and actually commoditized it.

And so what's left is for companies like New Relic to then differentiate on top of that in ways that actually move the state of the whole practice forward thanks to all of that pressure, which ultimately creates better set of tools for those people who are creating those experiences. And then, like we were talking about earlier, ultimately better experiences for the humans that are on the end of the phone or the laptop trying to use software to do something.

Jonan: I think to take your example from earlier, OS 10 didn't succeed by taking BSD and being like, yep, this is good. It's finished. We'll just stick to what's here. There's open-source underneath and a lot of room to improve the experience on top. So this conversation has been really interesting. I find the bit about the sales piece particularly insightful because I know that companies at an enterprise-level stuck to that model for a long time. But as an individual consumer, I can honestly say contact sales is my least favorite button on any website. Just show me the doc, show me how to use the thing. Let me play with it. I don't want to go and talk to a salesperson and have a long meeting, webinar demo before I get going. I always get frustrated when I get forced into that conversation.

Buddy: That has an impact on salespeople, too, right?

Jonan: Right.

Buddy: Because what it does is it paints them with a brush of somebody you don't want to talk to. [laughs]

Jonan: Yeah, right?

Buddy: Many of whom are very thoughtful, very smart, want customers to succeed, and all that. And so one of the great things that I think has happened with this inversion of the process is the evolution of that role from the traditional caricature of the glad-handing salesperson type of thing to someone who is more of a partner to help you through that process of understanding, which is increasingly product-led but where you can actually consult and help people find the pieces that they already have access to but might not have time to sort all this stuff out themselves. And so they can accelerate them through that process. So I think it just creates a much better relationship.

Jonan: It happens often that people working in DevRel...I have a friend who just last week actually reached out and wanted an introduction to a salesperson specifically to get the state of things here earlier, but it's a very different process; you’re right. He's already invested the time in going and learning about the product and understanding the documentation and has a reasonable expectation that this is the right choice. And he's looking for the icing and a partner to make sure that it's successful when they start trying it out, so yeah, a very different world today. And it will probably continue to get different every single year. We can expect change to be the one true constant. So what do you think that looks like? As a tradition on Observy, we ask people to make a prediction over the next year, so we could have them back a year from now and then mock them publicly for having been wrong.

Buddy: [laughs]

Jonan: What do you think is coming...maybe a year is too fast. But on whatever timeline you would like, what do you expect is going to be happening in our industry?

Buddy: I think OpenTelemetry is going to take over the world of software interoperability for metrics and traces and all the data around observability because of all the things that we talked about that the users of this software will demand it. OpenTelemetry itself is a standard that's on its own maturity curve, and it isn't complete, and it is a work in progress. But as it continues to progress, we'll see an increased demand from people who run software systems for everyone, whether you're building tools that are point solutions, your big incumbent platform, observability companies like New Relic, where I work, or other open-source projects. There will never be a consolidation where someone can run their software using just one tool. So there will always be a group of things that have to work together. And the projects, and the companies, and the products that win because of the fact that those customers are now in the driver's seat will be the ones who embrace the interoperability and embrace the open standards that create that.

Jonan: This is a solid prediction. I'm going to have a hard time picking at that, I think, a year from now. One of the other questions we ask is about what you might have done differently in your career? We talked a lot about how you got started here. And we actually touched on this a little bit. But what would that startup be today if you were to go do the same thing? But what advice would you have for someone who may be listening today and thinking, yeah, I want to be Buddy Brewer when I grow up and get out there into the industry? What advice would you have for yourself or them starting out?

Buddy: I would start off by saying don't want to be Buddy Brewer when you grow up. Want to be yourself when you grow up, seriously. Because a lot of bad decisions happen when you want to try and copy somebody else's path.

Jonan: Absolutely.

Buddy: One of the things that I come back to a lot personally, I call it the idea of having a compass needle. I like to check in with myself periodically and say, what do I want my next job to be? But ultimately, it's what do I want my last job to be? I think that's really the most important question. And it's one that a lot of people are afraid to ask themselves, and so they dodge it, and they don't put enough thought into it. And always having that idea of when you're thinking about developing your own career, if you're trying to decide is what I'm doing this month or this year, or this opportunity that I'm being offered, is this the right decision to make? Ask yourself, does it move you in the direction of what that ultimate thing is that you want to go to?

And I think one of the reasons people hold back from that is because they feel like if they say it out loud, they're making some kind of commitment that somebody is going to check-in, and if they change their mind later, they're going to say, "Ah, gotcha. You flip-flopped on what you said you were going to…" you know, whatever. But you have to reserve the right to learn as you progress and as you mature as a human being. But that doesn't mean that you don't have the opportunity to articulate and set that target because I think it makes a lot of other decisions easier.

And the other thing that I would say that's at least in my head, is related is having an open mind and not a preconceived notion about what that trajectory needs to be. Because personally, I've had jobs in engineering and engineering management, in product management, and in sales for a little while. I've gone back and forth between IC and management roles. But the common thread is that I've learned an incredible amount of stuff about how software businesses run along the way, which is my ultimate objective is I want to learn everything about how software businesses work. And so, having that open mind and being willing to take an opportunity even if it doesn't look like on paper what the paper trajectory is supposed to be. If you're still learning a lot, then it's worth at least trying it out for a while, and then come back and reevaluate six months or a year later and see if it's still directionally on that path that that compass needle is pointing in.

If you look at any of the most successful people in the world, I think I struggle to think of a single person who's had a linear path through their career. And I think they're all the better for the twists and turns along the way. So instead of trying to avoid those, embrace them, and that's where that compass needle is so important is as long as over time you're moving in the direction you want to move, then you don't have to worry so much about the twists and turns along the way.

Jonan: That's very smart advice. I think you nailed it early on in that when you were talking about people needing to get used to the idea that they can change. And they can learn that in making a decision or not committed to it, no one's going to show up and declare you a liar except for you. The trick, of course, is that we want to be right as human beings. We want to have the answer right on the first try. So it is an important lesson for all of us. And I want to thank you very much for coming on the show today. Is there anything you want to leave our guests with, any parting thoughts?

Buddy: No. Just to say thanks so much for having me, and looking forward to checking in with you on that prediction.

Jonan: A year from now. We'll see you again then. Thanks again, Buddy. Have a wonderful day.

Buddy: Thanks, you too.

Jonan: Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on In fact, most anything The Relicans get up to online will be on that site. We'll see you next week. Take care.

Discussion (1)

artsboy profile image
Paul Wright

Brilliant insight and really fun to read. Thanks 👍