Jonan Scheffler interviews Grafana Senior Cloud Integrations Engineer Jacob Plicque about cofounding The Empatheatre: a Twitch stream for live tabletop RPG playing that incorporates a variety of safety tools to keep players psychologically safe and comfortable, why failure isn’t just okay; it’s actually awesome, comfortable disagreement, and the recent explosion in the popularity of developer relations.
Should you find a burning need to share your thoughts or rants about the show, please spray them at email@example.com. While you're going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you'd like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.
Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's Developer Relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry, and we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on developer.newrelic.com/podcasts. We're so pleased to have you here this week. Enjoy the show.
Jacob Plicque: Doing wonderful. How are you?
Jonan: I am doing much better. I'm about to hit my two-week anniversary of having received a second shot. So this is the first time --
Jonan: Right? That's a big cheer right there.
Jacob: Yeah. [laughter] I'm so proud that you have that noise ready cued up.
Jonan: I think that's the first time I've gotten to use my soundboard on the podcast. That's perfect timing.
Jacob: Oh, I'm happy to have been here for the beginning of what I'm sure will be many, many times.
Jonan: Many, many soundboards. This has been a really long decade of pandemic for me.
Jacob: You know? [laughter] Preach. For me, I hit my two-week anniversary about two or so weeks ago. I kid you not, I was like, okay, cool. As a mega extrovert, I have to go out and see people. If you saw me in Downtown Jacksonville, my hands were up, and I'm praising the sun like I’m playing Dark Souls.
Jacob: Because I was just like, people and stuff, yay! Loudness.
Jonan: [laughs] It's amazing, isn't it? I still have this feeling, like, what do I do? What is normal, again?
Jacob: Yeah, there is none.
Jonan: There’s none.
Jacob: There never really was, I think. [chuckles]
Jonan: There really wasn't. At conferences, I would get up on the stage, and I'm not sure what the topic was that I was talking about, but I used to do a lot of talks about identity and brain science and things like that. And so I would do this thing where I'd be like, “All right, if I can get just by a show of hands who would consider themselves to fall in the spectrum of normal.” And [chuckles] out of 300 people, there's one person who raises their hand, and I'm like, “Well, congratulations to you, sir.” But I want to talk about what we mean when we say normal.
Jonan: We're talking about an average, and my point is that none of us are. We all feel a little bit like outsiders in one way or another. So that actually brings me a little bit to our first topic because we were discussing this Empatheatre thing. Why don't you tell us all what Empatheatre is?
Jacob: Backing up, I'm very obsessed with Dungeons & Dragons. I love that aspect of real-time live storytelling. In some cases, there are Dungeons & Dragons and political intrigue or sometimes just absolute ridiculousness. And I am someone, again, that is very in need of connection. And being an extrovert, a lot of these games that I was playing in my house no longer were happening anymore because of the pandemic. And also, I was trying to get my feet wet into the online TTRPG, tabletop RPG community as well, so these things started to begin to dovetail. On Twitter, I started to follow people and see the whos and the whats, and eventually, I was asked to join a few for charity, and those went really well. And I was asked to play in a D&D campaign for this particular Twitch channel. It doesn't exist anymore. It was called Thread Raiders. But that campaign ended up going for about seven or eight months, and all the players including myself started to become really, really good friends. And it was like, “Well, this particular Twitch channel is being sunset. What do we do?” And I raised my hand, and I was just like, “Why don't we just start our own, you guys?” Then it's a matter of we're not having to worry about other people's schedules; we’re building it. And you can imagine five people that just met less than a year ago starting a company because there's obviously back-end stuff that we needed to create in regards to starting a Twitch channel, especially if we're taking in things like Twitch subscriptions and stuff. So there was a monetary aspect to that. So there was some initial hesitation, but once I started to think about what we are about, and what makes us connected, and that's the combination of being, broadly, bluntly honest and transparent as well as having that underlying basis of empathy and having a commitment to we can change how people look at our world by telling different, engaging, unique stories. And so it just so happens to be tabletop RPGs or D&D that we're utilizing from a storytelling aspect. So that's how it got started.
Jonan: So I promise I had a proper segue here, which was in the context of this feeling of being other. We had talked a little bit because you'd mentioned that this was a gaming thing. And also, the name Empatheatre implies to me it's about empathy and theater. I was like, that's a lot of things for a thing to be about. But you talked about this policy that you had in this group where you have like a yellow card thing. When someone is in the middle of this storytelling -- I don't know if our audience have all played D&D, but it's a pretty immersive, creative experience. And in order to be creative, people need psychological safety; it is paramount.
Jonan: And so being in that situation, if you encounter something that makes you personally uncomfortable, it's really important to be with a group of people who you're close enough with that you can just be like, “Yo, can we just do that fade to black next scene, please?” We just want to skip over this part a little bit. And it speaks a little bit, I think, to something that is so important in building community, which is that sense of inclusivity and making sure that people are comfortable, and it doesn't really need to be a primary focus of the community. It's a safety guide so that people can focus on the things that they're actually there to achieve, and that should be the priority for that community anyway. Like, we're all here to hang out and play games. We're all here to talk about SRE or DevOps. Let's put some guidelines in place to make sure that everyone can focus on that, and we don't have to deal with these other pieces. So explain, first of all, how exactly you handle that on Empatheatre and then maybe how you've seen that apply in your own community work.
Jacob: Yeah, sure. So there are a few safety tools that are out there, and there are some really, really detailed ones around particular pieces of content like, “Hey, this dungeon is probably going to have giant spiders,” as an example. “How do you feel about that? Green?” “I love spiders.” “Yellow?” “I can take it or leave it; maybe, maybe we don't get super-duper detailed in the venom.”
Jacob: “Red?” “No, absolutely not. Replace it with something else.” And that's fine because, at the end of the day, we're playing a game, and we're all trying to have a good time. And it's really funny to me sometimes when the idea of fun in some cases is blindsiding your friends with a plot twist and oh, that thing that was supposed to happen and you were supposed to get the gold because you killed the dragon. Well, not really. You're actually being framed for murdering the dragon because the dragon was actually like a God that people were worshiping. And so it becomes this like, oh, rip the carpet from under underneath them but in a safe way, right?
Jacob: Because if it's going to cause a problem, then then we're not having fun, and then we're not telling a fun story. And so how that segues into the community work and the work that I do at the old day job is I realized that a lot of what we do in SRE or observability or even DevRel, in general, is safe storytelling. I can't tell you the number of times I've done a talk, and there's a slide where I say, “Failure is okay.” And everyone's like, “Huh?” Not only is it okay, it's actually awesome. That's the secret because guess what? That's how we learn. We learn from those pain points. The problem is we're not really comfortable with sharing those pain points with each other. A good example of this is I was at a conference, and this was, of course, during the pandemic, so it was all on Zoom. This was a Zoom of; I don't know, 50 people. And we were going to come together, have a good time, have a beer, and tell some favorite stories. Those are 50 people. There are two people talking, me and another guy, and we're just swapping stories. And there's a few folks heads’ nodding so they could definitely sympathize or empathize with the stories that we were telling, but they weren't telling their stories. And I thought that was super interesting. And what I've come to realize was that that's because, at least from my perspective, I need to get them into a space where they feel comfortable doing that. And so I started to tell other stories, and then I would segue into “Have you had something similar happen to you in your experience?” And then that was when the door started to open. And then now there's a dialogue. And so that was a big learning point for me.
Jonan: And that safety is so necessary to be able to inspire the dialogue in the first place. It's table stakes in that whole engagement piece. If you're trying to get a conversation started with somebody, you don't open with a contentious topic. Although next on our agenda, we have politics and religion. I hope that that's okay with you if we just bring those up.
Jacob: [laughs] That’s good.
Jonan: Like, you try to get to know someone; it’s a conversation, really. That's what DevRel was about is just being a part of the conversation so you can be a better friend to people on both sides of the line inside your company and outside, and help everyone out, understand the plight of the developer, both when using your product and outside of that. So you mentioned the thing about failure being awesome. And I totally agree, but I think this is perhaps an expected perspective for someone with a chaos engineering background.
Jacob: No. [laughter] It's true. And to be honest, I don't think I did until I started working at Gremlin. But it's because, to be blunt, I’ve started to be exposed to that a lot more. Because I'm talking to teams and engineers and companies that are like, “We are failing, and we need to figure out how to get better.” And that's where that chaos engineering mindset comes to play around being more proactive or understanding; hey, these are the things we're investing our time and energy in, and they happen to be these resources called people. [laughs] And burnout is real. And a company can make millions or billions of dollars, but if no one's there to make sure that the lights are on, does it really end up mattering? That was always a big deal for me. And then on top of that, I started realizing that on top of the fact that none of us are talking to each other about all these different failures that we're all running into, we're all failing in a lot of the same ways, in a lot of the same aspects too. I remember I had a conversation with someone from an observability perspective. We're going away from we don't really care if CPU and memory is high because we have Kubernetes, and we're going to scale up and out, and we don't really give a crap about that. But at the same time, we still have companies that are like, “Yep, I got woken up at 4:00 a.m. because this disk is full and that cron job didn't do the thing that it should have.” That was a conversation that I had in 2020. You know what I mean?
Jacob: You could argue that there are different maturity levels for different engineering organizations. But at the same time, just because that person is at level 2 versus someone that's at level 4, how do you get from level 2 to level 4? That's the story, and that's what level 2 wants to know.
Jonan: That's a really good point. That was my server, by the way, the one that filled up the logs.
Jacob: [laughs] I knew it.
Jonan: I remember the first time I couldn't SSH into my server and being like, what is going on? Yeah, log rotation is a thing.
Jacob: Yeah, exactly.
Jonan: Nowadays, you're exactly right; with tools like Kubernetes, we're treating these servers as entirely disposable. It’s like, oh, it's broken? Throw it away. Build another one. Switch all the traffic over there, then figure out what it is that broke and try and prevent it from happening again because it's typically going to be something that is time-bound like that. We are heading back towards disaster. We know that the disk is failing to back up, but now we bought ourselves a little time to figure it out. And while it's a much more complex world today, I really appreciate operating in that environment. These cloud services and this whole infrastructure environment we have today allows us to spend a little bit more time focusing on the people and the processes that ideally enable a company to grow to billions of dollars and still appreciate the fact that humans are not disposable resources like literally fuel to be burned in the machine of money. That's not what this is about. So you are now working at this company Grafana Labs who also -- Is the company Grafana Labs, or is it Grafana?
Jacob: Grafana Labs is the company. Grafana is the tool.
Jonan: Grafana is the tool. And I think that Grafana has exploded in popularity as part of the CNCF ecosystem, which is one about a lot of this technical complexity that has sprung up and how we're managing that with various levels of abstraction but also about this general DevOps movement, I think, and the focus on people and how they play into that. Because ultimately, you had said we are all failing all of the time and the natural inclination of human beings -- I was talking to my kids the other day I was like, “Look, every human thinks the same thing.” When something goes wrong, it’s like, well, I tried to make this computer program work, and it didn't; therefore, I'm bad. Or I did this thing; I’m bad. It's all internalized. We internalize all of these failures. And for that reason and many others, it's important to weed them out, and control them, and expect them, and prepare for them with the approach of chaos engineering so that we can start to value ourselves as people a little bit. And in the context then of Grafana and how that fits into the broader ecosystem, I understand that they are in the process, and you're going to be doing some of this work of building out a lot of different integrations with various platforms and trying to build that web that can enable that. What sorts of things -- I'm sure you can't talk very much about what's on the table but maybe give us a hint about where Grafana is headed.
Jacob: [laughs] Yeah, sure. It's a good question. And it's really zoned in on our Grafana cloud, and so that's the Grafana Labs. Think about it from a self-service perspective but also the idea that there are a lot of baked-in mechanisms that we've -- I think both at Grafana and outside in the community, we understand these technologies and how to look at them. How do we observe them? Because what happens typically is you build something, you monitor it; then it breaks, then oh, I missed that. And then it breaks again; oh, I missed that too. So the things that I'm thinking about right now are what are the level 1 and level 2 of these different technologies, and how should someone be observing them from a best practice perspective? And then I want to shoot it out into the community and say, “All right, this is what I think. Disagree with me,” and not from an antagonistic perspective, but my bet is that that's going to create the dialogue.
So I just touched on earlier around making folks more comfortable with joining the dialogue. And in my mind, the only way for you to disagree with what I'm saying in this particular aspect -- We can all disagree with many things. We were joking about politics and religion earlier. But the idea is this is what I believe, and I'm going to put myself out there in a way that's pretty comfortable with most anything. So I want you to feel comfortable with disagreeing with me and then oh, okay, so this should be in a different space, okay, cool. Let's do that. And the cool thing about working at Grafana Labs is I can say, “Cool. This is all stuff that you can do as part of this Prometheus mixin that I've built, so send me a PR, and let's talk about it, that kind of thing. So that's the bet. And then that's going to dovetail into other content and evangelizing with open source communities. And that's just scratching the surface, really, but that's the beginning of it.
Jonan: I'm excited to see how you weave the tabletop RPGs into these conversations. I want some Grafana dashboards live of you playing D&D with your stats. It's perfect, right? There are so many numbers.
Jacob: It's terrifyingly perfect, yeah. [laughs] But my last week at Gremlin, I did a talk at All Day DevOps. If I remember correctly, it was called What Dungeons and Dragons Has Taught Me About Chaos and Resiliency. It's one of those things where I was like, how are these two things together? And that was the pitch. That doesn't make any sense. But the storytelling aspect was just so close-knit. There have been many debates about what observability is like; it’s the three pillars: metrics, logs, and traces. No, it's not. That's not good enough. It's true, observability is X; it's all storytelling, and that's really what it zones down into. The nuts and bolts is this is what I know and then what I don't know, we can tie into your unknowns and stuff I go and I add. Now then, of course, my dashboard looks terrible, and I don't know what I'm doing anymore.
Jacob: But yeah, that's 100% on my to-do list. [laughs] It's not necessarily a sequel to that talk, but looking at it from the other side of things from an observability perspective is definitely on my list. And then anytime I could talk about D&D is just a dream.
Jonan: This Chaos Quest game that Gremlin had I think Tammy worked on over there. Have you seen this Chaos Quest? Are you familiar?
Jacob: Was that the Minecraft one?
Jonan: It’s similar. It's an old pixel art style top-down RPG view, a Dragon Warrior or Final Fantasy back in the day.
Jonan: And it's played in the browser. It was a fork from BrowserQuest, which I guess was a Mozilla project where they just open-sourced all of these assets, which is great because I use some of them for my stream overlay. You can see a little soldier icon in the corner, a little knight, that's totally from the BrowserQuest assets.
Jacob: That's awesome.
Jonan: They have this game, and it seems such a perfect opportunity to instrument and play around with those things. What they've done with Chaos Quest is they go and engineer situations where they break the server while everyone's playing live and your character lags out, or you're overwhelmed with bats because you can't move, observability, and they all just eat you. You can't really see the security -- That would be a cool application to go and instrument with things like Grafana. In fact, we have plans on our team. We've done it before with Minecraft, where we instrument Minecraft with New Relic, and we were playing around on stream. But I want to get on a stream with you and play Chaos Quest, or BrowserQuest probably will fork the original one and then just go to town on it with some observability. That would be super fun.
Jacob: Oh man. I'd be super game. Yeah, absolutely. And this goes back to like the tool, I think, in a lot of cases and obviously, we work at two companies where the tool is the tool. And it's we're trying to not democratize, but we want people to use it, and obviously, your company needs to make money. But I think we're both making the right bet around it's focusing on the outside of that, and then oh, the answer just so happens to be New Relic, Grafana Labs, Prometheus, Datadog, Dynatrace, Honeycomb, all this other stuff. And that's one of the things that I think has been most interesting over the last six months or so of my work life is after being really focused on Gremlin for almost two and a half years, it’s like, let’s focus on the story first. And then the end game is, oh, by the way, did you know that that was Grafana, Gremlin, et cetera?
Jonan: Yeah, exactly. And the interesting thing about what's happening right now you mentioned Prometheus. This polling approach that Prometheus has taken for data -- So for those of you who are unfamiliar, rather than what a lot of systems do when you're trying to instrument for observability purposes, is they'll push the data out of the individual application. So I, as an application developer, go in and add instrumentation code, and I push the data out to a mothership somewhere, and then it all shows up on some graphs. What Prometheus has done is flipped that, which is a much more light way to handle the collection of these metrics from your applications. And Prometheus will just go around to each of your applications and say, “Hey, tell me what happened on a /metrics endpoint.” And so then all of the various application frameworks, you know, you've got them for Rails and Flask and on and on, and on. You go to the metrics endpoint, and it gives you all of the data, and then that ends up in Prometheus, which can be displayed in Grafana or most every other platform that exists today, including New Relic. And so what it's done is it's -- You were talking about the democratization. It's opened up the ecosystem to a point where we're all adding unique value. All of these companies exist for a reason. We all have customers. We're all adding unique value by some token as much as we would like to pretend each of us that well, no, we’re the one and true.
Jacob: We’re the only game in town.
Jonan: [laughs] Yeah.
Jacob: We’re the best of the best, number one.
Jonan: Everyone else is wrong, and we're right.
Jacob: [laughs] Yeah, exactly.
Jonan: That’s not likely the truth of the matter. The truth is that everyone has their own unique strengths, and everyone has different values to bring, and the growth and success of Grafana and Prometheus and the CNCF ecosystem only benefits every one of us. We need to start, I think, as an industry, focusing on the people again in the way that we began this conversation that if you keep your focus on the people and adding value to the users' lives, it actually changes a lot out of the conversation because when we're doing the work we do, it's so easy to get caught up in the details and focus on “There were technically interesting problems to solve. Let's go and make some really cool graphs. And look at this new statistic thing that we're tracking.” Okay, but humans are using that. Are you adding value to their lives? Are you making their lives better? Because that's what's going to make the software better in the end.
Jacob: Yeah. I definitely raised some eyebrows -- I've only been at Grafana Labs...this is week seven or eight, something like that. And even during the interview process, I very pointedly said, “I don't care about dashboards. What I care about is, is this going to wake somebody up when it shouldn't?” I remember five or six years ago; I was so proud that a multi-billion dollar company like Fanatics was relying on 26 whatever year old Jacob Plicque wet behind the ears to save the day. This ties back around psychological safety too. That thought process of what I know now is incredibly dangerous. I was so proud of it. And then it's like, oh man, why am I tired all the time? Why am I burnt out? Now it's obvious. We joke about this being full-on and stuff like that. It’s like, should that have woken me up? Let's dive into that. Because it woke me up once, it probably will do it again. I could go on around alert fatigue alone. But I'll never forget I got woken up one time 3:00 in the morning. And I was like, wow, what does this mean? And, of course, when I was waking people up, I didn't have the PagerDuty option. I literally had to pick up the phone. I'm that old guy now.
Jonan: You actually got a page on a pager you carried on your belt, right?
Jacob: Oh yeah. And I was like, what is this? And then I go, and I look, and this is an alert from, no joke, three years ago from a system that was decommissioned that no one deleted, the rage that I felt. [laughter] And we're all over the place, but in a good way, I think. And this all ties back to the aspect I think of what process failed here three years ago? Did we not scrape and go, “Hey, yo, no one cares about this thing. The queue team deleted this guy a long time ago. Why is this still here?” But 2016 me was just like, I’m going to find this person, and I’m going to confront them [laughter]
Jonan: My new job is choking them, whoever they are.
Jacob: Right. It's actually in the job description. [laughs], So it went from being this badge of pride to why? The classic three whys: why? answer, why? answer, why? answer, which is an oversimplification. But I think it's still a fair one, and I think that's why we’re still figuring it out, and we're still learning. But hopefully, the tidal wave is coming together for everybody.
Jonan: I remember that exact feeling the first time that I was responsible for being paged for an application. I felt so trusted and valued. And the reality is that people pretty quickly get over the pager thing. Still, people were excited to be trusted with that level of responsibility early on in their careers. And about a year of being woken up in the middle of the night for a service that has been sunset, it gets real old real fast. And the industry, in general, has relied on the churn that exists to allow us to consume human capital that way, which is super unhealthy. And I can see the end of it from here. I know that that piece of what we do in tech is never really going to go away; treating people as resources in general in corporations tends to be how things go. But I'm hopeful for the future.
Jacob: Me too.
Jonan: I'm especially hopeful right now. So if you were going to make a prediction for the future, what in the next year or so do you expect we're going to see as far as trends in the observability space or even specifically over Grafana. What is the specific and exact product roadmap for the next year that we can expect, wherever you want to go with it, something we can talk about a year from now and be like yeah, we totally nailed it?
Jacob: Gosh, that's a really good question. The bet in both from what I'm working on internally at Grafana Labs is that we're making...and every tool says it, “We make it easier.” But I think the fact that our base is very open-source and empathy-heavy is I think we're creating content, and we're making things easier. Because one of the things that was a big deal about the high-level messaging at Gremlin was we're trying to build a more reliable internet, which sounds so hokey, and awful, and ridiculous. It's a hell of a bet to say. But the idea behind it was if we make this company more reliable like JPMC, then you can access your money, and the blast radius becomes so wide. So I look at that and then I look at what we're doing at Grafana Labs. And the bet isn't necessarily more reliability, but if we can help people respond and learn faster, then I think we win. Now, what that metric is, that's a little bit more difficult because math is hard. My bet is I'm having conversations like this with you around all the things that I've learned based off of other people's experience, which actually feels like I'm stealing, but it's because I've helped them get to that point, and they're giving it back to me, and that's the thing for me personally. And then how that ties into an overarching Grafana Labs perspective is I've probably built 50 integrations that have helped move the needle for a bunch of different companies, whether they're Grafana Labs customers or not. And that's my bet.
Jonan: I absolutely agree, and I hope the trend continues. I think people are slowly starting to realize that adding value to these ecosystems generally adds value to each of us and all of us and the communities that we support and embrace at the same time. I think the approach for a long time has been much more me-centric from a corporate perspective -- then is going to be successful today. We exist in large cooperative ecosystems, in many cases, open-source ecosystems where you need to participate or be left on the side of the road.
Jacob: Yeah. And I think you're right because there was a lot of momentum with us versus them mentality. To be blunt, many folks were very successful, but then okay, then what happens is they plateau. And then it's like, there’s someone from Honeycomb, and from New Relic, and from Datadog. I'm sure it's happened before, but from my perspective, it was like, this is the first time I've seen this, and it wasn't like we're all fighting each other. Everyone's disagreeing with each other, but also, we're all going the same way because that's what it comes back to is we're all trying to do the same freaking thing. And so that I think is going to continue. I'm very curious to see how that will begin to resonate at the highest of levels from a business perspective. That's easy from an open-source contribution perspective. Look at OpenTelemetry. Good Lord that has exploded, and good, it should. It's awesome. I want to make that easier because, personally, I had a hell of a time with onboarding with OpenTelemetry. But that's because I took, say, cosplaying as an SRE, cosplaying as a solutions architect, [laughs] cosplaying as an integration engineer now, cosplaying as a DevRel. [laughs] So it's imposter syndrome, ding. So I think that's going to continue as long as we continue to have the conversations, which is really what needs to happen and continue to.
Jonan: I think that the recent explosion in the popularity of developer relations, I mean, I have seen significantly amplified demand over the period of the pandemic. And I think that part of that is linked now to this understanding that the company values do ultimately need to align with developer values and what developer values are, open ecosystems, open-source software, interoperability, because they make our lives easier, and they protect us going forward, and they future-proof our solutions. I'm not trying to build a thing for a company on a proprietary protocol that's going to have to be updated a year from now. Come on, please. Just add value beyond the way the machines talk to each other because I'm over the walled gardens.
Jonan: You've now ended up in Developer Relations, and you touched on many of the other things you've done. But if you were going to advise someone coming into the industry today who aspires to be in your position as I'm sure many of our listeners do, what tip would you give those people to end up where you are today?
Jacob: Never stop asking questions no matter how silly they are. I can't tell you how many times I was in a situation where I didn't feel not comfortable; that’s not fair; my voice was warranted in a particular space. And then I got the courage to do it, and not only was it warranted, but it was valued. It's actually what brought me into, weirdly enough, where I am today because, at Gremlin, I was in a pre-sales solutions architecture position which the idea is get deals done faster, technical win, that kind of thing. I always said, I and my team definitely disagreed with me about this like; I always viewed myself, me personally, as someone that was not in sales. Yes, that’s a technical sales role. I completely understand that. And I was, to be very honest, very successful there, but I was more focused on the education and the long term bet that that would eventually lead to deals, and it did, which is what led me to do blogs and tutorials and podcasts and videos and all this other stuff. And then it opened up the floodgates from a content perspective, and that's when I realized I wanted to move closer to the top of the funnel. But that's something that wasn't just a nuance for me in the middle of my Gremlin experience. That also happened to me at Fanatics as well, which actually led me to join Gremlin because we'd had a big incident over Cyber Monday, and guess who was on call? It was me.
Jacob: We had this big Cyber Monday outage which the problem is this is like 11 million dollar-hours. That's the money that we're talking about here. So they would never hire from that perspective, at least in my career. And we had this big issue, and we were just like, why? How did this happen? And this was also the same week that Nora Jones did that amazing keynote about chaos engineering at AWS re:Invent. This is in 2017. And I've never looked at my career the same way. A bomb got dropped on me. Wait a minute, proactive failure injection? And we could then see this coming. And at first, everyone was, “Yeah, okay, whatever. That sounds very gimmicky, and market, and stuff.” And then I did it. And then I went and did, this is what happened with the incident, and this is how we would reimplement it, and this is how we could've seen this coming. And it was something really small. Like, if our EBS volumes were this much larger, we would have been able to hold the load that we needed that was hitting our service discovery cluster, and we would have been fine. And it was a difference of; I don’t know, a few gigs between staging and production. It was just that mismatch. That's what blew everything up. And I was just like, I really loved doing that presentation and just talking about that pain. And then it went really well. I gave that presentation four times all the way up to the CTO, and I was just like, I just want to do that. Oh, Gremlin is hiring? Cool. I can help other companies not go through the pain that I went through. Cool. And then that ties into I can help -- And then you strikethrough company engineer, strikethrough engineer me. And that's how it all comes back around to empathy.
Jonan: This is really good advice. I think if you are able to embrace that empathetic approach to software in the same way that we do all the other relationships in our lives, you're going to have a better career overall.
Jacob: No doubt.
Jonan: So, thank you so much for all of the advice for everyone. This has been an absolute pleasure. If people wanted to go and find you on the internet, where would they find you?
Jacob: So you can follow me on Twitter @DuvalKingJacob where I'm typically yelling about Dungeons & Dragons and video games and also observability and some of the things that I'll be doing here in the near future. I also have a Linktree that's linked on that Twitter where it has my Instagram, and Twitch, and all this other stuff, and a bunch of the content I'm creating as part of the Empatheatre as well as a lot of the different tabletop RPG stuff that I'm doing on the internet.
Jonan: I look forward to getting to game with you sometime on a Twitch stream. This sounds lovely to me.
Jacob: Yeah, we'll make it happen for sure.
Jonan: Well, thanks again for coming on the show, and I very much appreciate your time. We'll make a point of hanging out sometime in the future, making some game stuff together.
Jacob: Yeah, especially since we're both vaccinated. Yay!
Jonan: Yay! This is awesome.
Jacob: Get vaccinated, everybody.
Jonan: Go out and get your vaccinations, please. Please. And if you could, please get a pin or something, so I can know you by your walk. I'm very interested to identify those I'm able to approach within six feet safely.
Jacob: [laughs] Right. Who can I hug, and who can I not? This is the important metric. [laughter]
Jonan: Yes. All right. Take care. Have a great day.
Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on therelicans.com. In fact, most anything The Relicans get up to online will be on that site. We'll see you next week. Take care.