The Relicans

loading...
Cover image for Firewalls & The Fog of War – Clearing the Haze in Tech with Shelby Spees

Firewalls & The Fog of War – Clearing the Haze in Tech with Shelby Spees

mandymoore profile image Mandy Moore ・24 min read

Jonan Scheffler interviews Shelby Spees, Developer Advocate at Honeycomb.io about her background in teaching and speaking foreign languages, and the overlap between linguistics and software, because in large part, linguistics is about conveying information efficiently with different systems. Similarly, in software, we're always conveying information to machines and/or to each other, (and hopefully, more to each other than to the machines!)

Shelby also talks about her work helping contribute to the CNCF white paper on observability, which is collectively working towards a goal for the betterment of ourselves and our peers.
Should you find a burning need to share your thoughts or rants about the show, please spray them at devrel@newrelic.com. While you’re going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you’d like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.

play pause Observy McObservface

Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's developer relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry. And we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on developer.newrelic.com/podcasts. We're so pleased to have you here this week. Enjoy the show.

Hello. Welcome back to Observy McObservface, the observability podcast with a community problem or vice-versa where we talk about whatever we want. Speaking of talking about whatever I want, I should mention that I'm going to be talking about whatever I want at an upcoming conference here at New Relic called FutureStack that starts on May 25th. If you're interested in registering for that event, it's totally free. You can come hang out with The Relicans, the rest of the DevRel at therelicans.com/futurestack. So please stop by. So today, I am joined by a very special guest, Shelby Spees of Honeycomb.io. How are you today, Shelby?

Shelby Spees: I’m doing really good, thank you. How are you?

Jonan: I'm doing all right. I am hanging in there, seeing the light at the end of the tunnel here with the pandemic situation. It's been a rough year, I think, for all of us. And I am eligible as of yesterday to get a vaccine, although appointments in Oregon are a little bit more chaotic than in some parts of the country. We're an anarchist jurisdiction, so we have a little harder time with organization. How about you? Are vaccinations coming along well in your area?

Shelby: Yeah. My mom just got her first shot—my dad's supposed to get his first shot this week. And then I think it's opening up in California on the 15th. So we'll see if any appointments are available, but at least more people can get it.

Jonan: I remember seeing a number the other day that it said something approaching 15% to 20% of America had been vaccinated already. That's pretty impressive, considering that being eligible for a vaccine does not mean in most cases that people are going to jump right in and get one right away. A lot of people feel like, “Erm, I'm nervous about going outside to even get the vaccine,” or whatever reasons they may have. It's hopeful. I'm feeling hopeful today.

So, tell me a little bit about you. I'm sure people are curious to hear about how it is that you ended up here working in the observability industry. Do you want to give us a little background?

Shelby: Yeah. So I've been at Honeycomb for about a year now. Before that, I worked as an individual contributor and engineer on a DevOps team, an SRE team. Before that, I worked in applications. And so I've been in tech for about five, six years now. And I originally got my bachelor's degree in linguistics. I did a minor in education and did an emphasis in Japanese, and I taught English for a year. I figured out that I loved teaching English in Tokyo or teaching English in Japan. I loved the job. I loved my students. I got to work one-on-one with adults and work with them on their business English and their tourism type of stuff. But I found that there wasn't really any career trajectory, and there were probably opportunities. But I started learning more about everyone should learn to code stuff. And I started doing online tutorials and classes and stuff, and I started really to like it. So I went back to school, took some CS classes, and started applying to internships.

And so my background is very heavy on the teaching side and the language side. And when studying three foreign languages in my undergrad, I gave a lot of presentations in languages I didn't speak very well. So I got pretty comfortable with talking about stuff that I might not be super confident about. And so when I started my first or even before I started my first tech job, I’d be taking my classes or trying to follow online tutorials and stuff, and I'd run into these assumptions in the tutorial or just like all this RTFM type, not even covert attitude, but just like it's in the water. And it was so frustrating, and I felt like I'd always run into these corners and these alleyways and stuff. And it was just very frustrating. And when I started working, I got to take part in -- I was an intern at two different companies and the second company hired me full-time, and they had a great internship program. They had these summer tech talks series from VPs and principal engineers around the company and stuff.

And so the next summer, I immediately glommed on to the interns. My manager knew that I really wanted to be a part of that. So I got to supervise three different interns on two different projects. And I did all this work to onboard them. I was organizing workshops for...we had clubs at the company. It was The Aerospace Corporation, so it's a very big company with a very academic feel. And I got to be part of the career development club. And so I was doing DevRel type work without being a DevRel. And I hadn't even heard of Developer Relations or Developer Advocacy until, I don't know, 2018, 2019, something like that. I just didn't know it was a job. And there were a lot of things that I didn't even think about, like where web applications came from until multiple years into taking computer science classes. I was just like, there's software somewhere in the world, and it runs. And it took me a very long time to understand what happens when you type in google.com or whatever, and there are back end servers, and there are load balancers and all of that stuff. It’s very much been this exploration where I feel like my mental map of the software space (and I realize I'm totally meandering), but my mental map of the software space is a video game map whereas you explore more parts of the map, it's like all black, but more parts of the maps are filling out and stuff. And that's very much been my experience. And I think there are a lot of people who have the whole map and they forget what it's like to not be able to see at all.

And so I wanted to give back, and I've been doing informal mentoring. I've been the documentation person in every job I've held and stuff. And I was interested in observability because one of my managers at my previous job I went out to coffee with him and he was like, “You should follow this person, Charity Majors. She's really smart. She talks a lot about all this stuff. And she's talking about the future of this industry.” And I was just like, “That's saying a lot, but I trust you.” And so I started following her on Twitter. And this was early 2019. And I think at that point, Honeycomb didn't have a big marketing department. They were still navigating the marketing waters. And so there was a lot of marketing –– that was very much like high cardinality problems, and instrumentation, and dimensionality of this and that. And it was all these multisyllabic words I had never heard before.

And on my team, I was six months into working that job, my first time supporting production web services. And I was trying to make sense of our Datadog, and we had Sumo Logic, but no one really used it. And these tools just seem really opaque to me. And so, what Charity was talking about really appealed to me. But I just didn't -- how do I get from here to there? And I didn't see a clear path. And it took me six months to start sharing this stuff with my co-workers, and it took me till the end of the year until I was like oh, structured events, oh, distributed tracing, that's wild. How is this stuff different? Or capturing stuff all the time instead of going in and adding timers around specific blocks of code and then having to deploy it or whatever and wait around, and being able to go back and look at the stuff that happened last night because you're already capturing that instead of having to guess at what to add instrumentation to. And so I'm like, this sounds great and stuff. And so my team actually started using Honeycomb. And I had a teammate who just immediately jumped on, and he's like, “Oh my gosh, this is what I've been wanting for years.” And so, at that point, it was out of my hands. He really hit the ground running and went nuts with instrumenting his services and stuff. And it was such a small team that he basically handled it all himself. And I'm trying to help in my realm. I had Chef, and I had Terraform, and that's all I was working with. And so observability wasn't really -- I didn't see how it would support me yet. But then I saw the job posting for the Developer Advocate role at Honeycomb. And I was like, I'm so excited about this stuff, and this sounds like the perfect role for me. And I messaged Liz. I'm like, “It says senior, but I don't feel senior.” And she's like, “Oh no, no, no, go apply.” And I guess the rest is history. So anyway, that was a very long, meandering story. [chuckles]

Jonan: No, it's fascinating. I love the background piece, especially when we get the detail. The motivations and everything are far more interesting to hear about. So the teaching and how you came to teaching, and you ended up in this place in Japan, and you're teaching English, which is already quite an experience. I think it's interesting to see the overlap between linguistics and software because, in large part, linguistics is about conveying that information and how we do that efficiently or with our different systems. And then, similarly, in software, we're always conveying information to the machine or to each other, hopefully, more to each other than to the machine—that knowledge transfer piece I want to come back around to real quick.

But I also want to compliment your analogy on the fog of war thing. You were talking about the video games. I feel exactly like that where I'm walking off, and you don't really know what's coming, but as you get out there, you lift this fog. And in some games, you actually have a sphere of visibility around you, and as you move forward, then things close behind you. It very much ties with my perspective of this industry that I've been around for a little while. And I remember very early on, part of what inspired me to get into DevRel, much like you, is that I felt like things could have been explained better. I would finish some explanation of how a thing is. And when I finally got it into my brain, it would be like, oh, it's like this? That's what you've been trying to say this whole time? And now I've been here, and I feel like I'm losing some of that. It's like following me through but maintaining those knowledge transfer abilities that inspired me to come here at a time where you and I are both gaining this experience in the industry is difficult. And I wonder what thoughts you have on, I guess, the fog of war and knowledge transfer and how you keep yourself able to have that empathy.

Shelby: Yeah, absolutely. And there are so many things where -- this happened the other day where I'll randomly be talking to somebody, and they'll use some networking term or just something that I never really put a lot of thought into. And so then I'll go look up, like, what does this actually mean? How do firewalls actually work? I remember when I first learned how firewalls work; I’m like, you mean it’s like access control, but for IP addresses or something like that. I probably just described it totally wrong.

Jonan: No, it is. And I always thought of a firewall as being this fancy thing, right?

Shelby: Like at the wall. I want there to be a wall. It’s not. It’s just a list of allowed things. And so I was so mad because your metaphor is wrong. I can't rely on the metaphor. And so there was one I wrote on Twitter the other day. I was just okay; why is it called scaling up? When we are talking about auto-scaling groups, you scale up, and you get more servers. But then we talk about horizontal and vertical scaling, and horizontal scaling is you add more servers, but vertical scaling is you make bigger servers. And I am a cloud-native engineer. I've never touched a server rack. I've never walked into a server room. I've never plugged in anything like that. And so I was like, do you imagine the servers get taller or something? And it turns out they literally used to get taller. The servers used to get bigger when you needed to beef up your servers. I imagine like big warehouses of servers. I'm like, well, if you just have a whole bunch of servers in a row, it doesn't matter if you add hardware in the slot above it or if it's the slot next to it or whatever. But actually, it's like urban planning where you can't just move into your neighbor's house. You need to build on top of your own house or whatever. And it blew my mind. I'm like, oh, like physical servers; this makes way more sense. And so there's a lot of stuff like that where I have this silly realization. I look up where the word came from, and it does make a lot of sense, but then there are so many times where it doesn't, and I get so mad. I'm like, you had the opportunity to name this so much better. [chuckles] Like, why?

Jonan: Naming things is hard.

Shelby: It’s very hard.

Jonan: It's a very difficult thing to solve, and I get that. But I also feel like the analogies are less involved with the names like the analogies that you choose for things like firewall, for example. I'm imagining some kind of complex wall that is reactive somehow because it's flamed. It is just a text file with a list of IP addresses that are allowed to talk to me.

Shelby: It sounds like a whole service that does something. It shattered my dreams of how cool firewalls should be because it's such a cool name, too.

Jonan: [Chuckles] Maybe once we get some AI firewalls. Firewalls are desperately in need of innovation. To all of you aspiring startup founders, make the AI firewall. I'm waiting for it.

Shelby: Oh God.

Jonan: That’s a thing that already exists. I’m sure.

Shelby: I’m sure it does. And I don't want to use one. [chuckles] I don’t trust it.

Jonan: Yeah, I'm good.

Shelby: But yeah. And so there's a lot of stuff where, for example, with tracing, I remember learning at least how Honeycomb implements traces; it’s a directed graph. And I can rely on having enough data structure knowledge that I know what that means. But to the freshman CS student who's never taken data structures, they don't know what that means. And so I try and think about -- and it's hard as I start to interact with people in more different roles. But I do think about who is the bright high schooler that I'm explaining this to or the freshman intern? I did actually work with an intern who was straight out of high school, like two weeks after high school graduation. And she took one programming class but didn't really learn anything. And by the end of summer, she was using Git and contributing features on a Python codebase. And so she was a rockstar. She was absolutely a delight to work with.

There are people who I had to teach. I had to explain how Git works, and I used the metaphor of okay; imagine that branches are parallel timelines in a sci-fi show, and you can go back in time and then branch off a new timeline. And that was like my mental model for that. And that's even how I used to tutor my classmates in my language classes; I would find the closest metaphor or the closest comparison in English. But there were a whole bunch in Japanese that I was like, “This is just like this thing in English.” And so the teacher would explain the grammar, and then I would turn to my neighbor and be like, “Oh, it’s like this.” And they'd be like, “Oh, I get it now.” And I've been tutoring since high school. And so I have this knack for finding the thing that clicks.

But I think the other thing is when I went back to school, I did this questionnaire that was like, what kind of learner are you? And I took it, and they're like, “You’re a multimodal learner.” So it was like I'm not just a visual or kinesthetic or auditory learner; I’m all of them. And it's not like I can use any of them. I actually have to use all of the different formats before a thing will actually click. And so it takes me like five times as long for things to click. But when it does, when it finally does, it's like, okay, I can now share this in any format. And so that's my weird superpower that is very, very hard because I feel very, very slow sometimes.

Jonan: I think that's a good thing for a Developer Relations person because we understand, we empathize with the learner. These things are difficult. Programming is difficult. Don't let anyone ever tell you that any part of tech is simple or easy. There's a famous site, Think Like (a) Git is the tutorial site my friend Sam wrote. And the opening line is like Git is an acyclical directed graph.

Shelby: Oh my.

Jonan: And it's like the whole quote from Wikipedia, and Sam was presenting it as kind of a joke. Like, this is not actually the easiest way to understand this thing, but the timelines is perfect for that. It's a directed graph. You've got time moving forward in the case of these traces thing, the traces that we were talking about. You have an application that kicks the thing off, and then maybe it goes to the login service and then the email service and then some other service. And then, it goes back to the user service that it used with the login service and then the request is done. And so you have this directed graph where each node to node you're moving in one direction. That's a very easy thing to explain in the context of timelines for Git or, in the case of a trace, maybe a slightly different metaphor.

But those metaphors and analogies I think coming up with them is directly descended from having that superpower, the ability to understand the struggle of having to learn, a thing that is difficult in a way that you can then transform into conveying that information back to your audience. And then there are other pieces to DevRel like determining your audience and know who you're speaking to, which I'm sure that you do a similar thing in conversation with people. There's like this calibration that happens where you're like, “All right, where are we technically? What are you really asking? What do you already understand so I don't waste your time?” I want you to get the value, and then you connect. It's a very interesting industry to be in, especially from a languages background. So you'd talked a little bit about this multimodal learning and this knowledge transfer piece. What do you do regularly to make sure that you can stay in touch with that beginner's mind? Because there's a learner's mind here, right?

Shelby: Mm-hmm.

Jonan: In every moment, you are trying to embrace the mind of a beginner. I have said often that I aspire to never be an expert in anything. I feel like experts know a thing. Once you are an expert, you are declaring I have achieved this knowledge. And I hope that I am never shutting myself off to learning in that way. What kinds of things do you do regularly? Like for example, if you were to offer a younger version of yourself advice on how to end up where you are today, maybe there are people out there listening right now who want to be where you are in your career; what would you recommend they do differently or the same?

Shelby: I have to stop right there and be like, oh my gosh, people would want to be where I am in my career? Like, they aspire to this. And I guess I don't think about that. But that's absolutely a thing. And I guess I just wanted to acknowledge how cool is that? And I feel very grateful to be in that position. So it's funny that you said talking to a younger version of yourself. And that's actually how I think about or how I approach a lot of my writing and my explanations. It’s just like, how would I explain this to Shelby from two years ago? And so I would tell Shelby from two years ago to explain it to Shelby from two years ago, and then it's like turtles all the way down.

Jonan: A directed graph.

Shelby: Yeah. So I don't want to lean on this cliché of how would you explain it to your mom or whatever. But my parents are wonderful, and they love me, and they support me. And they were never really interested in my geeky interests. And so they weren’t big about school. They didn't go to college, you know, working-class kind of stuff. And so there were times when I, especially when I went back to school, and I was living with my parents at the time, and I wanted to be like, “Look at this cool thing I just did in my code.” And it's funny because my mom had the reaction to my code that I had to my friends who were CS majors in college, which was like, “Oh, look at the pretty colors of all your different texts,” which is 100% the appropriate reaction. But also, I wanted to get my parents to understand, “Look how cool this is,” or whatever. Or “What shall your job be like? What does a DevOps engineer do? What are you working on at work?” I would say like, “Hey, you know how you have Facebook on your phone, and you talk to your friends on Facebook or whatever? Well, there's the part of Facebook that's on your phone, and then that talks to Facebook's servers which live in a warehouse. And there's some software on there that manages like, hey, here's where all your pictures are. And that talks to your friend's phones and stuff.” And so just enough that people understand how satellites work and so the just enough technical part.

So I'm lucky that I have enough variety or a big enough range of people's engineering expertise in my life that I regularly talk to people who just don't care what a server is and how can I relate this to their life. And so that's really what I do a lot of. It comes up at work because we have a lot of people who are technical in their role. I think marketing is a very technical role, but it's not an engineering role. And so they're not expected to know what Kafka is and why that would impact us when we have a problem. And so thinking about what's the just technical enough explanation so that people have the information they need to be able to do their role? And so that's something that I've even told co-workers, especially new people who come in and might be embarrassed to ask what they think is a silly question. It's like, “You know what? You can DM me your question, and I'll ask it anonymously on the channel.” Like, “Hey, I just got this great question. Here's my answer. What does everyone think?” Which is another part of this knowledge transfer stuff; it’s very hard to ask questions in public. It's very vulnerable. And so I think creating ways for people to safely ask questions in public or have those questions be more public is really, really valuable. And I know as well as anybody just the gut-punch of “You didn't know that already?”

Jonan: Yeah. This is my least favorite part. I think maybe of being in this particular type of work is that I hear it so often I hear someone do the “I can't believe you didn't know that. And community spaces try to make rules about this thing like; you can't say you can't play; you can't say you should've known because nobody knows the thing before they've learned it. We're all on this together. And we all have a different set of rules and a different set of toys in our backpack. And you're just doing your best in every moment. The fact that somebody doesn't know something well is legitimately astonishing to you. Just recognize that you would never have known of the thing without having not met versions of yourself right now. You only know that because you didn't encounter you who shamed you for not knowing it, right?

Shelby: Yeah. And I think it's really important, especially for people in teaching and mentoring and DevRel-type roles. But I think it's important for everyone to remember that technical knowledge like software knowledge is not linear. There's no single path. Just like someone can study history and be a super expert on the American Revolution but know nothing about ancient Greece. People in tech can be super knowledgeable about Android development but know absolutely nothing about distributed systems back-end services or serverless. There are just so many possible places you can go deep on something that it's totally reasonable for someone to have a super long successful career in tech and never touch the thing that you're an expert in.

Jonan: Exactly.

Shelby: And I feel very lucky to be in the observability space because it's a relatively new concept for most people in tech. We have maybe five years of collective knowledge or five years of examples of people using the word observability and talking about it. And we're still establishing -- like, I'm helping contribute on the CNCF white paper on observability. And it's very much an active discussion, and we're still discovering how to talk about this stuff. So I'm very lucky that I can go talking to someone who's a principal engineer like the most technical person in their company and can't expect them to know as much about observability as I do or articulate it as well as I can because that's my whole job. And so I'm lucky that everyone has a blank slate, and I get to feel kind of smart helping them [chuckles] because that's my whole motivation is I just want to feel smart sometimes.

Jonan: It feels good to help. It feels good to lift up people around you. And you're talking about the CNCF white paper, and I read through it; I saw your name pop up in that document. This collaborative effort to build something new is another piece that really inspires me to be here, that we are collectively working towards this goal that is for the betterment of ourselves and all of our peers is very inspiring. We are ultimately here to make developers’ lives and hopefully the world better at the same time with mixed results day-to-day in technology. But some days, I find plenty to be hopeful about. So this white paper, among other things in the industry, I love the idea that we are going to try and come together with a consistent definition of a term like observability, which, when left to their own devices, marketing teams across the world will pollute into obscurity, right?

Shelby: Mm-hmm.

Jonan: Getting that concrete definition is important. What do you think is coming for the future in Observability? What would you make as a prediction that we can then chat about a year from now when you come back on the show?

Shelby: So at Honeycomb, we tell people to instrument their code. And it sounds like this really tedious process. And so I'm hoping with things, especially with OpenTelemetry, that people can get more out of the box from their frameworks and from their platforms. Jaana Dogan talks about platform observability and bubbling up the AWS level telemetry that might be relevant to you when debugging your application and stuff. And I think there's going to be more of that where it's out of the box, better observability better than what we've had. And thankfully, with OpenTelemetry and with the tracing being generally available now, and with so many vendors and providers (that's what I meant, provider observability), so many vendors and providers being a part of that effort, we're all speaking the same protocol. We're all speaking the same language. And I think that's really, really, really powerful. And that's what's going to empower the developers and the software teams and the people responding to incidents to do what their actual job is. My job is to care about observability. My job is to advocate for all of these practitioners to make their jobs easier. Their job is actually to build services and create business value. And I don't want people to have to think about “Okay, but the protocol of my traces, am I sending the right trace headers?” I want that to be automatic and out of the box. I don't want people to have to think about it. And so I think that developer experience is going to become increasingly important in that next year.

Jonan: I absolutely agree with you. And I'm really excited about OpenTelemetry and technologies like eBPF that are going to allow us to do this whole -- I mean, it's all well and good to tell people, “You should be striving for observability, and you should be getting all of the instrumentation in there ahead of time so that you know and you don't have to go and instrument things bespoke and all of this.” But it's so much easier to make that argument when you're able to make a single installation of this thing, and then suddenly you have that data because you're using a consistent protocol. It's very likely, I think, in the near future that we're going to see frameworks like Rails implement OpenTelemetry features under the hood.

Shelby: Under the hood.

Jonan: Under the hood, right. And you’re just like, well, I've got Rails. And so then where do you want to point your Telemetry data? You just put in which service you're using. Here's your API key, done. And it works.

Shelby: 100%

Jonan: I really am looking forward to that future. So thank you so much for coming on the show, Shelby. If people wanted to follow up with you on the internet, where would they find you?

Shelby: I'm very much on Twitter. So you can find me at @shelbyspees, just one word. And you can reach out to my Honeycomb email if you have questions or anything, shelby@honeycomb.io. I'm always on our community Slack, The Pollinators Slack workspace. And you can go to my website, shelbyspees.com, to see what I'm up to.

Jonan: That is awesome. I love that you have so many options for people. I need to be better about that. Twitter is like my default, though. And if you are new to tech, I cannot emphasize enough how important it is for you to get on there and go and find some people. There are huge communities of people on Twitter waiting to support you. Don't feel like you're wandering in the dark anymore. Go on there and figure out how hashtags work, and start following a few that you are interested in.

Shelby: I can say with 100% certainty I would not be where I am today without Twitter.

Jonan: Same. Yeah, I agree. I tweeted almost like a Rolodex of the people that I have met at the various events in this industry, and it's how I learn of most new things happening in the tech world. It's just that that recency is so much harder to find elsewhere in the world. All right. Well, thank you all for coming to join us today. Again, if you are interested in coming to hang out with The Relicans at our FutureStack conference, which is coming up here in May, you can go to therelicans.com/futurestack and register. It's going to be silly. It's going to be fun. You definitely want to come, and it's free. So I look forward very much to having you here. And thank you again, Shelby, for joining us on the show. I hope you have a wonderful day.

Shelby: Thanks so much.

Jonan: Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on therelicans.com. In fact, most anything The Relicans get up to online will be on that site. We'll see you next week. Take care.

Discussion (0)

Forem Open with the Forem app