Jonan Scheffler interviews DevOps Advocate at PagerDuty, Mandi Walls, about Sysadmin vs DevOps, how people are the heart of DevOps, PagerDuty tooling processes, and how Mandi wishes she’d taken more time to learn more Python in the beginning of her career because it's everywhere!
Should you find a burning need to share your thoughts or rants about the show, please spray them at email@example.com. While you're going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you'd like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.
Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's Developer Relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry, and we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on developer.newrelic.com/podcasts. We're so pleased to have you here this week. Enjoy the show.
Mandi Walls: I'm all right. Thanks for having me today.
Jonan: Thank you for coming. I meant to ask you; I see this sign in the background that's for United Airlines. And it has a certain French statue that was donated to the United States on it. Does that mean that you're in the New York area?
Mandi: I am. I'm in Northern New Jersey, so yeah.
Jonan: Oh, cool. Did you all have this heat bubble recently where we almost died?
Mandi: Yes. It's been hot and hazy, and we have all the humidity that everybody on the West Coast loves to remind us that they don't have. So, yeah, it's been pretty gross.
Jonan: It's interesting to me because Oregon in some areas is very much like a rainforest, and yet the humidity hits me. When I land in New York, I'm like, I need another shower. I come out of the shower, and I want more.
Mandi: Exactly. It's like walking into a sauna every day.
Jonan: Yeah. Nice. Well, enjoy that heat. We're cooling down a little bit out here, fortunately. And then we just immediately found ourselves on the weather. Like, well, how about those...we used to talk about baseball next or whatever.
Mandi: I got nothing on baseball. I can talk about the weather forever, but baseball, I got nothing.
Jonan: So this podcast and the other ones we run was designed to kind of replicate the hallway track experience, one because I just missed meeting people for fun.
Mandi: Yeah, of course.
Jonan: And just having a conversation about the weather or even baseball. I mean, y'all can try. I will root for whatever team you tell me to, so I'm there.
Jonan: So, tell us a little bit about yourself. How is it that you ended up here?
Mandi: So my name is Mandi Walls. If you'd like to tweet at me or whatever or follow me on Instagram, I’m @lnxchk. I am currently a DevOps advocate at PagerDuty. And before that, I spent almost nine years at Chef Software. So if my name is familiar, I may have taught you Chef in the past. And yeah, my background is in systems administration. That's kind of how I came into the world through operations, and this is what we do now.
Jonan: Doing Chef things. You were Chefing all of the stuff. And nine years you did it at Chef?
Mandi: Yeah, it was nine years at Chef.
Jonan: So you mentioned that you came into this through systems administration, and it kind of like...and alarm is the wrong word.
Jonan: But it kind of piqued my ear because I was like, you don't hear a lot of people say sysadmin anymore. In fact, I have a friend who is almost trapped by that title, that their professional title has been sysadmin in places. And they're like, oh, no, no, no, no, what we want is a DevOps person, which is certainly a transition. I'm not saying that the DevOps movement is a small thing. There's definitely a difference but also, let's give those nice people jobs and bring them along for the ride. [chuckles]
Jonan: Do you think that that is a significant shift, like, for you going from the work that you did as a sysadmin? I mean, it's significantly different today, right?
Mandi: It is in a way looked at in sort of abstraction. If I look at the job that I had in 2004 versus what system administration or operations and DevOps looks like today, yeah, that's a huge change. But at the same time, it was a spectrum. It's like boiling frogs. It's changed slowly over time.
Mandi: And there probably wasn't a day where you all of a sudden were not running SSH in a for loop, and suddenly that had gotten replaced by some more sophisticated tool. It was a more gradual process for folks. But yeah, I understand. I’m like; it can be super scary. But walking through some folks through that transition, especially the years I was at Chef, talking to folks who had that more traditional role…and we were trying to get them to do some automation things and modernize a little bit. And picking apart the things they were doing every day wasn't that far off but getting the right mindset was a big part of getting them to the next step. And some folks like to only want to be in their parkas in the data center and doing that stuff. And that's still needed, but not by everybody who used to do it.
Jonan: I would like to be in a parka in a data center. And I probably still, I mean, probably I know for certain I still run SSH in a for loop. [laughter] I've built some nonsense things.
Mandi: It's right there. It's right there.
Jonan: It's really funny to me. I've been playing a lot with Raspberry Pis lately. And I'll write these really long CLIs because I love writing CLI. It's very calming for me to be in my little world and write a CLI. But I also recognize in every moment that a tool like Chef or Ansible or whatever else is out there could replace my whole CLI in a more efficient, repeatable way. I enjoy writing them anyway. So what sort of work do you get up to these days at PagerDuty? You page people.
Jonan: Are you the person who actually types them? Do you have a pager, and you type the phone numbers in?
Mandi: No. All of that's been automated now.
Jonan: What? On computers?
Mandi: Yeah, computers are behind it. Oh my goodness. And we do have people like our interns who are like, "What's a pager" It's like, oh my gosh. They've never even had to see one for any reason. So PagerDuty, the team that I'm on, we are officially DevOps Advocates. And we help folks with all of the other stuff that you want to be able to be successful with while you're being successful with PagerDuty. So we help them with running blameless postmortems or helping them put some rules around their incident command process to help them through using PagerDuty better. It's like that extra generic stuff that we get up to that's more modernized operations.
Jonan: This is, I think, one of the things that people misunderstand about DevRel. They think that somehow we have this motivation to sell the product. Certainly, I like New Relic. It's a great product. It has made my life significantly better, one of the first things I install. Also, it's not my job to sell it to you. We have a whole team of people who are here for that.
Mandi: Right? Yes.
Jonan: I'm just here to help you out. And I'm not even probably going to talk about the product all that much just because it gives people the impression that I am here to sell. I'm here to help developers who are doing cool stuff with data and trying to figure out what their code is doing in all capacities. Yes, it is tangential to my work. But you have a hard time finding something in software that can't be measured. Therefore, I get to talk about most anything in software and make a reasonable case to my employer that they should keep paying me, right? [chuckles]
Mandi: Yeah, absolutely, and that's where we are too. We have lots of people whose full-time job is staying up on all the idiosyncrasies of the product and, like you said, being able to sell it because that's not me. But yeah, all this stuff that's going to make you successful with it down the road, all that extra bits and pieces and the cultural aspects. It’s like, we know you can't beat it into people's heads enough that DevOps is not just tools; it's also people and culture. So we come in and help them with all that stuff as well. So we get all the fun conversations and none of the hands-on keyboard stuff.
Jonan: It's a pretty cool job.
Mandi: Yeah, it's pretty neat.
Jonan: So this part where you are claiming somehow that people are at the center of DevOps.
Mandi: Oh, imagine that?
Jonan: Yes. Tell me more about that because I obviously agree. I think that at the center of most pieces of software, there are humans. I actually, to some degree, I was a little bit surprised coming into the industry because you have this picture in your head of someone who writes software for a living, and there are certainly people who still do this. But you're just heads down, hands on the keyboard all day long, typing as fast as you can. That's maybe 20% of my life as a developer at best and before I came into DevRel. So when we talk about people being the heart of DevOps or a component, people and culture being just as important, what do you mean?
Mandi: It's right in there. The parts of your work on a day-to-day basis are less about sitting in a dark corner hacking away in that stereotypical feeling and coordinating with the rest of your team. The best software that we like, the things that we like, that we rely on, that we use, is not a single developer that put that stuff together. It takes hundreds if not thousands of people coordinated to get all the pieces together. If one of us had to sit down and start over with Microsoft Office or any of these other applications, it would take us years to rebuild them single-handedly; that’s not efficient. So we work as teams, and we have to be able to communicate. Oh my God, talk to people.
Jonan: Of course.
Mandi: And all these horrible things that the developer sitting in a closet isn't supposed to have to do and changing that whole mindset. And it's interesting; I’ve been working with software vendors now for ten years. And it's interesting to see how different parts of the industry have been moving more towards this because we all kind of went through that first version of Agile transformation where it felt like you were in high school PE learning a dance. Like, you're going to follow the steps exactly, and there's no room for improvisation or any kind of...it's just very awkward and stilted.
And as people get more comfortable with it and learn more about what works for them, they can improvise a little bit and step out of the frame a little bit where they need to, and get better and better at it. And part of that is knowing how your team functions, knowing the kind of communication that your team needs. Some folks are happy with Slack. Others are like, "If you don't put it in the Wiki, I'm not going to read it, not going to find it." So all that organizational learning is all part of the process.
Jonan: Slack, especially, I'm terrible about this because I'll be in a meeting with my team, and I'll be like, "Hey, we need to do this thing. Make sure you fill out this thing this week." It's gone instantly in a meeting or in a Slack message. You've got to put it somewhere to track it. We use Asana now. And it's like every time I try and get away with it, someone on the team is like, "Make a ticket. Make it a ticket. [chuckles] Put it in Asana." In doing the work that you do on your own team, do you use tools like this? What does your process tooling like?
Mandi: Oh goodness. We have so many things. So PagerDuty is a Slack user. So there's more than one Slack channel for every employee at PagerDuty, which is just an astronomical number of Slack channels. So that's all there. And then we've been experimenting with...because we have so many different projects. We have the talks that we do for conferences. We have the talks that we do for customers and scheduling those. And then we have a podcast of our own, Page It To The Limit, so scheduling that and the guests and getting all that stuff squared away. So we have all these different work streams that we've been playing around with. And we tried some Smartsheet for a while, and they're great folks. And then we tried some Asana for a while, and we've been working on that because it integrates with some of our other tooling. And some stuff is in Confluence, and some stuff is in our sales portal, and some stuff is somewhere else. And we've got a million things in Drive and…
Jonan: Tools sprawl. That's just what we do.
Mandi: It's everywhere.
Jonan: Yeah, everyone has their favorite thing to keep their stuff in. We've been lucky in that we spun up the team from scratch, and so we got to decide like, all right, Notion is where our team Wiki lives. And if it's something that is like tribal knowledge, it all goes in there. And if it's an action item, it goes over here, but still, even then in the six months we've been around, the sprawl is incredible. I love PagerDuty, by the way. You are in my phone. I have a little feature that only lets my phone ring for certain people. It's basically my immediate family and PagerDuty. You get to ring my phone.
Mandi: Of course. Yup, me too. That's how mine is set up as well.
Jonan: I've been playing with that API; I feel like since I first became a developer. I've got some real OG PagerDuty swag, but the number of applications for that thing...I imagine people mostly use it to let you know when things are bad. But I could imagine a world where I build all sorts of silly robots and things that can page me in the middle of the night to let me know that the temperature in my house has dropped two degrees or whatever. [laughs]
Mandi: Or whatever, yeah. Anything that's important. I think we're up over 500 integrations right now. So like any given environment, we can hook into all kinds of crazy things. And just trying to wrap your head around all these components is wild.
Jonan: What's the weirdest thing you've built with it?
Mandi: I don't get to build too many weird things.
Jonan: So what's the weirdest thing you've seen with it? Have you seen anything, really unique ones?
Mandi: There are some unique ones. There are some interesting ones. Like you said, the IoT stuff is the interesting stuff. Everybody can hook up their CPU utilization or their Kubernetes health or whatever they're doing and file that through because it's built that way. But we have a couple of use cases on the website. One is food delivery, but they have their chillers, their refrigerators hooked up to PagerDuty. So if the refrigerators go out of range too far, they'll lose everything that's in that fridge. It will no longer be fresh food. So you have your food handling requirements or whatever, so they can monitor their chillers, their refrigerators and then be notified if things go out of range. And that is such a wild thing to think about that yes, that's the important part of their job is ensuring the food safety for all that stuff.
Jonan: I have an above-ground pool in my backyard that we set up every summer, and it is always just a comedy of errors. It's always broken for some reason or another and leaking somewhere. And I have this theory that I'm going to put a little Pi Zero on the side with a liquid level sensor, and I can have it page me if the water drops rapidly as it does whenever a child trips over a hose or whatever.
Mandi: Yep. There you go.
Jonan: But I am now suddenly reconsidering the plan because that's a lot of pages.
Mandi: It could be a lot of pages. There's other stuff that you could put in there like your pH sensor if you need to add more chlorine or whatever or shock it or et cetera, et cetera.
Jonan: That's a really cool idea, actually. I could build a whole sensor because I need another project in my life.
Mandi: Of course you do, 100% you do.
Jonan: I'm always looking around for more work to make up for myself. Why do we do that?
Mandi: I don't know.
Jonan: Why are we always...developers invent things all the time to make their lives hard. Oh, speaking of, we were talking a little bit about titles earlier. We were talking about sysadmins and DevOps. And I have been in the habit of using developers to just mean humans that write code sometimes. And I also know that many of my DevOps friends identify with that term, and some of them don't. They feel like it's kind of...I'm all about inclusivity. I want everyone to feel like they're on the boat, but terms like practitioner just feel so clunky to me. What do you use?
Mandi: I use practitioner. I know it feels clinical. It feels kind of detached. But at the same time, like you said, I feel like for folks whose main role is site reliability or automation or all these other things where it's code but not exactly the same way as it would be as open up your IDE and put stuff in…and yeah, so we kind of slide towards practitioner. I haven't found a better word. Just call them cool people, and that's going to lose context. There are just so many cool people.
Jonan: Yeah, but most of them write code.
Mandi: Most of them write code, yes, in some fashion.
Jonan: So then let's imagine a world here where you have the kind of work exactly that you want like in your current role. We're not doing like a millionaire scenario or anything here where you win the lottery.
Mandi: Oh, come on.
Jonan: Well, we could do that one too.
Jonan: But you get to build the company, but you don't have to run it. And you get to design your job. You effectively get to write your job description as long as it's plausibly related to your current role. What does that look like? What would you spend your days doing if you didn't have to go on podcasts with silly people halfway across the country?
Mandi: Well, maybe I would just do more podcasts.
Jonan: Maybe it's all podcasts all the time.
Mandi: All podcasts all the time. I like doing a variety of different things. So like I said, we've got a lot of different stuff that we work on. So I like that combination. I like talking to customers and even folks who aren't customers. I like seeing that...understanding that negative space where we're not fulfilling their needs and thinking about those things. But talking to customers about their interesting solutions, the things that they're doing, how the product that we're working on happens to help them do whatever amazing thing it is that they're doing. Because they're out doing education or healthcare or retail or transportation or whatever it is. And being a part of that is super interesting and figuring out, like you say, all these unique use cases and all the things that they're thinking about because we're coming at it as software developers, software people. We don't really see that side of it. So that part is super interesting to me. I like talking to our customers. I like going out and meeting other users.
It's kind of interesting being part of PagerDuty because we're the messenger that everyone loves to hate in that we're going to wake you up in the middle of the night, but it's not our fault. We're just the messenger. So that part is interesting where we meet community members, and they're like, "I have PagerDuty, and I hate you, but I love you."
Jonan: We have a similar issue sometimes that New Relic is mostly people only open the dashboard when they're angry.
Mandi: Yes, exactly.
Jonan: I'm like, you can just come say hi anyway.
Mandi: Anytime. We're always there.
Jonan: You can just have it open while you write your code. Watch for regressions. We can just be your friendly partner in this instead of the UI you curse at way in the middle of the night because you only have one eye open.
Jonan: Yeah, I know the feeling. I've been there. So, I have a couple of questions that I try to ask everyone who comes on the show. And the first one is for posterity's sake. I want to be able to say next time we have you back on the show, "Mandi, do you remember when you predicted this thing was going to happen?
Mandi: Oh no.
Jonan: I can't believe you thought Lucky Charms were the future of the internet." Please don't sue me, General Mills. I should not have used an actual product name there.
Jonan: You make lovely cereal; please disregard. But what do you expect is going to be the next big thing in the next year or two? What do you think we're going to see more of or less of in trends in the software and observability industry?
Mandi: That's hard to say. I feel like in the near term; there’s still going to be a lot of people catching up. I feel like we're still in the early part of the hype cycle on real observability. And over the next 12 to 18 months, I think that'll start to get to the next part of crossing that chasm and into a place where the folks who don't have...they're not the twiddlers, right? They don't have the kind of time to sit and pick apart all the deep, little details. Not that they wouldn't want to, but more that they've got other million things that they're trying to work on.
So I think we'll get to a point in the next year or so where the products will start to call less around more recommended practices and better on start guides and things like that for those folks who are like...so we'll start to see more people adopt the practices as they are because it will be helpful for them. It's just, I think right now, things have been too raw and too unstable, and as things sort of call less, that's what starts to bring those folks on, and that looks like where it will be headed.
Jonan: This seems like a perfectly reasonable direction so that the people who are maybe busy with the business of doing busy business things over the next year...I mean, the kind of company that has just in the last couple of years embraced the idea that cloud is a good idea.
Mandi: Cloud exists, and you can use it, yay.
Jonan: They still have a lot of learning to do to catch up.
Mandi: They do. They do. And they're busy. They're so busy. And part of the issue there, too, is this is a lot of your always up kind of businesses, and they don't have the time to play with things that aren't 100% there yet. And so they might have an innovation lab or something like that. They're just maybe starting to play with things. But yeah, it will get to more mainstream for them.
Jonan: They don't have access to a lot of downtime like banks and hospitals, for example.
Mandi: Absolutely not.
Jonan: [chuckles] Yeah. They're like, "Sorry, you can't get your money this week because we're trying Kubernetes."
Mandi: Right? Welcome to failure Friday. The ATMs are down. What are you going to do?
Jonan: HugOps, am I right?
Jonan: No, you are not right. I need to pay my bills right now.
Jonan: But also, HugOps. If you work at a bank and you bring something down, I will be there to support you.
Mandi: 100%, 100%.
Jonan: So our other question is what advice you would give to yourself. So imagine you've traveled back in time and you're there starting out at Chef however many years ago, your beginning of your nine-year stint at Chef. What advice would you give yourself earlier on in your career, or what advice would you give someone today? There's someone listening today who wants to grow up and be the next Mandi Wells. What would you tell them to guide their career?
Mandi: I'm thinking back. So I joined Chef in 2011. So at that time...so my deep background, I have degrees in computer science and all that stuff. But I'd never been a developer as my full-time day job, except for short stints. But I would say one of the things that I wish I had done at that time was...one of my favorite programming languages was Perl. And that's unfortunately not...it's only helpful to me on the command line when I feel like writing things that look cryptographic. And one of the things I wish I had been able to help people with more when I worked at Chef was running more stuff like Python and things like that in production. And I had come in from a Java environment. So you could run Tomcat six ways to Sunday, but it wasn't able to help anybody with the deep Python stuff. So I think that's probably one of the things I wish I had taken more time to do was learn more Python at that point. It's everywhere.
Jonan: Or Ruby even. You could have just learned Ruby too. I just got to put a plugin for my language.
Mandi: Yeah. Well, Chef was written in Ruby. And I say that, but at the same time, when we were actually out dealing with customers…like, Rails is pretty straightforward to run. And because Chef was in Ruby, we had a lot more tooling for it.
Jonan: Yeah, running Python apps in production at the time...I mean, nowadays, it's a whole new world.
Mandi: It's completely different, absolutely. 100% crazy talk.
Jonan: I can imagine, though, that there are a lot of people at Chef who still have some animosity towards Ruby after that DSL, after what went down with the...yeah.
Mandi: Yes. Every time I turn around, there's some kind of drama in Ruby land. And for a while, it was all Nokogiri. And all you had to do was to look at a Chef engineer and say, "Nokogiri," and their face would collapse.
Jonan: I know. Those native C extension gems that...and I get it. Nokogiri is actually a fine piece of software. It's very, very fast for what it does, but getting the dang thing installed for a period of a year, at least, was just like pulling teeth.
All right. Well, we have a prediction for the future, and we have your advice for the up-and-comers; learn Ruby but also Python is an option I think was the advice that you gave if I remember correctly.
Mandi: Yes. Yes. [chuckles]
Jonan: Okay. What else do you want to share with our listeners before we call it a day?
Mandi: Yeah. So if you are a PagerDuty user, and we hope that you are, and if you're out there and looking for tips and friendly faces, we have a community site where our support folks hang out, and they're there to help you. And we post stuff about the things that we're doing, so that's at community.pagerduty.com. And we'd love to see everybody's smiling faces there. So yeah, it would be great.
Jonan: If you build one of those sensor rays that pages you when your pool is empty or PH is off, let me know. Tweet at me. I want to read about it.
Mandi: My plan will be for the water in my garden beds, so we'll see how that goes.
Jonan: Nice. There you go. Have a lovely day, Mandi. Thank you so much for coming on the show.
Mandi: Thanks, man. Thank you.
Jonan: Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on therelicans.com. In fact, most anything The Relicans get up to online will be on that site. We'll see you next week. Take care.