Yet Another Blog

DevRel Strategy

2022-10-06T00:00:00-05:00

Here's a quick view of my take on developer relations strategy.

I was chatting about DevRel and community strategy with some folks at DevOpsDays Houston after my keynote earlier this week, and I promised I'd write a bit about how I view strategy. The key thing about strategy to me is providing a framework for evaluation of tactics.

Strategy is a definition of risk valuation and reward opportunities across multiple focus areas. If you're defining strategy simply by naming a bunch of tactics, you're missing out on the creativity boost a good strategy framework can provide for a devrel team. Developer relations opportunities can appear and disappear quickly, like a collaboration proposed during a hallway discussion at a conference or a last-minute invitation to be on a livestream or podcast. You need to empower your devrel team to make decisions quickly and decisively. Strategy provides the framework to make swift decisions that benefit the company and the community as a whole.

A good devrel strategy explains what risk your company can handle for different focus areas. For example, a speaking engagement requiring expensive travel is high risk. To make it worthwhile, the reward for the company and the community should be as high or better. Maybe that means you get to reach a lot of people. Maybe that means you're reaching a new, very receptive community with a talk that will teach a lot to that community. You'll need to define that reward level for your company and communities. Conversely, a low risk opportunity may also be low reward and not worth the time. A generic blog post that isn't saying anything new might be low risk, but it also does nothing for the community (and so never benefits the company, either). If you can define the levels of risk your company can take with the reward levels you want to reach, you don't need to map out a bunch of tactics (the actions you'll take to reach a goal). Instead, you can map out a few and hunt for unique opportunities along the way.

By using a true strategy framework instead of defining a bunch of tactics, you open up the door for creative ideas—the kinds of things that catapult your devrel program to new heights—to flourish. So the next time you find yourself in a planning session, ask: Are you defining strategy or tactics? You'll surprise yourself what you can come up with when you have the mental space to innovate and evaluate those innovations (and prove value quickly to your leadership).

(I originally put this up as a thread on Twitter, in case you want to explore the original.)

An Analogy about DevRel

2022-03-19T00:00:00-05:00

So I’ve had a few folks who apparently liked my developer relations (DevRel) analogy that JJ Asghar outed for me at our panel on community building for GAN and Orbit for SXSW this year. So, since folks are asking, I'm replicating it here:

So imagine your company is a bed and breakfast in a quaint little town. Your engineering team and product team are all the people keeping things running, from cooking to cleaning to fixing. They work their tails off to make the bed and breakfast a great experience. The sales team is the folks who greet you at the door. They help you find your room, or they seat you at your table. They’re working with your business ops folks to ensure the money is gathered and counted so everyone can keep doing what they’re doing. The marketing team, meanwhile, are the folks making the front porch the most welcoming place to be. They’re maintaining the lawn, keeping the paint sparkling (and shiny compared to the neighbors), and ensuring that anyone who sees the building is drawn to it.

So where’s DevRel in all this? Well, they’re out in the town square five or six blocks away. They’re helping folks find where they’re going, they’re explaining the town’s history to passersby, they’re recommending restaurants to tourists. Oh, and they’re wearing the bed and breakfast’s logo and recommending it as a place to stay or eat with a wink and an “Of course, I’m a bit biased as they ensure I can be out here to help you.” People respond well; they go find the bed and breakfast that this nice, helpful person talked about.

DevRel professionals are out in front of your company, introducing the company to new communities by being the helpers and the teachers and driving a positive association so it’s easier for sales to call out and encourage someone to come in for a glass of sweet tea. People like me are trying to be 2-5 years ahead of where the company is now, laying the groundwork for the company so when it’s time to introduce yourself to a new community, they’re already interested in listening to you. We're making sure everyone has a great experience, one person at a time. Because that's how you win communities' hearts (and their minds will follow when they experience your product, hopefully). As some smarter folks than me said, "It's up to us to ensure they know we're here."

(I originally put this up as a thread on Twitter, in case you want to explore the original.)

Adding Logs to Legacy Applications

2019-11-15T00:00:00-06:00

As the final interactive in my mini-workshop at DeveloperWeek Austin 2019, I posed the following scenario to the audience:

You have a legacy application that has not been updated in 5 years. The system is running Python 2, which is sunsetting in January 2020. The system recently had its first incident in nearly 4 years, and your team was among the group that had to bring it back up. The logs that you received were not very helpful, and bringing the production instance back up ended up being a lot of trial and error.

Management has decided all applications must be on Python 3 by the end of code freeze in January 2020. Your team has been tasked with updating the application to use Python 3. It's the ideal time to add proper logging. How would you go about planning and executing that logging update?

The scenario generated a lot of discussion, and people had some very good answers. However, someone on Twitter (thanks for the question, @russellyazbeck!) pointed out to me that I completely forgot to provide my take on the same scenario. Whoops! I did respond in a thread on Twitter, but I'd like to lay it out and expand on it a bit here as it's definitely easier to find.

First and foremost, the members of the audience that started with bringing everyone together were spot on. You can certainly have your own data that your immediate team gathered during the incident, reaching out to the rest of the teams that responded to the incident and gathering their data is also important. Why? They have different perspectives. It's very easy to get lost in your own context and stop noticing quirks or issues unique to your programming language of choice, to a platform you're somewhat familiar with, or even to a team that you have worked with in the past. Ask anyone who has ever taught a concept to someone else successfully, and you'll find that nine times out of ten, those people will mention how they were surprised initially where someone got lost or how hard it was to avoid jargon that needed further explanation. So ensure you get data from anyone who was involved in that incident so you can start understanding what other people might need to understand the same issue.

Once you've gotten everyone together, start talking about the ideal incident response and the ideal data that you would have gathered. What should those logs have shown? What data did you actually need? In addition, what data was just noise? Was there any data that duplicated information? In addition, you can use this time to discuss which log levels would be useful for each type of data. Log levels help reduce or structure the noise coming from a logging system. Since a good production system allows for tuning the logs based on which environment you're in (dev, test/QA, staging, prod, or some combination thereof) and since any and all teams doing ops work on said system would likely love you if they don't have to decide whether a log raised by your system is a deprecation warning (WARN) or something that isn't acting right but won't take down the whole house of cards (ERROR) or something that took down everything including your databases and networking (CRITICAL), coming to a consensus on which logging levels are necessary and how to define them is really important both now and in the future.

Now that you have a much better idea of what kind of data you actually needed, you would pick a library or logging structure that could help give you what you needed. If you were in a scenario that had a bunch of other apps going, as would be likely in a scenario like this where there's a legacy application and multiple teams that likely are working on multiple projects, I'd definitely look to a structured logs library like structlog. While I could roll my own on top of the standard logging library, my guess is the rest of the team (and future team members) would likely find a library with good docs and standardized uses much easier to use to maintain good logs in the long run. An opinionated logging library would likely be best to ensure everyone logs well. Personally, I wouldn't use only text logs for this sort of situation, even if there's only one application that your company owns. Start as you intend to continue so that it's a lot easier down the line to ensure future systems are easier for others to onboard onto with similar features, common style, and other familiar elements. However, you have to keep in mind that this mentality includes a point that I made in the workshop: The audience of structured logs isn't really a human or a set of humans, but rather many machines parsing the data for you.

By the way, I want to point something out. I chose the Python 2 to Python 3 conversion scenario because it's one of those moments that's an ideal time to add logs. You're already in the codebase digging around and touching everything. You're getting to know what's there, so you're unlikely to skip anything major (well, assuming you're not using six or the built-in 2to3). It's also the ideal time, as noted by a few folks in the audience, to add in a deprecation warning for anything that relied on the Python 2 conventions for hitting the application. However, it is a bit of a red herring. You can add these kinds of logs to any system at any time. Legacy systems are often viewed as the most dreaded to work with, hence the scenario, and incidents are one of the ideal times to take a step back and understand what data is flowing through your system. However, you can use this same thought process for a modern application, an application that hasn't had an incident, or even an application that's brand new. Walk through the scenario as if your application just had that moment happen (or knock it over deliberately in dev or staging when those environments are not in use for anything critical), and see what comes out of the brainstorming exercise with the various teams that would theoretically be involved. Then add logs and monitor the outcome.

How else would you respond to this scenario I laid out here? Come find me on Twitter and let me know.

IRL

I'm going to be at KubeCon North America 2019 this coming week! Come hack on the broken k8s repo with me (more from that here), and see what swag I have for folks that contribute environments or significantly update documentation. I'll be at the LogDNA booth or floating around the conference.

Working Within Imposter Syndrome

2019-11-04T00:00:00-06:00

Imposter syndrome is a huge topic to cover in tech right now. I constantly see people on Twitter bringing up their imposter syndrome when talking about their everyday jobs, their interviews, their life history. I especially see it now during the ramp up to the next conference season as CFPs are opening and closing. I'm not immune, either. I nearly posted my own version of this frustration today while sitting here staring at my screen like my next CFP response would write itself. Writer's block and imposter syndrome is a very brutal combination. As I started on my own coping strategies, I realized I've never written down the tactics I've used to combat my own imposter syndrome. I've shared it during talks, coaching sessions, and informal gatherings, but I've never really written some of them down. So how about we talk about how to work within imposter syndrome to get a positive result, instead?

What it looks like for me

For a bit of history, I actually come to the development world through a fairly circuitous path. I'm a self-taught dev. While I was surrounded by tech growing up, I almost always ended up watching or reading the manuals instead of doing any tech myself. I loved to tinker with hardware, though, from doing FIRST Robotics in high school to messing with old laptops in college. I installed a lot of Linux distros on the old laptops just because I could afford it and I wanted to see what my older brother was talking about. In college, I took one programming course, and I picked up the rudiments of a few other languages while I went through my degree in earth and atmospheric sciences. My love of hardware brought me to Arduinos and some C work and then to Python, but I was a hobbyist at most. Even when I ended up starting to teach Python to beginners, I never felt that I was a real developer. I was just an editor that did programming for fun. It wasn't until I joined Rackspace that I was given the chance to take development on as a profession, and I grabbed the opportunity with both hands.

As you might imagine, this checkered history means imposter syndrome for me comes out in a lot of ways. When I sit down to respond to a CFP, for example, it hits when I start looking at featured speakers or at how many attendees they expect. It hits whenever I sit down to write on this blog about anything remotely technical. It can even hit in the middle of a talk if I see someone getting up to leave—or someone coming in after I start to sit down and listen. The manifestation should be fairly familiar to many: Who am I to talk about this? How can my small experiences even compare to the history that person has on this topic? Why are they here listening to me?

Strategies from the writing world

When I end up facing my imposter syndrome periodically like when I need to respond to a CFP or when I know I want to get a blog post out, I start treating it a bit like I would treat writer's block. One thing that always worked well for me for writing was to change how I was writing. If I was staring at a screen, I picked up a pen and grabbed a notebook. I actually did that to start writing this article. Once I got moving, I could switch back to the medium I intended to write from and keep going. Just that change of venue helped tremendously. My theory is that it works for two reasons. First, my brain gets stuck in a loop staring at a screen thinking that I have nothing to write. By switching to a different medium, that thought process falls away and allows my brain to start a new train of thought. Second, the tactile stimulus of a pen and a piece of paper sparks different neural pathways, prompting the ability to think creatively about something new. Not the most scientific rationale, but if it works, it works!

Another strategy when I'm dealing with imposter syndrome in the middle of something is to get up and walk away. If I'm writing from a place of feeling inferior or a fraud, my writing will reflect that. I know that, even if I'm still moving, the momentum I have isn't going to get me to the end of the task because I will need to rewrite it again later, or I'll be fretting about it for hours afterward. So I actually physically leave my desk, often to go make a cup of tea. Leaving the task to go do something else, especially something so process-driven and calming as making a cup of tea, resets my brain. I stop thinking about how I don't know what I'm doing when I get a whiff of my favorite black tea or a good herbal blend.

Now, sometimes I can't physically walk away. Maybe I'm in the middle of helping with an incident, to go completely outside of the world of writing. The problem with imposter syndrome for me is it's often caught up in a charged situation. High stress, high nerves, strong reactions. In short, a situation where I care a lot about what's going on and about getting things right for one reason or another. Even in those cases, though, I can take a moment to take a sip of water, or to turn to a slightly different facet of the problem. It's a forced disconnection with the situation at hand, even for a moment. Something just long enough to let myself release that white-knuckle grip and release whatever thoughts of insecurity I have to remember that I'm sitting where I am for a reason, even if I can't believe it at the time. These kinds of highly charged situations really are more often solved for me, though, with strategies I learned as a performer.

Strategies from the performance world

Right before I got into editing, I was a science museum educator. I had to give shows and demos all day long. Despite my degree, you better believe that I had imposter syndrome come up daily! I learned a lot of coping strategies to help myself get through every talk.

Have you ever heard the phrase "fake it until you make it"? Well, that attitude applies to imposter syndrome for me when it comes to being up on stage. The illusion of confidence and knowing what I'm talking about goes a long way to actually making me confident enough to stand up on stage, and believing that I am knowledgeable enough to answer questions helps a lot. I get there by having a confidence booster collection that I'll read before I go onstage. It's a collection of messages, comments, testimonials, or anything really that come from other people that tell me I did a good job. If you use Slack a lot like me, starred messages work wonderfully for this. When I was at Rackspace, I starred every message that noted things I did well, from solving a production problem to positive messages about hosting our internal conference. Whenever I started really spiraling into imposter syndrome before a talk, I would flip through those messages to remind myself that others respected my voice. Back when I was at the museum, I would remind myself of where I came from by periodically writing the name of my degree or alma mater on the inside of my wrist under my watch in tiny letters. Whenever I needed that confidence boost, I would look at that message to myself and take a moment to remember. It's amazing how a small reminder that you've done harder things can help you fake that confidence in the moment until you actually believe yourself.

Another thing I had to learn was how to handle mistakes, missteps, or failures, including demo failures, during live shows. The fear of exposure as a fraud that comes with imposter syndrome is a lot like the fear of failing at something while on stage for me. Both situations, as I mentioned before, are highly charged. So I learned the art of acknowledging a failure live. Depending on the situation, I would turn the failure into a joke ("I guess the volcano is still tired from daylight savings time") or would just forge onward ("Well, that dataset isn't the one I thought it was. However, we can still talk about this one."). Sometimes, none of my coping strategies work and I still feel like a fraud. That's ok. I force myself to change the lyrics live. Rather than "Who am I to talk about this?", I respond "While I might not be the one to talk about this, I won't know until I try." Or instead of "Why are they listening to me?", I make myself think "Well, they're here, so let's make it fun anyway." Acknowledging that fear and then changing the focus is a strong strategy that I use whenever I can't seem to shake that feeling of inadequacy. It takes practice, just like handling situations live, but it's well worth the time it takes to learn.

IRL

I'll be using these a lot this week as I'm going to be presenting at DeveloperWeek Austin. I always get nerves and imposter syndrome before I go on stage, no matter how many times I've done it. If you happen to be here and look really closely before I go up on stage, you'll likely see some tiny letters on the inside of my wrist, waiting for me to need that boost, or me flipping through my phone reading my confidence collection. In addition to the conference, I have a lot more CFPs to respond to this month. I'll be using all of these strategies to get through the stressful process of getting words on a page.

How do you work from within your imposter syndrome? Come find me on Twitter and let me know.

Introducing the Broken k8s Project

2019-10-22T00:00:00-05:00

One of the most common things I see with people new to development or devops is a lack of resources for them to learn good troubleshooting tactics on their own in a near-real-life environment. There are great tutorials, but I haven't been able to find a good, free sandbox filled with deliberately broken environments that need fixing outside of a corporate environment. Sure, you learn tactics on the job or on your own system when you accidentally bork it, but those scenarios can be very frustrating for a beginner who doesn't even know where to start to find the cause of why their systems aren't working when there are multiple reasons it's broken. Most of the time, you have to have a more experienced mentor to help debug, and those mentors can be hard to find for some folks in the community due to factors such as geography, time, or biases against them for one reason or another. In addition, containerized systems have the added drawback of being ephemeral, making it harder for new folks to figure out just how to begin when they don't even know how to access the broken system in the first place.

As a result, I thought it would be great to take a number of systems and just start deliberately breaking them in specific ways that I often see when dealing with broken systems and then providing them to the community as a set of stepping stones to debug more and more complex Kubernetes systems. So I started with a minikube build that is fairly bare bones to set a control environment. This environment is mainly to ensure that the user's system works and is configured properly. Then I started tweaking things here and there to break the system in fairly straight-forward ways to give people a chance to get some easy wins. In addition, these easier systems can teach a new user to hunt for problems in specific places, isolating the noise in the logs to surface common errors and their symptoms in a way that will be recognizable for the next set of systems. Each system lives in its own namespace on minikube to avoid cross-contamination by accident. Next, I aim to build out complex systems, make some attempts at breaking them in multiple ways to get a new person used to the idea that solutions are almost always not perfect, and introduce some variability into the mix, as well. Hopefully, building a set of troubleshooting tutorial sandboxes in this pyramid-like fashion will teach good practices and hone troubleshooting skills before those skills are truly needed.

I'm mainly posting about this idea to ask for help, even if it's just sharing the tweet I'm sending out with this post or sharing this repo with people you know. I've started on building some systems, but I know I haven't seen everything out there, and I know there's people with a lot more experience than me. The GitHub repo I started, https://github.com/nimbinatus/broken-k8s, is open to the public, open sourced with an MIT license, and open to contributions, including an initial contributor's guide on what should be in each broken system's directory.

Join me in making the devops world a bit easier to break into. I'd love your help.

A First Dive into Python Logging

2019-10-21T00:00:00-05:00

As part of a workshop I'm preparing for, I decided to do a deep dive into Python's built-in logging module from the Python Standard Library. I always had just used the module without thinking too heavily about how it was put together or how it worked. import logging and then away I went. Well, that's about to change, and I thought you might enjoy coming along for the ride. This post will go down into some of the module, especially the bits and thoughts that aren't detailed in the library reference, but I'm sure I'll miss spots along the way. We'll come back to it eventually!

The Python standard logging module makes the basics of logging easy for a developer. Import the module, then use the built-in error levels to send messages to standard out or standard error (stdout or stderr). The error levels are debug, info, warning, error, and critical. Pretty simple, pretty obvious&emdash;just the reason that I love Python. As you might expect from a logging system, the debug and info levels are suppressed by default. Python also allows you to configure the module to send those lower levels through direct changes in the Python interpreter if you're calling your script in the interpreter, through a command-line argument, or through a config file you've generated yourself. Python, like many other languages, doesn't tell you what to use each logging level for. It doesn't decide that for you. Knowing what to send to where at which level is an art in and of itself as many an ops or devops person would tell you after they've seen overly verbose logs and nearly non-existent logs. Perhaps I'll dive into the different logging levels in general in a different post.

The module is really made of four main classes: loggers, handlers, filters, and formatters. Think of handlers as the orchestration layer, with loggers doing the actual interaction, filters doing the job of deciding what to send to the handlers, and formatters generating the strings to send on to the handlers. There are as many handlers as there are destinations for your logs, from stdout and stderr to files to cloud logging providers. Logger objects can be created in a hierarchy of loggers, meaning you could create a logger for a parent class, for example, then have loggers for each child class that inherits from that parent logger and propagates logs back up the chain through the handlers. That sets you up to use filters and send logs to different places based on the handlers. Filters can be attached to loggers or handlers, meaning they are explicitly added to one or the other through a method call on the logger or handler. One interesting attribute of filters is that you can use them to add context details to a log line, which is interesting to me due to the structured logging formats that we'll see in a minute. Finally, formatters take in LogRecord objects and generate the final message structure you want to send to a handler. While these messages are often strings, you can send dictionaries or objects, too.

Some other, less known bits from the logging module include LogRecord and LoggerAdapter. The LogRecord class is the actual object that holds all of the pertinent log data. The class comes with a number of parameters that are already preconfigured that the standard logging module recognizes. However, you can also modify the base log record factory or create a new one to use in addition to the base one to create more attributes for the log record to hold. That's really important for structured logging because good structured logs tell the system exactly what it needs to know without being too large of an object to parse. So rather than having extraneous data that you might not need in one app versus another, you can target the data that you actually want that will be helpful. Reducing noise while providing actionable information is always helpful. The LoggerAdapter, on the other hand, can help you add context to your logs, which means you can send stuff like network data or other details that your logger might not necessarily collect on its own. I think the basic idea is that you would use an adapter rather than a filter in cases where you might need to send data once with a specific log line in a specific section of your code so you don't have to write a whole new filter just for that tiny use case.

Now, the workshop I'm working on is about structured logging. I can certainly get into why we should use structured logs, but I'd rather stick to the deep dive on Python specifically here. In reality, yes, you can implement structured logs yourself in Python, and the Python maintainers have added the how-to to their documentation as a cookbook. There are also libraries now that can help you implement structured logs. An example is structlog, a Python library. As always with Python, loggers, handlers, filters, formatters, log records, and adapters are all objects. That means they all have attributes and methods. You can modify any object in the pipeline to match your specific need, including to add structured log formats. I've been playing with structlog, and it seems like a simple plug-and-play setup if you don't feel like coding the necessary bits yourself. However, I'm of the camp where you build it yourself if it's simple, mainly because of the leftpad controversy from the NPM/Node.JS world a while ago. It's really up to you to see just how complicated of a system you need for your logs.

Interested in learning what I've found in general about logging more useful data and give it a shot yourself? You can join me at DeveloperWeek Austin 2019 for my workshop if you're in the area, or you can get a sneak preview if you're early or just enjoy the content on the workshop repo. I'll likely be working on it right up until the end of October 2019, and then probably tweaking it based on feedback, so enjoy! Feel free to also join the conversation on my Twitter account.

Pelican and GitHub Pages

2019-09-28T00:00:00-05:00

I thought I'd throw together a quick post on using Pelican with GitHub Pages and GitHub Actions.

I decided to use a Python-based static-site generator instead of Jekyll because (a) I love Python and (2) Ruby and I aren't always on the best of terms. In addition, I wanted a system that I could really dig into and throw plugins together whenever I wanted. In another job, I managed a custom Jekyll system with plugins and hooks that I built myself, but I know doing that sort of thing in Ruby would take me a lot longer than just knocking it together in Python. Perhaps in the future I'll build the site with a different system depending on how complex it gets, but a simple static-site generator really has a lot of appeal for something this simple. I chose Pelican mainly because I wanted to figure out how it worked; nothing Earth-shattering there.

The biggest challenge I had was automating publication of the site to GitHub Pages. I wanted to be able to write a post on my phone on a plane and be able to publish that post from my phone without the ability to build the site locally. I certainly could have done that with a more traditional content management system (CMS) like Ghost or (shudder) Wordpress. However, there was an appealing challenge in making it work, and I admit to jumping at the chance to try GitHub Actions. So I started getting the site in order locally, and then I joined the beta for Actions to get started.

First, I needed to trigger a build when a push goes the right branch. I decided to have a different host branch for the source code of the site since user pages on GitHub require publishing from master in a specially named repo. So my workflow had to trigger from a different branch:

on:
  push:
    branches:
      - source

I decided on an Ubuntu image to build on as I run Ubuntu at home, and therefore the build should be fairly close to my local builder. Everything from here on are job steps.

Next, I had to check out that specific branch. I borrowed an Action to make that happen:

    - uses: actions/checkout@v1

Then, the workflow needed to set up the overall environment:

    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

Next, a make command! The Makefile ensures that I don't have to remember all of my options I run, and that means a more consistent build when I do run it on my local system.

    - name: Build the prod site
      run: |
        make publish

This next bit is really, really important for publishing to GitHub Pages, and it's a bit silly of a step, really. You need to stop Jekyll from trying to parse your new raw HTML files and failing. The way to do that on GitHub Pages is an empty file named .nojekyll. So, a quick touch command to the designated output directory for the publish build before moving it to the master branch, and we're ready for publication:

    - name: Add nojekyll
      run: |
        touch ./output/.nojekyll

Finally, deployment time! We just need to send the raw HTML from the output directory to the master branch. However, I had to find a good Action to do it because I have a custom domain. All of the other Actions I tried actually broke my custom domain, so this one actually adds the custom domain designation every time. The creator (see the uses value) made some clear docs around the Action, too, which makes me very happy.

    - name: Deploy to GitHub Pages
      uses: JamesIves/github-pages-deploy-action@master
      if: success()
      env:
        ACCESS_TOKEN: ${{ secrets.GH_PAT }}
        BASE_BRANCH: source # The branch the action should deploy from.
        BRANCH: master # The branch the action should deploy to.
        CNAME: nimbinatus.com
        FOLDER: output # The folder the action should deploy.

So there you have it. I just push changes like this post to my source branch, and GitHub Actions publishes the whole thing for me. And all on the back of Python, which makes me happy.

Check out the full workflow in my repo. Questions? Feel free to reach out to me on Twitter.

Complexity of Logging

2019-09-25T00:00:00-05:00

I started here at LogDNA a few weeks ago, and it's funny how something as "simple" as logging can have a large load of complexity the farther down the rabbit hole you go. Now, I already had an appreciation for good, actionable logs coming in. I'm actually a big fan of the 12Factor App methodology, and good logging generated by previous developers have been lifelines for me when trying to understand and debug legacy systems. Sure, devs know, if they need to create an application log, they can pull in a logging library or use a built-in logging solution from their favorite language or the language-du-jour, pass a couple strings or an array, and then move on with their work. And sysadmins pour through selected parts of aggregated logs on a daily basis to monitor systems, troubleshoot servers, and otherwise understand the health of an environment. However, have you ever thought—seriously thought—about logs and logging systems and what makes the whole thing tick?

Logs may seem like a simple thing. The log files really are just simple sequences with data, the log line, being added at the end every time some event triggers the write mechanism. The process of writing to this append-only file generates an inherent timeline in the log file structure. Technically, the log line doesn't have to include a timestamp as a result. We include timestamps for each line, though, because we as humans think in terms of our calendars and clocks. We need time, whether relative or absolute, so we can correlate to external disruptions or changes and because the advent of asynchronous streams and concurrent, distributed systems means our computer systems are no longer bound by a linear path of function following function.

As a result of having distributed systems, though, we often have multiple log files from many different systems, and we have to manage all of them. At the top of the stack are our application logs, the ones that most devs are familiar with and that are generated by the application itself. At the bottom are the system logs, which come from the operating system or the BIOS underneath the operating system. Everywhere in between, from your networking calls to changes in permissions, are different logs like event logs, audit logs, and transactional logs. All of those logs follow the same basic principle of data added to an append-only file, and all of them are built around the small bit of data called the log line.

There's an art to writing a good, solid, actionable log line. Standards and ideals have changed over the years, too. Logs have evolved from a straight I/O buffer to strings with parsable data to tokenized arrays with human-readable messages as one of many datapoints. Those arrays of a single event can include a metric ton of data stored as callable keys with associated values that can then be parsed by a machine. We've collectively learned to generate logs in such a way that a separate system can aggregate the data, identify patterns and anomalies, determine the difference between a small blip and a fatal programmatic error, raise human-readable alerts—all in the time it might take for you to take a sip of your morning coffee (or tea, if you're a tea drinker like me).

We've added the concept of logging levels to the basic log line to make those massive generated logs parsible. We can choose to see only the most critical of logs: The fatal system error. We can dig into details by turning on debug levels. And, of course, we've created multiple standards, with the next great "unifying" standard always on the horizon as a real-life xkcd comic. It's human nature, after all, to want to create a unifying pattern and then disagree on exactly how to bring that pattern to life. In general, though, we collectively decided that having log levels for each line is a Good Idea™.

At their heart, though, log lines are the announcements of changes in state of your system. A system's state turned from off to on. So-and-so's state became logged in. This server turned from healthy to unhealthy. The choices around what to log is a constant source of debate among engineering teams. Some people think everything should be logged, just tagged carefully with the right log level. Others think that logs should be carefully curated to generate only the most useful, most critical information. I find the divide often follows the same arguments around commenting in code. However, really, the choice of what to log revolves around which state changes you need to monitor, and despite a lot of advice out there, only the creator, maintainer, and user would know which state changes are the most important and needed for decisive action.

All of this discussion about the complexity around logs and log lines doesn't even take into account the complexity of managing those logs. As a dev advocate at a log aggregation company, I'm getting a real look at the underbelly of the beast. The sheer amount of data that we take in on a regular basis is staggering. When the phrase "petabytes of data daily" makes your engineers shrug in a world where the majority of us are still thinking in gigabytes and terabytes, then you really are dealing with a mind-bendingly large amount of data. Once I've conquered my initial wow factor a bit, we can talk more about the complexities of log aggregation and management itself, especially with distributed systems and microarchitectures.

What are your thoughts about the complexity of logging? Join the conversation on Twitter.