One Mo' Gin

Introducing Kapell, a music player

Sat, 01 Aug 2026 07:51:00 +0000

Holy shit it’s been 6 years since I wrote a blog post here. [raps microphone] is this thing on?

I really love music and to that end I’ve developed Kapell, which is an Apple Music player that connects to MusicBrainz, Wikidata, and more. It aims to improve active listening, giving you a way to learn while you listen.

I wrote about using Kapell to listen to Rolling Stone’s 500 Best Albums of All Time and I hope you’ll check it out!

Improving Automation: Working Together

Thu, 28 May 2020 07:51:00 +0000

I painted a bleak picture of runaway, people-eating automation in Stop Hitting Yourself: How Automation Is Hurting You. It was a bit alarmist, but for good reason. In my experience at many companies and with many customers, automation is treated as a cure-all that is rarely designed to work with human teammates. Instead, it removes the human completely while providing little in the way of substitute. This post is meant to provide guidance on how to overcome this challenge and improve your operations by giving automation some extra capabilities.

This post is part of series on automation, the result of many months of reserach and reading. I may adjust these posts as my research grows. If you've got comments leave feedback at the bottom!

Thanks to Arijit Mukherji, Franklin Hu, Jay Shirley, Rajesh Raman, and Sam Boyer for their feedback and reviews of these posts.

How To Improve

To be a good team, the parties on that team must agree to a basic compact wherein they coordinate, work toward shared goals, and prevent breakdown in communications.¹ The rest of this post is concerned with doing our best with current technology to make automation a part of this deal.

As such, in this post I will be speaking about automation anthropomorphically as part of a team. This might feel a bit funny, but doing so encourages us to make it behave less… assholey. If we start thinking of these systems as teammates who can improve with feedback, we can begin to change our approaches to one where our humans and our robots work together instead of in isolation.

Automation Exists In A Team

Automation often comes from a place of frustration or failure, like an incident remediation. As such it is easy for the author to focus on the outcomes rather than the context in which the automation operates. A form of incident-driven-development, if you will.

To build automation that works well with humans — or even other automation — we need to consider these techniques at the time of design. Many of the improvements suggested will require changes to the automation’s approaches, algorithms, and capabilities. Let’s discuss how.

Automation Must Be Designed

The point of automation is to free the human from some effort. Unless you work alone, other humans will become responsible for working with and caring for the automation in the future.² To that end you must treat automation the way you’d treat a product. That means talking to stakeholders, getting feedback, and considering ergonomics.³

An appropriate design should assume the existence of error, it should continually provide feedback, it continually interact with operators in an appropriate way, and it should have a design appropriate for the worst of situations.⁴

We must also consider the cognitive load of the automation. Is it hard for people to use? If the complexity exceeds the perceived benefit they may avoid its use all together.⁵ Automation that is bypassed is both wasted effort and a potential future problem.

Automation Must Communicate

Above all, automation must communicate with its human partners to be effective. Consider your automation as another person on the team. What might they say to their teammates as they work? This can improve the gulf of evaluation³. How you do this will depend on your internal capabilities, but here are some suggestions:

Emit metrics that track currently monitored values (gauges) and critical operations (timers and error counters).
Track estimates and error. What did the automation expect versus what actually occurred? Consider some lessons from control theory and PID controllers. Emit telemetry about these measurements as well as judgements.
Emit metrics that make clear any goals, thresholds, or limits. These may be fixed values, or values that change with the state machine.
Emit events around actions, intentions, and decisions.

This feedback is vital for humans in their work and performance.⁶ Build interfaces or dashboards that curate this information, specifically aligned with the goals — e.g. control, stability, cost, latency - of the operator and the automation.⁷

As a final note, remember to allow access to the raw information (logs, metrics, events) so the operator can dig in deeply if needed. This is important, because automation merely removes the need for the human to focus on the tasks in question. When they need to take over for automation it is essential that raw information be available to reestablish control, see also Ecological interface design.

Automation Must Be Predictable

It is generally advisable to avoid surprise in software. Humans will inevitably forget stuff and work with imperfect mental models.⁸ This is a normal form of energy conservation for humans who can’t possibly remember everything at all times. Maybe they are new and don’t even know the automation! As such we must work to ensure that our automation considers its teammates and acts predictably. (Remember our toxic teammate example!)

Here are some ways to improve predictability:

Emit notifications and reasons when goals change, errors occur, or any time when a human might need to be aware of an action.³
Emit information about the current, past, or next step in the process.
Avoid sudden movements. Aside from being surprising to humans, smaller adjustments at faster intervals are typically better at stabilizing systems.⁹
Add (and emitting notifications for) cool down or observational periods such that the automation can transmit its intent to act in time for a human to react.
Increase urgency of communication as the bounds of control get closer. Do not snap from normal to freaking out in one tick of a loop, do so gradually. This might signal a need for a tighter feedback loop or lower interval!

As an aside, another way to improve predictability is to keep things simple. Some ML or AI features promise great benefits, but may be opaque and unpredictable to operators. Take care when evaluating and using such features.

Automation Must Be Directable

So far we’ve focused on ways that automation can transmit information about its work. Now we must turn the tables and think about how our automation can receive information and adjust its behavior.

Add a kill switch or other mechanism to quickly disable automation and make it accessible during incident response. Train people on its use.
Add APIs to your automation that can enable, disable, or influence it’s parameters, like feature flags.
Configure “policies” that adjust the approach or the aggressiveness of actions, with ways to change this easily at runtime.
Messages, APIs, or signaling to sense other teammates actions — i.e. other automation or humans — and to react accordingly. If a human intervenes, maybe I should sense this and adjust my actions.

These features allow teammates to direct the automation in a way consistent with the given context.¹ With these capabilities, working with automation can become a part of the documented process for responding to loss of control. Without them, automation may be forgotten and exacerbate the problem through well meaning but opaque activity.

Automation Must Be Respectful

We’ve discussed at length how vital it is for automation to make information about its work available. We must also, however, balance all that yelling with a respect for the other agents. Humans typically rely on mental models of others to determine what, if anything, needs to be said to keep coordination going.¹ Our automation, being of simple mind, needs our help. Here are some ways we can improve things:

Use log levels or other filterable tagging to indicate matters of different importance. Normal operations are no big deal and should be kept to low priority of communication whereas nearing the bounds of control warrants yelling loudly.
Leverage consolidation and filtering tools that allow dialing up or down the attention given.
Accept feedback that mutes notifying users, either at the source or through a common filter.
Incorporating context, such as if other teammates are acting, to help guide internal decisions about how much and often to direct attention. Are we in incident?

In practical terms automation needs to be aware of the state of the world around it. If the humans are stressed out automation should aim to be helpful and not overwhelming. This could begin as a human controlled flag or signal, but over time could be adjusted to track critical health metrics of the system.

Automation Must Evolve

We commonly celebrate a new bit of automation being turned on. We’re done with the toil and frustration of doing this manually. No more human error for us, no sir! It might be appropriate to celebrate the end of one bit of work, but now it’s time to shift over to a whole new effort: the care and feeding of automation.

Being a part of a complex system means that the automation will need to adjust over time. Working with humans will also mean learning from errors, sanding down rough edges, and generally improving understanding for the operator.³

Ensure that you’ve allocated time for this work, and that you revisit your automation. Think of it like a quarterly review. It’s a chance to evaluate how the automation has met the organization’s expectations and communicate what the automation can do to improve. Since it’s not as adaptable as you, any improvements will come from your own keyboard.

Too often we pave over this critical automation, tossing it aside as soon as it serves its purposes. This forces our future teammates — maybe even ourselves — to act like technological archaeologists, digging up old bits of automation and struggling to decipher what they are doing.

Revisiting The Autoscaler

When we last left our autoscaling example from Stop Hitting Yourself: How Automation Is Hurting You we hadn’t we read all this rad stuff about how to improve automation. Let’s count off some ways that the autoscaler can be improved to fit the techniques described (less design, since that ship has sailed):

Communicate: Emit metrics for the current, target, and expected values. Measure durations of any adjustments (calls to scaling APIs, etc). Track difference between expected and current values with consideration to duration and emit this as an error value.
Predictable: Emit log lines for all decisions and any problems. Emit log lines for potential next steps. Adjust frequency of checks and sizes of adjustment to smaller amounts, with backoff and/or cooldowns before actually making changes to scale.
Directable: Add feature flags to disable completely or adjust aggressiveness of changes. Add API calls or feature flags to adjust parameters on the fly. Monitor health KPIs over affected services and bail out (with messaging!) if things look fishy.
Respect: Ensure that logging uses appropriate log levels. Monitor for changes to current values that did not correspond to automation’s changes and consider bailing out or at least logging. Don’t page or otherwise bother people directly.
Evolve: Provide a convenient way for users to give feedback on tooling, preferably one that doesn’t require 11 different required JIRA fields or anything to be filed in triplicate. Periodically review performance metrics and track any tickets generated for improvement. Hell, just ask them.

Future And Summary

At this point, I hope you’re glancing around at the automation that holds together your systems with a combination of fear and excitement. Fear because there many footguns and time bombs in your midst. Excitement because the above techniques can defuse and improve automation!

There are additional improvements we could imagine, like making automation aware of how humans work through modeling and allowing that automation to predict and adjust it’s behavior. Sadly, this and others are still a bit science fiction. The steps above, however, are all practical with today’s technology and a bit of forethought!

By incorporating these concerns into our design and maintenance of automation, we can not only create a more harmonious environment, we can look forward to a less toxic, more helpful teammate. A teammate that keeps us informed of what’s going on, and backs off when we take over.

I encourage you to check out the citations listed in this and the earlier posts on automation. My suggestions are just that, and are largely viewed through the lens of an observability wonk. I’m excited to hear what ideas you might have for improving automation.

This post is part of series on automation, the result of many months of reserach and reading. I may adjust these posts as my research grows. If you've got comments leave feedback at the bottom!

Thanks to Arijit Mukherji, Franklin Hu, Jay Shirley, Rajesh Raman, and Sam Boyer for their feedback and reviews of these posts.

References

Klein, Woods, Bradshaw, Hoffman, Feltovich, Ten Challenges for Making Automation A Team Player ↩ ↩² ↩³
E. Hollnagel, The Role of Automation in Joint Cognitive Systems ↩
Donald A. Norman, The Design of Everyday Things ↩ ↩² ↩³ ↩⁴
Donald A. Norman, The Problem of Automation: Inapporpriate Feedback and Interaction, Not “Overautomation” ↩
Raja Parasuraman, Humans and Automation: Use, Misuse, Disuse, Abuse ↩
Woods, Cook, Nine Steps to Move Forward From Error ↩
Mica Endsley, Automation and Situation Awareness ↩
Besnard, Greathead, When mental models go wrong. Co-occurrences in dynamic, critical systems ↩
Philipp K. Janert, Feedback Control for Computer Systems ↩

Alternatives To Automation: Amplification, Complexity, and Control

Wed, 27 May 2020 07:49:00 +0000

In Stop Hitting Yourself: How Automation Is Hurting You I talked a lot of shit about automation. What can we do to avoid toil and error if we cannot automate? Well, we can automate but that’s coming later. First, let’s discuss the middle ground.

This post is part of series on automation, the result of many months of reserach and reading. I may adjust these posts as my research grows. If you've got comments leave feedback at the bottom!

Thanks to Arijit Mukherji, Franklin Hu, Jay Shirley, Rajesh Raman, and Sam Boyer for their feedback and reviews of these posts.

What Is Complexity?

I have a lot to learn about complexity. I’m still researching, but I really enjoy David D. Woods’ take in Coping With Complexity, summarized by me:

Complexity isn’t a thing, it is a situation composed of a world, an agent, and a representation. The world is dynamic, interconnected, uncertain, and costly. The agent is one or more humans and… other stuff like computers. The representation is whatever those agents use to understand and manipulate the world.¹

Complexity can be expensive. It makes things slow, error-prone, and frustrating. My primary interest is in complexity’s impact on the operator.

As the complexity of a system increases, the accuracy of any single agent’s own model of that system decreases rapidy.²

The Goal Is Control

In many of the citations I’ve used in this series the discussion is usually about control. The systems we operate hum along, doing their thing, within a desired performance envelope. This envelope is defined by what our customers are willing to accept, what we’re willing to pay, and how well the operation works.

Now and then things go wrong and, like a plane blown off course, we steer, throttle up, or however a plane works — I am not a pilot — to get things pointed in the right direction.

Amplifying Human Ability

Sticking to airplanes, the human pilot exerts control by operating the various control surfaces on the wings and such. Apparently, I’m still not a pilot. Originally these surfaces were operated with cables and pulleys and required raw human strength. As airplanes got faster, more strength was needed and hydraulics amplified human strength. Some modern planes use fly-by-wire to replace all those heavy hydraulics with wires and actuators. Regardless, the human is still in control.³ This is just one example of how humans have engineered increased capabilities to exert control. But it’s a cool one because planes are rad.

Before deciding to remove the human from the loop you should consider amplifying the human’s effort. Take Terraform or Puppet as examples. Both allow a single user to define a desired outcome and affect change across tens of thousands of machines in a fraction of the time.⁴ Critically these and other tools of their shape keep the human “in the loop”: deciding on action, monitoring progress, and responding to error. They can still cause incidents — believe me, I’ve done it plenty — but the human is more likely to be present, aware, and up to speed.

One way to deal with complexity is not to add it. The gist of my argument in Stop Hitting Yourself: How Automation Is Hurting You is that creating automation adds complexity.⁵ Before we add this complexity we could instead amplify the human’s abilities of strength, perception, or attention.

Autoscaler Example

In Stop Hitting Yourself: How Automation Is Hurting You we talked about automating an autoscaler. Before removing the human, what if we tried making a small tool that speeds up, simplifies, or otherwise improves this process? Maybe a small command line tool with best practices as defaults that allows multiple systems to be scaled all at once?

If we had done this we could keep the human involved. We could amplify their effort as well as coding in a few smart defaults. Maybe the human can even submit patches or, at a minimum, we can interview them to get some feedback to improve further.

You may recall the Law of Stretched Systems⁶ from the last post. The additional time gained by leveraging our tool will undoubtedly result in our humans doing more things. The difference is that we’ve not cast the work aside for our humans to forget about. We’ve merely made it easier. Furthermore we’re good on the Law of Requisite Variety because our tool can continue to rely on the human to cover all of the things we’ve left out. The tool is a supplement to an adaptive human.

Tools Can Be Complex Too

The creation of such a tool will obviously require some investment. It will require consideration about best practices, repercussions and outcomes. The adoption of any technology — tool or automation — fundamentally changes the job, the problems, and the future of the operation.⁷

The difference I am advocating here is to include the human in the use of the tool, versus automation which replaces the human. When we remove the human we remove the only adaptive, safety-creating⁸ component and replace it, if at all, with a poorly coded analog. When we amplify, we retain the best human parts.

Amplify Before You Automate

There are other ways to deal with complexity such as increasing reliability to the point that it’s mostly ignored, or embracing complexity’s representation. Those are topics for another day, though.

Amplification via tools allows us to increase output while decreasing, or at least keeping constant, human effort. We retain the best traits of humans while decreasing effort or toil.

This post is part of series on automation, the result of many months of reserach and reading. I may adjust these posts as my research grows. If you've got comments leave feedback at the bottom!

Thanks to Arijit Mukherji, Franklin Hu, Jay Shirley, Rajesh Raman, and Sam Boyer for their feedback and reviews of these posts.

References

David D. Woods, Coping with complexity, the psychology of human behavior in complex systems. ↩
Richard I. Cook, M.D., Above the Line, Below the Line ↩
Yes, there are autopilots which are automation but that’s supplemental and the subject of many more papers. ↩
Both also have their faults, but they obviously allow people to control huge fleets of machines. ↩
Lisanne Bainbridge, Ironies of Automation ↩
Woods, Hollnagel, “Joint Cognitive Systems: Patterns in Cognitive Systems Engineering” ↩
Woods, Dekker, Anticipating the effects of technological change: A new era of dynamics for human factors ↩
Richard Cook, How Complex Systems Fail ↩

Stop Hitting Yourself: How Automation Is Hurting You

Tue, 26 May 2020 07:50:00 +0000

I really dislike writing post openings. It feels tedious to define the problem in leading ways that will entice a reader, especially when the important bits are further down. I wish I could just press a button and get a Cory-like opening blurb with some toilsome bits like “so and so is defined by wikipedia as” and a pithy joke. I wanna automate it.

This post is part of series on automation, the result of many months of reserach and reading. I may adjust these posts as my research grows. If you've got comments leave feedback at the bottom!

Thanks to Arijit Mukherji, Franklin Hu, Jay Shirley, Rajesh Raman, and Sam Boyer for their feedback and reviews of these posts.

Automation is a process being performed with minimal, if any, human involvement. DevOps and SRE commonly recommend aggressive automation to do more with less in modern, complex systems. There’s certainly a lot to be gained from automation, but there’s also a downside when this technology replaces human involvement.

Toxic Teammates

Have you ever worked with a stubborn, uncommunicative teammate? Maybe they didn’t contribute to joint projects or did contribute but didn’t share until the end? Perhaps they insisted on dropping everything at exactly noon every day so they could get their favorite sandwich for lunch, work be damned, and left everyone else to deal with the mess.

Such a teammate is often considered toxic and ruinous to teams everywhere. My goal today is to show you how the automation you’re making in your job is generating these “teammates” and wrecking your happiness, reliability and ability to do cool new things.

Toil and Trouble

One of my responsibilities is keeping an API library up to date. Inside this API is a list of acceptable names for cloud provider services. That list changes periodically and is kept in a bit of Java source code as a map. So every few weeks I copy the block from Java into Go and — by hand — rejigger it into Go syntax and commit it. Each time I think about how I could write a parser to do this… or maybe ask the other team to move the definitions to YAML files so I could rid myself of this toilsome work. Sometimes I even typo a definition and release it to the world. “Damn!” I say, reminding myself that if this were YAML and I had a script I could avoid this happening.

This sort of tedium is common in jobs. Despite my whining, I’ve only had problems with this process two or three times in a year. Each time it was obvious to me what had happened and it was fixed quickly and effectively with very little customer impact. This is because I — as a human — am full of wonder: foresight, imagination, adaptability, and ingenious. While I make errors on some occasions, I more often create safety.¹ When I do make errors, I quickly recognize and correct them using my adaptability and ingenuity. Compare this to the heap of new problems I might bring by introducing new dependencies to a well understood process.

How Automation Hurts

The point I’m getting to here is that despite our seemingly genetic distaste for toil we should be extremely careful about reaching for automation.

Automation has generally been introduced to meet the needs of the process rather than the needs of the people working with the process²

My plan is to scare you sober by showing all ways that automation is like that toxic teammate. In a follow up I’ll help you learn how to avoid these problems.

Automating a process without proper consideration, design, and planning can create technical debt, incidents, and undermine all your hard work.

For the rest of this post, imagine a common bit of automation in many orgs: autoscaling. Some sort of latency metric is monitored and, based on thresholds, some other resource like compute is scaled up or down. This replaces a human looking at charts and making a judgement call, or missing an unexpected surge because they were busy watching Tiger King and eating half-gallons of ice cream. Not that I’ve done that.

Imagine our new autoscaling system has worked well for weeks, then a surge in traffic occurs. The autoscaler automation does its job and no human needs to get involved! Sadly, this load consumes all the compute your cloud provider has allotted to you. Tasks across the org begin failing with esoteric error messages as compute grinds to a halt. Since most of us skip error checking for operations that generally succeed we may not even have error messages!

“But Cory”, you say with a smirk, “we have tons of automated things underlying our entire lives and we’re getting by”. Sure, we are. The key difference is definedness. There are some processes which are so well defined, and so unlikely to encounter problems that we’ve been able to free ourselves completely from the toil. For more complex or poorly defined situations, however, human capabilities are still essential.²

Automation Requires More Of Humans

Automation removes the human from involvement in the operation. This is a blessing in reduced fatigue or improved productivity. Unfortunately it’s a curse in situation awareness.

These kinds of second order effects are common with automation because humans are “out of the loop”. This means additional time is required for all these folks to get acquainted with the system, how it works, and what can be done about it.³ If the user doesn’t know about the automation or has forgotten the logic, they may end up fighting what seems like unexplainable behavior! This price is being paid at a shitty time, as we may be dealing with customer side effects and blowing up the entire org’s productivity by invoking the incident machinery.

This situation leaves us with an irony and a paradox.

Irony: the more complex an automation, the more crucial the human becomes.⁴ Our autoscaler, meant to improve latency, has instead caused a complex series of second order failures that a — or many! — humans most now sort through.

Paradox: our automation was intended to remove the need for humans, but instead we’ve made a new, different joint human-computer doodad.⁴

Automation Creates New Problems

When we set out to make our autoscaler or any other automation, our goal was to reduce the effort and/or accuracy of a task. This goal is so strong that we generally miss, or don’t bother to imagine, the side effects that come with the benefits. Adoption of any technology, which automation is a form of, increases needs for coordination, creates new situations, and new failures. “It changes what is canonical and what is exceptional.”⁵

Consider our earlier example. Not only do we have new failure modes, we have additional process and state that humans must internalize. Before the autoscaler we had one set of problems, now we have exciting new problems!

Automation Increases Complexity

The productivity gain from automation is tantalizingly quantifiable. The ramifications are frustratingly qualitative. The time spent staring at charts, editing files, and executing changes can be added up in a spreadsheet and celebrated at review time. What will we do with all this extra time?

We create even more complexity, that’s what. We’ll go and automate another thing, or create a new thing that needs automation later. This is a form of The Law of Stretched Systems⁶:

Every system is stretched to operate at its capacity; as soon as there is some form of improvement, for example, in the form of new technology, it will be exploited to achieve a new intensity and tempo of activity.

We’ll take this newfound free time and permission to make more complexity without realizing we’re going to pay later.

Harkening back to our earlier problems we can also look to cybernetics for the Law of Requisite Variety which warns us that a controller — which is what our automation is — must have at least as many states as the system it controls. This culminates in a combo finisher by Brian Kernighan’s famous admonition that debugging is twice as hard as programming. Can you debug this automation if it’s more complex than its target process? What about when multiple pieces of automation start interacting?

Automation Is Design

Automation usually begins from a point of frustration. Our autoscaler was likely born either from an incident remediation or someone who was sick of staring at charts. Our aforementioned quantitative improvements spur us into action. The autoscaler is only a state machine, right? You’ve written a zillion of those!

The repercussions of adding automation warrant research, user interviews, collecting feedback, and all that other work that isn’t coding. I’m talking about design here.

Design is never neutral, so every change you affect or error you emit benefits from design.⁷ How will users know the autoscaler has taken action? Will the autoscaler make correct choices when faced with increasing latency and capacity for other functions? Can users disable the autoscaler? These are all essential questions to factor into your design.

Automation Reduces Optionality

Humans have created some beautiful — and some despicable — things. A human as part of a system means that system can still be adaptive¹. When we remove humans we remove this adaptive capability. Yeah, yeah, I know we have ML and AI but these are, for now, very crude in comparison to humans. Using them is, in effect, even more automation that we must understand. Eep!

To automate a process requires a very specific, fixed set of instructions. Do you understand the process and its ramifications well enough to do that yet? Automating a process requires design and choices, which can reduce the freedom of afforded from continued learning, evolution, and adaptation from the human operator.⁵ Doing this too early can result in shortcomings, bugs, and technical debt.

A human is aware of seasonal differences, like Black Friday, where a human would temper their actions. A human would recognize a network outage and not scale the compute down to 0 when the latency metric is 0 or missing. These lessons must be learned before we can rely on automation, lest we realize the repercussions in embarrassing incidents.

Automation Is Brittle And Dangerous

You’re still here reading, so you didn’t bail early. Those that did probably think this post is some sort of Luddite position that we should stop or cast off automation. I’m ok with that assessment if it slows engineers down and encourages them to think through when and how to automate something. By all means read this half way through and talk shit about it, so long as it scares you.

Really, the opposite is true. I’m in awe of our automated accomplishments. My issue is with the wreckage we leave in our wake in the form of half-ass resilience. Automation is incredibly powerful, but so is human capability. Deploying automation too soon can result in a rickety, dangerous foundation that humans prop up with grueling on all schedules and unhappy customers.

The next time you feel the urge to automate, instead begin a design document. Better yet, marvel in your own antientropic powers and keep learning so you can write a better design document later.

This post is part of series on automation, the result of many months of reserach and reading. I may adjust these posts as my research grows. If you've got comments leave feedback at the bottom!

Thanks to Arijit Mukherji, Franklin Hu, Jay Shirley, Rajesh Raman, and Sam Boyer for their feedback and reviews of these posts.

References

Richard Cook, How Complex Systems Fail ↩ ↩²
E. Hollnagel, The Role of Automation in Joint Cognitive Systems ↩ ↩²
Mica Endsley, Automation and Situation Awareness ↩
Lisanne Bainbridge, Ironies of Automation ↩ ↩²
Thomas B. Sheridan, Allocating Functions Rationally ↩ ↩²
Woods, Hollnagel, “Joint Cognitive Systems: Patterns in Cognitive Systems Engineering” ↩
Donald A. Norman, “The Design of Everyday Things” ↩

Dashboard Organization and Naming

Tue, 20 Aug 2019 07:49:00 +0000

It starts innocently: a monolith with some simple KPIs to keep an eye on. You whip up a quick dashboard. The simplicity of your needs means that things are well organized and all in one place. Order is kept. Everyone knows where to look.

This post is part of A Practitioner's Guide to System Dashboard Design.

Fast forward a bit and as you’ve microed your services and kubered your netes the sheer fecundity of services, teams, operators, and needs have resulted in a hot mess of dashboards. Operators use bookmarks or address bar completion to find their trusted sources. Friends meet over lunch and realize they are looking at two different views of similar data. Shady vendors erupt from allies promising magic spells that point out all your problems. It’s so confusing!

via GIPHY

Let’s discuss how we can design our dashboard organization to increase awareness and findability.

A Very Particular Set of Skills

The folks making dashboards are often the folks running the systems they monitor. This is both important from a DevOps perspective as well as for accuracy. Unfortunately these folks — the skillful artisans of backend services, data pipelines, and machine learning — might not know best how to set up an entire company’s monitoring assets.

If you squint at dashboards and how they interact with your operations this can be reduced to a problem of organization. Luckily there is an entire discipline around this called information architecture or IA. Information architecture, a branch of information science, concerns itself with the design of information environments to support findability and usability. Yay for the -ilities!

Your first step is to ask around: does your company have folks that are experts in this discipline? If so, seek them out and ask for some of their time. They can likely help you through some basics on what to work toward. Use their advice over any you get here!

For those without you might find the U.S. Department of Health & Human Services’s Information Architecture Basics page helpful. Also a run through Fundamental Concepts of Good UX wouldn’t hurt.

Note: It may help you to explicitly appoint someone this responsibility and applaud their efforts rather than relying on unofficial or “shadow” work that goes unrewarded. Most tools have APIs for fetching assets and you can set up policies that enforce the ultimate design.

See What You’re Working With

Most companies looking to improve their dashboard findability will be using some sort of tool, be it vendor, OSS, or in-house. Regardless you’ve likely got some existing capabilities. What does the tool support? Common mechanisms are tags or labels, hierarchies, groups, or layouts like matrices. This is vital, as trying to fight against your tools preferred mechanisms will only yield frustration.

Empowered by knowing the way your tool works we can start looking at how to best leverage it’s strengths.

Ship Your Org Chart

Ever noticed that you tend to move toward things you’re looking at? Since organizations are full of people we tend to also go where the organization is looking. The trusty org chart is a strong indicator of how a company is orienting itself, and our products tend to reflect that. This idea is called shipping the org chart.

People tend to understand the product they work in as well as the org chart that shapes it. To that end, consider arranging your assets using the org chart as a guide. For example, you might have Product and Infrastructure Engineering teams. The Database team, nested in Infra, might have a team for Transactional versus Analytics.

Does your org chart have these fancy icons?

The folks that work in these areas, as well as those higher in the org chart probably know how to navigate this structure. You can leverage this hierarchy by placing appropriate dashboards at each level. High level KPIs for each team might live at the Infra level whereas a dashboard for use with Hadoop would live in the analytics team’s area.

Maybe these dashboards in these places?

Naming

I know you put a lot of thought into the clever names you’ve given your services and projects. Naming the data pipeline service for that Lord of the Rings character, complete with backstory, was really a stroke of genius. Unfortunately many teams use this sort of process and it quickly becomes impermeable claptrap.

Name your dashboards and monitoring assets for activities, not for nouns. These activities likely share some DNA with your org chart and team names. Furthermore most companies use similar words for things like orders, ETL, RPC. This allows new hires or folks from other disciplines a better chance of knowing what they are looking at.

Findability

Lastly let’s talk about how one finds these assets. You’ll again look to your tools for help. Do they offer search features? Are names the only searchable field, or can you also search descriptions or even chart titles within a dashboard? Knowing this will help you create a strategy for improving findability.

Use words that help

Data Pipeline Overview is a better name than Smeagol Overview, but Data Pipeline (Smeagol) Overview that lives in the Analytics team is much more helpful.

Add more words

Many tools have descriptions or other text fields beyond just the title. Consider something like “Contains high level indicators for data pipeline health useful for stakeholders interested in investigating delays and statuses”. This might help search tooling or just those that browse in and wonder the goals.

Linking, Lists and Ordering

Many tools allow you to create relationships between assets. A data pipeline dashboard might link to it’s destination or source, helping operators hop from one place to another. In hierarchical or list-based systems the order of dashboards might be a prominent UI feature. Consider placing more important or higher-level dashboards at the top.

Additionally, many dashboard tools allow placing text widgets. Consider including these with links to helpful resources.

Favorites or Usage-based Metrics

If your tool allows users to flag databases as favorites or upvote them, this might help bring attention. Be wary of this, as popularity might not match your IA design.

Keep It Up!

In case you missed it before I’ll reiterate: Appointing someone to do this work is a big help. Designs usually need designers. Your tools might have a framework out of the box, but helping the folks in your company know how best to use these tools, to create policities, and to enforce these policies can pay dividends. This function might be a part-time role for a skillful individual or a function of a larger team like an observability or SRE team in conjunction with specialists from your product arm. Regardless, this investment can prevent duplication of effort, improve responsiveness during problems, and generally level up your cross-team awareness. Have fun designing!

via GIPHY

Observability Crash Course

Wed, 07 Aug 2019 07:49:00 +0000

The title is succinct, but in practice an organization’s “observability” efforts range a number of disciplines. This document aims to compress the breadth of topics into a succinct (fitting on 1 printed page) set of best of breed write-ups that avoid any ties to vendors or implementation.

Bryan Cantrill’s Visualizing Distributed Systems with Statemaps: Leads with a master class in what observability is and how our brains, our systems, and our tools are important.
Cindy Sridharan’s Distributed Systems Observability: This is a free, 25-page eBook that does a great job summarizing the various tools and concepts available. You could also read Monitoring and Observability from Cindy’s blog for a shorter and less formal version of the same.
Google’s Site Reliability Engineering: This book has provided a lingua franca for discussions around the practices of reliability. It includes lots of great material like “Golden Signals”, error budgets, and more. You can read it online for free.
- Tom Wilkie’s RED Method is a focused explanation of the SRE book’s Golden Signals and tends to resonate well with people.
- Brendan Gregg’s USE Method is an exceptional framework for measuring resource consumption. It takes more effort, but has fantastic results.
Fred Hebert’s Operable Software careens from observability to user experience and reminds us why this all so important.
Specific guidance for some of the “pillars” of Observability:
- Prometheus’ Guide To Metric Types covers the basics of metrics and their Metric and Label Naming reminds us that our metrics are also an interface for our engineers, and how we can standardize.
- Charity Majors’ Logs Vs Structured Events describes how we can turn logging from a burden into a blessing.
On the practice of measuring and using this tooling for the day-to-day:
- Coda Hale’s Metrics, Metrics, Everywhere touches on everything from mental models to OODA loops and generally explains how to measure and why you should. It’s 7 years old and low quality, but is the best summary I’ve ever heard.
- Baron Schwartz’ What Metrics Should I Monitor helps to frame what to pay attention to in systems. It’s aimed at MySQL but can be applied to other systems.
- Kavya Joshi’s Applied Performance Theory is an excellent talk who’s title couldn’t be more apt. It gives quick and practical advice on using many of the formal topics from performance engineering.
- John Allspaw’s Owning Attention (Considerations for Alert Design) is a class in alerting for humans.
On incidents:
- Gremlin’s How to Establish a High Severity Incident Management Program provides a good example of how to think about and handle incidents.
- The STELLA Report is the findings from the review of a few incidents and how engineers cope with them. It provides some strong food for thought for organizations and is a good gateway drug into the work of

Periodic Reading

Enjoy mailing lists and such? Here are some good ones:

Monitoring Weekly is exactly what it sounds like.
Thai Wood’s Resilience Roundup summarizes papers in the resilience space and adds special insight from his combined tech and EMT background.
Lex Neva’s SRE Weekly frequently hits topics in or adjacent to observability.

My Contributions

As a long time advocate of observability I hope it’s ok to add a few bits of my own. First, my definition:

Observability is a quality of software, services, platforms, or products that allows operators to understand how systems are working. Observability makes investigating and diagnosing problems easier; the more observable a system, the more tools we’ve made available to diagnose problems or understand behavior.

And some of my works:

My talks on observability across the years.
Structure and Layout in System Dashboard Design aims to condense the work of many other sources into practical advice for how to make great dashboards.
The CASE Method: Better Monitoring for Humans aims to give a vendor agnostic, manual-if-needed process for controlling alert fatigue and measuring value.

Honorable Mention

I’ve not read all of these yet, but seen them referenced enough to think they are worth a mention.

Seeking SRE is a supplement to Google’s SRE book, aimed at how the SRE role can be applied to organizations that aren’t Google.

Context Improvement in System Dashboard Design

Wed, 17 Apr 2019 07:49:00 +0000

With a strong layout and helpful visualizations our next goal is to orient the user with as much context as possible. Failing to provide any context is one of the most common mistakes in creating dashboards (Few 38)¹. The context our user needs often comes from other sources not included in our carefully considered charts.

This post is part of A Practitioner's Guide to System Dashboard Design.

Basics: Text

Humans are pretty good at reading. Those that make the dashboards often lean toward terseness. In part 2 we covered Scale and Units. Labeling our data and considering the form of that label can yield comprehension improvements.

But let’s not get too deep here: just label your damn charts better. There’s no charge for letters! Imagine you’re a new operator who’s just been paged for the first time. What do you need to know?

The left chart has no effort. The right side has reasonable labels to help the user understand the context of the visualization.

Basics: Next Steps

If a user wants to know more about this visualization, how can they get there? A common pattern is clicking on the title to open a fullscreen version that may allow further slicing and dicing. This a convenient place to put data tables or other helpful information:

Opening the chart in full screen gives us new tools since the user has demonstrated an interest in further digging.

Seeing a bad thing in a chart might lead to another tool or to notifying teammates. Why not embed these into a menu on the chart?

A menu could take you to an exception catcher, Slack sharing, or beginning the incident process.

Human Actions, Control Data

In part 3, Line Charts we discussed the value of time as our X axis in run charts. This is extremely helpful for readers in that it shows change over time, but the passage of time isn’t the causal factor in our data. For that we need more context.

Lots of thing can cause change. Pesky ghosts!

Tufte suggests improvement by “smuggling additional variables into the graphic design” (38) to increase the causal explanations for the reader.² This intersects neatly with system dashboards because changes from humans or automated systems are often the progenitors of system effects. Capturing these events and visualizing them provides our user with crucial insights into what’s being done that may correlate with system behavior.

The blue diamond shows that my deploy correlates strongly with a decrease in latency. Yay!

This control plane data is much lower rate than the signals we measure in our systems. Ensuring that such changes are instrumented and available in the same place as your dashboards should be a high priority. It can significantly improve how quickly our users can accomplish their goals.

Instrumentation: Here Be Dragons

Unfortunately most tools for system dashboards have spotty support for this data, often called “annotations”. The events tend to be hard to reuse, untyped, and often assumed instantaneous. Some packages support events with a start and end timestamp for deploys and other “stepped” changes.

Furthermore, getting annotations to your dashboards is spotty and often requires hand instrumenting dozens of disparate systems. Then you must find a way to connect the relevant events into dashboards. Eek!

Some advice on how to implement this well:

Leverage common outputs like CloudTrail Events and CI/CD systems.
Try and reuse events rather than having users mint their own that only work in their dashboards.
Be careful of clutter, too many events is as bad as none!
Provide a link in the event for your user to follow to find more information. This provides the user context but avoids turning your dashboarding tool into an administrative interface. Less is more if they can click on it.

Norms, Goals, and Agreements

Some KPIs have acceptable ranges, internal goals, or even contractual obligations. Including this information in the relevant charts can be a part of the visualization (see Gauges, Bullets, Ranges, etc in Part 3) or included as context within the widget. This context helps the user avoid the requirement of what the chart “should” look like, or what impact the current value may have on stakeholders.

Many tools support this type of watermarking. Adding them provides additional context to the reader so that the time series isn’t just an arbitrary value, but a comparison to our expectations.

The SLO for our service is included as a watermark in the chart.

We can extend this further with time shifts, moving averages, or forms of anomaly detection to draw in expected values:

Here’s what normal looks like. This cyclical pattern seems ok!

Since we know these limits and expectations we can go a step further and process the information for our user ahead of time. Where a norm or expectation is violated, we can preemptive highlighting the violation. Now the user is drawn to the most important information (Few, 54)¹.

We seem to be violating the SLO on this instant chart!

We’ve gone from an anonymous line on a chart to a situation with lots of context and clear guidance as to the state of the system. Our SLO is being violated!

Flagging Alerts or Other Bad State

A common use of system dashboards is to diagnose failure, especially in response to alerting. Our earlier control-rate context can be combined with data from the alerts. When and how did this fire?

Timeline annotations for alerts, highlighted violations, and coloration of values over a threshold!

This information helps our responder understand what brought about the alert, helping them orient to the problem. Other charts may provide additional context or perhaps our added menus help with next steps.

Projections or Other Synthetic Data

Much of the data we present is direct measurement from our systems. Because of this, you should take special care when presenting data that is projected, predicted, or otherwise synthesized so that the user understands where to place their trust. The Consumer Financial Protection Bureau provides guidance of deemphasizing such data.

An example of this consideration is the fan chart.

The estimated value and the range of possible future values helps the user see what may happen.

Review

Spend some quality time labeling your charts with titles, subtitles, and whatever else you can find.
Our job doesn’t stop with just time series. Users need context to understand what else is going on. That context comes from all over our organization.
Colocating event data into time series charts helps our user know what might’ve changed. These events are observability data, and arguably have better signal-to-noise ratios.
The data in our charts can be improved with normal, expected, or objective values to orient the user.
Coupling alert information can speed incident response.
Using any sort of projection requires careful visualization.

Summary

Wow, this and the other parts of the series have been a lot of words about slapping charts onto a page! I’ve learned a lot about how to make good dashboards that help my users. Much of this was contrary to my past opinions and advice I’ve given. Beyond my own growth much of this advice shows how much our tools need to improve, especially in the area of context.

This stuff is important. Many organizations rely on these dashboards as the primary source of information for the health, effectiveness, and sheer upness of critical services. They are also a cultural watering hole for engineering organizations. In many cases, they are the business, since you can’t exactly go look at a queue of people using your website. Our only visibility is the data we collect and display.

Dashboards and charts are resources that multiply the creator’s effort. A few days of thoughtful design can repay as thousands of hours of engineers, leadership, and/or users that are well informed. In many cases these can result in improved response time to problems.

That’s worth your time, right?

I hope you’ve enjoyed this guide as much as I’ve enjoyed research and writing it. Please check out the sources cited below each part for more in depth reading on this subject.

This post is part of A Practitioner's Guide to System Dashboard Design.

Citations

Few, Stephen. Information Dashboard Design. Analytics Press, 2013. ↩ ↩²
Tufte, Edward R. The Visual Display of Quantitative Information. Graphics Press, 1998. ↩

What Charts To Use in System Dashboard Design

Wed, 17 Apr 2019 07:49:00 +0000

With Part 1: Structure and Layout and Part 2: Presentation and Accessibility you’re probably itching to have some damned dashboards made. Enough theory, which chart should you use?

Thanks to Lita Cho, Shawn Moore, Rajesh Raman, and Joe Ross for reviewing this post.

This post is part of A Practitioner's Guide to System Dashboard Design.

Tables

Wait, a table isn’t a visualization is it? Heck yes it is! Few¹ reminds us that if you’re showing single values, there’s no reason to have any sort of chart (114)¹. For just a few values, using a table is quite effective.

The trusty table.

Tables are common and well-understood visualizations that excel² when looking up a value. The above table makes finding a value and the dimensions related to it very easy. This version may be too dense for a system dashboard, but we can adapt it into a “single value” like so:

A table compressed to something widget-worthy.

If the user needs to know a single value such as the sum, average, or “instant” then using simple numbers or tables is an excellent fit.

Line Charts

Line charts are well suited to most data you’ll be displaying. The line chart begins with points, then connects the points with a line. This is worth saying out loud here because on a run chart this interpolation of data gives the user a sense of shape over time. You’ll commonly hear people say that a metric is trending up or down as they watch updates.

The shape is clear. Something is happening to our latency!

Line charts are popular visualizations because they embody many pre-attentive attributes with a minimum amount of pixels. Lines give us slope and angle, position, and color to derive lots of information without thinking hard. We can even get fancy and use line width and intensity to draw attention:

The brown line is clearly the important one here.

Not All Line Charts

Before you close this tab and make everything a line chart, there are a few things to be aware of. Having too many lines in a chart can hide shape, muddy color, add perception latency and generally make them useless, aside from spotting outliers. Skip ahead to heat maps for some help there.

The lines are so close as to be hard to differentiate.

Line charts interpolate data between two points with a line. This is desirable in most cases, since the shape provided aids perception. Unfortunately interpolation can also hide missing information or the opposite; make us think we have data we really don’t. Take our example line chart from earlier, now with points at each reading:

The points show that we only get one reading at each change in latency.

The distance of the line might imply there are more measurements than really exist, such as the big changes in the above chart.

Area Charts

First, don’t use area charts that aren’t stacked. Occlusion just hides data, which is confusing.

What even is this? I can’t see anything but green.

Area charts are best used when the whole of sums is the important value, as in showing the contribution a few steps make to a total duration. They share a lot with line charts, in that they show shape. Unfortunately this shape is also their downfall: when one of the bottom areas changes, so do all the areas above it. A user may be confused where to attribute the change. Spotting the real change in this situation can be tricky.

Our latency moves around 4s, but we can’t tell what is contributing to the change. Is it all of the lines?

Few suggests that instead of using an area chart, use a single line chart showing the sum and an additional chart showing the contributions (Few, 146)¹. This makes spotting the change significantly easier.

This is the same data as the area chart above. The left chart shows sum of latency, right chart shows each contributor. The contributors to latency are obvious and we can see our total!

Bar Charts

Where line charts show shapes, bar charts show “exact” values. For most system metrics you’re better off using a line chart. The exception would be a chart which compares a few metrics:

We can easily compare these two time series using bars. More would be tough.

The bar chart aids comparison of values at each interval through clear interval association and easy comparison of length by the user. Choosing a bar chart therefore means that your user’s goal is the comparison of a few values over other concerns.

Be careful to avoid moiré patterns. This can happen when your bars are too small and evenly spaced. Do include a little space between bars to aid the visual separation, as seen above.

Stacked Bars

See stacked area charts. You rarely want to use a stacked chart, unless the focus is on the whole value. Users will have difficulty comparing sizes in the stack.

Heat Maps

Reach for heat maps when you have a line chart that has too many lines to make sense of. Heat maps vary the intensity of color so that a lot of data can be packed in to a small space. Where occlusion hurts line charts, heat maps use the overlap to add more color. This has the novel side effect of showing banding where there is overlap aiding in pattern and outlier detection.

The line chart is terrible at everything but outliers. The heat map shows where the population is grouped.

This ability to view more data and spot patterns isn’t free. It comes with a decrease in accuracy because humans can’t easily differentiate more than a few intensities of a color. Few says that more than 5 intensities is the limit for distinctiveness (Few, 86)¹. This means that heat maps are best deployed in situations where accuracy is less important than the general shape of the visualization.

Gauges, Bullet, Ranges, etc

Only use a visualization with limits if the data actually has those limits. For example, displaying requests per second as a gauge is misleading, as it has no maximum!

If you do have a limit to show, such as a percentage or a queue with a maximum depth, then using these sorts of visualizations can be incredibly helpful. It allows you to communicate what the limit is as well as how close a value is to that limit. Use these types of visualizations when your metric has a basis for comparison, such as SLOs!

This is a suitable use of a range as it has bounds and orients the user.

Combining this with color yields a quick understanding of current value, upper and lower bounds and urgency of the current value.

Sadly, I am not aware of any tools that allow the use of bullet graphs in their fullest form. At best, you can usually find only a single bullet.

Others

Some that don’t deserve a whole section.

Pie charts: Don’t. Humans generally can’t understand the area of a pie slice. Use bars.
Donut charts: Mostly cute ways of showing a single value. Or a pie chart with a hole in it.
Stacked Line Charts: Area charts, but worse since the lack of area fill makes it a surprise. Avoid.
Flame graphs: Great for some uses cases, not really in scope for system dashboards, more for profilers.
Sparklines: Great to supplement single values, but often lack context without the familiar bits of a line chart. Few proposes what he calls sparkstrips that add bands of colored or hue to orient the reader. Sadly I don’t know of any tools that support these improvements.
Box plots: Not common in our tools
Dot plots: Use bars. May have some use cases for rare data over a long period?
Scatter plots: Rad for correlation, but that’s a specialized case that we’re not covering here.
Dials and gauges: If your data does not actually have lower and upper-bounds, presenting it in a dial is misleading. They also waste a lot of space.

Summary

Before you flame me for missing some weird visualization or an option that your tool has, let me remind you that if your user understands the dashboard, chart or visualization then it’s good. As your use case broadens then so will your user base and we fall back on the guidance from Part 2 to choose our visualizations.

Line is probably what you want. Maybe others. A lobster is not a chart.

Use line charts, they are great! Single value or tabular charts are excellent for instant/single values. Bars make for great comparisons of small numbers of series and heat maps have some utility.

Please read on for the next items in my series:

This post is part of A Practitioner's Guide to System Dashboard Design.

Citations

Few, Stephen. Information Dashboard Design. Analytics Press, 2013. ↩ ↩² ↩³ ↩⁴
No pun was intended, honestly. ↩

Presentation and Accessibility in System Dashboard Design

Tue, 16 Apr 2019 07:49:00 +0000

After part 1 we’ve got a nice grid and proper visual weighting within that grid. We’re now on to thinking about the presentation of the data to our user. This is where things often go awry, as many a pixel has been lit in pursuit of catchy graphics. Keep your design decluttered and consistent.

Our guiding principle must be the user’s goals and the integrity of the data we’re displaying. Since this will vary widely — counts of users, fleeting time spent in garbage collection, and mercurial error counts — we must build up a framework we can use to make choices.

To that end, let’s explore what it takes to make a good visualization of system metrics.

Thanks to Lita Cho, Shawn Moore, Rajesh Raman, and Joe Ross for reviewing this post.

This post is part of A Practitioner's Guide to System Dashboard Design.

Choice of Data, Gardening

You might expect this piece to tell you what specific data to put in your dashboard. Well, fair reader, that’s mostly up to you! Remember our users and their goals from part 1. Use your list and decide what best fits the bill for what Few describes as the four stages of monitoring information (32)¹:

Updating high-level situational awareness.
Identifying and focusing on particular items that need attention.
1. Update awareness of this item in greater detail
2. Determine whether action is required
If action is required, accessing additional information to determine appropriate information.
Response.

In general the RED method and/or the USE method provide with the best signals for high level service function and awareness. Optimally, you should only use key performance indicators (KPIs) as a dashboard is for situational awareness and not for data discovery (Few 30-31)¹. That said, there are plenty of use cases where more specific information is needed outside of common KPIs. Refer back to your goals for help when you aren’t sure. Does the user need this to fulfill his/her goals?

We should also remember that these dashboards are often tracking systems that change frequently. You’ll need to add occasional one-off charts or temporary widgets to track new and interesting KPIs or just data that is important for current issues. Your organization should build in time to regularly look at dashboards with a critical eye and sweep away any artifacts that are no longer useful. Check that one chart you added back when you were leaking file descriptors and ask if it’s really needed anymore. You know which one I’m talking about.

Effectiveness Of Visualization

When you open the palette of widgets for your dashboard, there are probably lots of options. Temptation is high to use the fancy ones! But not all of these are equal, as humans do not perceive all forms of visual information the same. Our user is trying to quickly orient themselves, so our job is to present things in the most efficient way.

Ranking of perceptual tasks in order of accuracy.

These human quantitative perceptual task rankings (Mackinlay, 125)² provide a handy guide for determining how easily our reader may be able to process our visualization. Some visual components are actually processed subconsciously using pre-attentive processing. The careful use of color, form, position and motion can greatly ease and speed up important information in a dashboard (Few, 80)¹.

Examples of pre-attentive attributes You can see these without even thinking about it!

Scale and Units

Moving a bit deeper into the form, Cleveland and McGill (830)³ ranked tasks by accuracy for us humans, finding position along a common scale more accurate than on non-aligned scales. Few (40-41) reinforces this by specifically advising the use of units and appropriate detail for data to avoid unnecessary computations for the user. For this reason it is best to use common scales, axis and units where possible.

Three charts of the same data, without units, with units and no longer aligned. 250 what?! The left and middle chart are easy to compare, the middle is easier to understand. Alignment and units matter!

With this information you should be able to more effectively choose visualizations, picking those that help your reader quickly perceive the data.

Time

We’re using run charts heavily, so we should take care with how we present time. Charts should use a common time window. If the first chart shows a slice of 1 hour anchored by “now” then so should they all. This is especially true when there is a time selector or other “global” control involved.

Many dashboard tools provide a helpful “cursor” when you are hovering your mouse over a chart, pointing to that same instant in other charts nearby. This can both orient the user’s understanding of time as well as make clear any deviations in the relationship between time and space, as evidenced in this example where one chart is wider than others:

The bottom chart is wider than the top two, but the vertical bar “cursor” keeps us aligned.

Recall our earlier points about alignment, as it also applies horizontally. Widgets or charts that do not adhere to the same time/space relationships should clearly communicate their temporal choices either through visual differences (i.e. a “single value” widget) or clear labeling.

Sneaky Aggregation

Be aware that many visualizations will need to deal with aggregation that happens automatically.

For systems work our dashboards must deal with wildly varying time windows as our users scoot around. The default for most system dashboards is a “current” view showing something ranging from the last hour to the last fifteen minutes. Assuming ten second intervals, every hour will contain 360 data points! Because there are only so many usable pixels for each of our precious little charts, dashboarding tools must make decisions about how to fit all those data points in a small box.

The top chart covers 15 minutes, the bottom 1 week. Note how we completely lose the periodic increase in latency due to the default average aggregation. The resolution is shown to the right of the chart title.

This is usually done via aggregation: showing an average, sum, or something similar. These defaults can make a big difference. Consider our earlier chart, but with the aggregation set to use a maximum instead of an average:

With a maximum aggregation of the weekly data, our latency is ~30% higher.

Consult your tool’s documentation for how this works! We’ll cover concerns about this when we talk about specific chart types.

Aside: Rates and Sums

This is, in my experience, one of the most asked about problems in system dashboards. Showing a counter as either a rate or a sum can have a large effect on the user’s understanding. If you’re viewing the number of errors as a rate then each point on the chart will be the summed count of errors divided by the time elapsed. Viewing as a sum will, of course, be the sum. This means the shape of the data should be the same, only the scale differs.

When using rate or sum it is advised that you make the distinction very clear from the title of the chart. Work with your user to choose the form that best suits their needs when choosing between rate and sum. What do they expect to see?

Note: Be mindful of rate and sum when interacting with aggregation!

Warning: Missing Data

If something goes awry in your measurements you may miss some data points. This can be problematic if your user is assuming regular data points! If this happens, be wary of interpolation. Commonly, system dashboard tools use linear interpolation to connect missing data. Sometimes they are completely unaware the data is missing at all. Sometimes the resolution of the data changes.

The blue line is filling in zeros for missing data. The red line uses linear interpolation.

In a fit of wonkery, I actually contacted Stephen Few about this problem. I’ve grappled with it for a long time and was curious of his take when trying on missing data in irregular, varying intervals:

If you’re display[ing] values along a timeline, that timeline usually exhibits intervals that are consistent. As such, they are evenly spaced. If the interval changes, the the spacing between intervals should also change accordingly to visually indicate this change and should remain consistent unless the size of the interval changes again, at which time the spacing would change again. Values associated with consecutive intervals of time should be connected with a line, but when intervals are missing values for any reason, the line should not continue through those intervals. If values are collected intermittently (e.g., only when particular events occur), rather than at regular intervals, then they should not be connected with a line. A simple data point (e.g., a dot) or a bar should be used to encode the values in cases like this. Good luck in finding a tool that can handle this.

How interesting! To Stephen’s point I wasn’t sure how to do this in software. To solve that, I sketched it.

The middle section is faster intervals. We’re missing some data on either side. Missing data does not connect. The pink highlight is informational and would not be in the chart the user sees.

You might wonder why no dashboard packages do this for you? Mostly because they don’t know if data is missing or not. We commonly infer the interval that metrics come in, and in a modern world of ephemeral containers these metrics can come and go in a flash. A tricky problem for sure, and something to aspire to.

General Guidance

Some other great bits:

Favor soft colors found in nature. Save the use of vivid, attention grabbing colors for thresholds or other health information you want to call attention to. Stephen Few’s Practical Rules for Using Color in Charts is an excellent guide.
Mute or eliminate background grids, when possible. Consider lightening them. Keep the principle of closure in mind and eliminate some of those boxes.
Avoid faked perspective, such as in 3D graphics that serve no data purpose.

Accessibility

Your user may have difficulty interpreting your dashboards if they are not accessible. The W3C has lots of content on accessibility to ensure all your users have a great experience. Depending on your tools, some of these may be easier than others, but all are worth considering.

Color blindness is very common, with 1 in 12 men and 1 in 200 women affected by Red-green color blindness. This is especially worrisome in dashboards as there is often a heavy reliance on using green for “good” and red for “bad” signaling. Leverage multiple techniques for signaling by leveraging the “effectiveness” section above: size, color, text, or icons.
Still on color, look to the W3C’s guidance for a 4.5:1 contrast ratio between text and backgrounds. You can also use this as a reason to choose clearly distinct colors for data as well. The Consumer Finance Protection Bureau has an excellent guide on color in visualizations.
Consider screen readers and what they read for your widgets. In addition to descriptive titles, consider additional helper text in the widget. Additionally, consider the export of CSV data with descriptive headers so that users with screen readers have a way to tab through the data.
Make use of accessible navigation and data enrichment features to leverage browser assistance where applicable with tools like WAI-ARIA.
What is the target display device? A dashboard for a large TV likely has different needs than one used on a 13” laptop display by an on call engineer. Be sure and use your dashboard during the design phase on an appropriate device.

Summary

There’s a lot to think about when displaying data to our users. There’s a lot to consider and this post provides a good explanation as to the thought that designers and engineers creating our system monitoring tools have to take into consideration. The buck doesn’t stop there though, as you, the designer, must continue to make good choices when building up your dashboards!

Review

Use KPIs like RED and USE. Your user’s goal will dictate any additional metrics.
Review dashboards periodically for improvement and cleanup, like the CASE method but for dashboards!
Preattentive attributes like position, length, slope and color can help with quick comprehension. Some are more accurate than others, choose them wisely!
Use units everywhere use common scales.
Time can be tricky, so use consistent time windows. Label any deviations clearly.
Be aware of forced aggregation and choices on display like rates and sums. Clearly label your charts.
Consider accessibility in design and data display. Avoid reliance on red and green and ensure sufficient contrast.

Please read on for the next items in my series:

This post is part of A Practitioner's Guide to System Dashboard Design.

Citations

Few, Stephen. Information Dashboard Design. Analytics Press, 2013. ↩ ↩² ↩³
Mackinlay, Jock D. Automating the Design of Graphical Presentations of Relational Information. ACM Transactions on Graphics, Vol. 5, 1986 ↩
Cleveland, William S., McGill, Robert. Graphical Perception and Graphical Methods for Analyzing Scientific Data. American Association for the Advancement of Science, 1985. ↩

The CASE Method: Better Monitoring For Humans

Tue, 09 Apr 2019 07:49:00 +0000

Riiiiiiing! It’s 3am and you’ve just been dreaming about something great then poof: the phone rings. You’re on call this week and something seems to’ve gone awry. Automated systems are beckoning you to assess the situation and take action. Welcome to a critical point in running modern computer systems. Let’s talk about how to make alerting better for humans.

I’d like to introduce a philosophy for monitoring borne from my decades of on-call experience, my role in multiple large observability teams, and heavy influence from Rob Ewaschuk’s seminal My Philosophy on Alerting — which has since been encoded into the Google SRE book — and John Allspaw’s Considerations for Alert Design.

Thanks to Kelly Dunn, Arijit Mukherji, and Maxime Petazzoni for reviewing this post.

What is CASE?

Inspired by Brendan Gregg’s USE Method and Tom Wilkie’s RED Method I have backronymed a method. I call it the CASE Method and it defines four points that a team should consider and maintain when working with automated monitoring:

Context-heavy
Actionable
Symptom-based
Evaluated

Using CASE, an organization will have a healthy skepticism about interrupting humans. Monitoring will be evaluated regularly for value and effectiveness. Humans will have better mental models and greater confidence when alerted.

To get catchy for a second, the idea is that you need to make a CASE for each alert’s existence. :sunglasses:

Why do we need this?

On call can suck. There are a lot of reasons for this and CASE won’t help you fix all of them. It can, however, improve the quality of the things that wake you up in the night. Sneakily encoded into it are a number of organizational processes that may help as well.

I’ve found the RED and USE methods really helpful not only in building things, but also in the shared vocabulary they provide. It is my hope that CASE brings some clarity and conversation to the alerts that protect our systems and harangue our colleagues.

The core idea is to establish a culture in your organization where the existence of an alert is viewed with healthy skepticism. Alerts may be created for good reason, but we should be skeptical of their continued value unless it is demonstrated. What justifies this alert, and has that criteria been re-evaluated lately? CASE provides a framework for these questions.

Context-Heavy

Reading a jargon filled text message on your phone at 3am is hard enough at the best of times. Responding to that page effectively requires information. Optimally the information is targeted to the problem in question and helps the responder understand the system’s context as quickly as possible. In a sense, one must “design” the alert in such a way that this is possible. This is effectively the observe and orient portions of an OODA loop. Spending time on this design should be worthwhile, because interrupting a human being is costly and should be treated with respect.

Lots of things cause problems. Especially ghosts.

How can we help the responder? Alerts are one of the first stimuli that an operator will receive and are therefore a strong contributor to any hypothesis generation they may do. Runbooks and dashboards are common destinations, but are they suited to the alert rather than being full of general guidance? Allspaw claims that we must “think about the way the alert could be interpreted or acted on” (slide 29)¹. A good alert is designed for the operator rather than just set at a threshold and pushed out into production.

To that end, here are some ideas for improved alert context:

Link the user to something helpful and designed, not just to a generic runbook or dashboard. In the past colleagues and I used “investigation dashboards” that were tuned to specific alerts. This may help well known failures, but might be misleading for other things. It’s a tightrope!
Provide insight into the history of the alert: is it new? How often does it fire? Is it seasonal?
Recent change awareness: Help with system state. Has anything changed lately like a deploy or feature flag?
Show relationships and inform the mental model: Are the system’s dependencies clearly shown, preferably with indications of health?
Get the user to a team quickly: Can the responder see any in-progress incidents or determine who else in the org has already been paged? Has the incident management program been invoked?

Optimally your incident management program feeds more suggestions into how alert context can be improved as you investigate failures. We can always improve this!

Actionable

Is the responder supposed to do something in response to this notification? If action is not necessary or clear why have you interrupted them? The intent is to avoid alerts that pester the operator when no action is needed.

What am I supposed to do now that you've surprised me?

What am I supposed to do now that you’ve gotten my attention?

In the early days of a system when complexity is low and few people are involved we often establish monitoring as a form of informational awareness. A note saying that heap usage has grown may be a nice bit of context for us in case we see a later interruption in service. At scale this becomes unmanageable because our systems operate in various states of degradation at any time. This quickly leads to alarm fatigue and can result in desensitization that harms an operator’s ability to respond effectively either due to ignoring or outright filtering of such notifications. Don’t fall into the trap of allowing alerts to continue, only to route them to email and then into a folder you never read.

These are the attributes of an actionable alert:

The situation requires action, it is not merely informational.
There is no obvious automation for this action, or automation is unsafe. If the action can be automated, automate the damn thing and stop bothering people!
The situation contains urgency guidance in the form of a service-level agreement (SLA) and perhaps a recovery time objective. This allows the responder to enlist the help of the organization’s incident management program.

To be clear I am not advocating that you only alert on the topmost SLOs for your API or similar. This concept of monitoring SLOs is fractal in that each service in your organization can be approached the same way. You will obviously be monitoring top level SLOs that face paying customers, but you will also monitor infrastructure SLOs like databases. Pretty quickly these become focused on internal customers and supporting them. Turtles all the way down!

Symptom-Based

Like it or not, you’re probably working with a distributed system (Cavage)². As a result, your work is likely to employ many tactics to isolate and buffer services from failure (Treynor et al)³. While pangs of high garbage collection durations or database query times might mean something is awry, these rumbles are not urgent if users are not going to experience problems either now or very soon.

These sorts of signals may be important and/or actionable, but if they are not causing user difficulties then they are likely not urgent enough to warrant interrupting our operator. Cause based alerts are snapshots of our mental models about system failure. It’s best to monitor important symptoms rather than attempting to enumerate every possible cause of failure.

To ensure your alerts are actionable you can focus on the performance indicators that are important to your users. Ewaschuk called this aspect “monitoring for your users”. Also recall that you’ll be applying this philosophy through your entire organization. Once service in the bowels of your infrastructure begins having urgent problems, you can expect that the relevant team will be involved. Further hardening your systems against that failure is a wholly separate line of work (Treynor et al, section Strategies for Minimizing and Mitigating Critical Dependencies)³.

Symptoms Change Less

Cook reminds us that complex systems contain a multitude of flaws, faults, and problems⁴. If you’re trying to enumerate each of these possible causes you’ll be doing a large amount of unhelpful work that never ends. Many of the problems you’re attempting to spot will shift away over time. Sridharan says that systems are “not necessarily going to be operating while 100% healthy at any given time“ so we should focus on more “human-centric” situations (“Distributed Systems Observability”, 7)⁵.

Avoid Incident-based Alert Debt

A common incident remediation is to make an alert for a cause. This “debt” of narrowly useful alerts may create a false sense of confidence, since your system will continue to change how it fails.

We are monitoring the SHIT out of everything but that…
— Honest Status Page (@honest_update) May 23, 2017

Rather than creating a false sense of confidence from cause-based alerts, instead ask yourself:

Why didn’t a symptom-based alert find this?
Could improved context for the responder help?
How can we improve our observability tooling to make this diagnosis faster rather than creating alert debt?

Observability tooling for diagnosis will only improve if you consider it the means for getting from symptom to solution. Without this feedback loop you end up with a collection of reactive alerts and charts that document your past failures and none of your future ones. This provides a wonderful opportunity for an organization to shift from a reactive stance on alerting to a proactive one. It also elevates the conversation between engineering and product via common expectations and clear value. The CASE (:wink:) for any alert’s existence is clear.

Cause-Based Is OK In Moderation

Sometimes the system being monitored leaves us little choice about a cause-based alert. Other times our operators are acutely aware that the symptom will lead to imminent failure and is therefore very actionable. Maybe you just aren’t sure what’s up and set up alerts out of an abundance of caution. This need for action is hopefully temporary until the system can be modified to deal with degradation.

Keep the other parts of CASE in mind when dealing with these situations. Being temporary doesn’t remove the need for thoughtfulness.

Evaluated

The changes — new code, new infra, or new… whatever — to our systems introduce new forms of failure (Cook, 3).⁴ Do we still trust that this alert works as expected? Having sharp, recent mental models of your systems and experience responding to some alerts to aid anticipatory thinking is a key part of a learning organization. Because the faults in our systems will continue to evolve, so must we.

We need to regularly evaluate the performance of each alert to ensure they work the way that we expect. Listen up management, you can have a strong impact helping your team establish this! Here are some ideas for how to do evaluation:

Leverage chaos engineering, game days, or other forms of testing to ensure that alerts do what you expect. You can do this within your team without the process of larger incident management machinery!
Include collection of data about all relevant alerts that participated in incidents are part of your incident management program. Flag help, harm, irrelevance, confusion and more. Use this as feedback.
Healthy alerts fire on occasion and are well exercised. Verify that all the links work and point to relevant context, etc.
Alerts that never fire or that fire frequently are unhealthy. Improve or eliminate these. Be wary of both overload and underload!
Keep an expiration timestamp on alerts. If an alert expires, reevaluate it against CASE and update the timestamp. Think of this as a freshness date and review with your team regularly.
Make improvement of alerts easy. Use monitoring as code and keep your alerts in a Git repository. Pull requests help you involve the team as well and you get history for past experiences. This may help remove fear of changing the alert or getting sign off from those that “own” it.
Allow feedback on alerts, even if it’s just a quick Google Form so that responders can signal an alert being unhelpful or noisy. Put a link or call to action in the body of your alert and review the feedback regularly.
Establish team norms that those on call have time in their schedule to improve on call during slow periods. Try and leave things better than when you found them!

Closing

I believe that the CASE method provides teams and organizations a way to discuss the care and feeding of automated alerting. A single engineer can begin evaluating alerts through the CASE lens and from that scale up to an entire org working with teams, management, and incident management programs to keep alerts groomed and valuable. It requires no fancy tools or complex process.

As an industry we must continue to consider the human factors of on call responsibility whilst giving our customers the best experience possible. The tools and practices that we use have a vast opportunity for improvement. I hope that CASE contributes to this improvement.

Bask in improved alerting!

via GIPHY

Citations

Allspaw, John. “Considerations for Alert Design.” Monitorama 2013 Portland, OR. ↩
Cavage, Mark. There’s Just No Getting Around It: You’re Building A Distributed System. ACM Queue, 2013. ↩
Treynor, Ben et al. The Calculus of Service Availability. ACM Queue, 2018. ↩ ↩²
Cook, Richard. How Complex Systems Fail. Cognitive technologies Laboratory, University of Chicago. 2000 ↩ ↩²
Sridharan, Cindy. Distributed Systems Observability. O’Reilly Media, Inc. 2018 ↩