Automating processes with software: When is it a good idea?

Let’s face it: we all want software to do the things that are tedious for us to do ourselves. We imagine empowered departments that can increase their efficiency by multitudes through a click of a button that triggers some intelligent automation process and allows staff to sit back and watch with a dreamy smile while twiddling their thumbs (or better yet, being productive on some other task). The reality, though, is rarely so chimeric.

For today’s read, let’s take a look at some basic groundwork for deciding when is a good time to implement custom process automations via software: what are good target processes, what are the risks involved, what we shouldn’t even think about automating and similar thoughts to help make sound decisions.

The Tale of the Overzealous Automator

We’ve likely all seen it before, but in this case, we were exploring some new Project Management (PM) software for internal use. Information tracking for large custom software development initiatives can easily involve thousands of various tasks, documents, references, and other such things to be tracked. Coming up with a good system for doing so is… a lot of fun, let’s (sarcastically) say. We have years of experience in what works and doesn’t work and how to build it around our teams or clients’ needs, and our internal needs. The result: a multi-faceted, full-featured, and comprehensive PM system that allows us to do some pretty amazing things in the hands of a highly capable PM team.

These days, many things within that system are automated. In the early days, most of it wasn’t. We had a fledgling sysadmin (let’s call him Jimbo…) who just loved the idea of automating everything.

Jimbo: “Oh, you need to convert that request into a set of requirements and specs? I can make an automation for that!”

Jimbo: “Hey, when this request’s dev tasks are all completed we can automatically roll up the statuses and place it into user acceptance testing and so on and so forth and so and so…”

Jimbo… slow down there buddy. These are all really great ideas, but we’re not ready for them yet. Until we have a solid process that’s been exercised and proven, it’s usually not worth doing an automation. Jot it down in the idea pad, and take some notes on what such an automation might look like, but don’t do it yet.

But… Why Not?

Good question Jimbo, why wouldn’t we want someone to click a button and have all these things happen automatically for them? Seems short-sighted, doesn’t it?

Here’s the issue: we’re in the early stages of developing a new and significant business process. No matter how great the planning, we’re essentially in a prototype mode here. We’re trying this out to make sure it works. If it doesn’t work, we need to change how we’re doing it. Usually not major changes, just minor adjustments.

But automations are the nemesis of change… once we put an automation in place, it’s pretty much locked in. Through a process prototype stage, and even into the “settling in” phase, a primary tactic is our ability to react and refine things quickly and without headache.  Automations are great but make it very difficult to pivot. Once we have the automation in place, sometimes the best way to pivot is to remove it altogether. (If you ever want to see unhappy staff members, remove some automation for them even if it’s only been there a week)

So, Jimbo… we want a general strategy to only introduce automations into processes we have very high confidence levels in. If it’s an “ehhh, looks good but let’s give it a few weeks and see what rears its’ head” situation, we probably want to get some ideas down for the automation, but hold off on implementation until we’re done playing whack-a-mole with off-the-beaten-path issues.

This isn’t to say we want to entirely rule out automations in early stage development. They do have useful implementations even as short-lived, prototype-stage tasks. For example, if it’s a bulk processing task you’ll have to do more than once, it may make sense to spend the time scripting pieces of it to reduce man-hours. I’m not saying to avoid those altogether, just that it’s easy to become overzealous about the idyllic paradise of full process automation.

Ok Boss, I Get It… What About This Complex Process Though?

Jimbo, Jimbo… I feel bad raining on your parade.  It’s a good line of thought; it really is – the more complex the process is, the more attraction there is to automate it, no doubt.  Still though, we need to take this with a good strong dose of bitter reality.

This really boils down to the same: good and evil, light and dark, automation and change. The more complex the process, the more potential for efficiency through automation, but also the more prone to change.  Sometimes it seems if a two-step process can have n chance of change, then a three-step process can have n2 chance of change. That 15-step process that you’re looking at automating? Hmmm, how can we go about this…

When we talk about the scope of a particular automation, we want it to be small, and well-targeted.  An automation should bridge one gap, and one gap only.  This makes things much less brittle and much easier to adjust when needed. Therefore, when we start thinking about automating some big complex process, what we want to do is take a more granular approach: the goal is to find the small gaps that we can connect via automation, and fill those.  Maybe in a 15-step process, there are really only two or three of those steps that are particularly onerous to handle manually, and we can hit an 80\20 right there.

So We Have All These Little Automations… Can We Chain Them Together?

Jimbo, ‘ol buddy, now we’re getting somewhere! Indeed we can, and by continuing to treat them as a series of small, isolated, and highly targeted automations that happened to be chained together, we get all sorts of advantages.

Of course, there’s the advantage of being able to come in and change any little gap without crashing down some big elaborate, multi-island-spanning epic bridgework. Another great thing about chaining them is that it gives us injection points.

“Injection points?” asks Jimbo.

Sure, it provides us a handhold to do… something with. Let’s say we have something we want to put an automation stop on for manual review before proceeding. No problem, just toss in a guard before that next chained item. Maybe we want to automatically examine the results of the previous automation operation and validate that it’s correct before moving forward. Easy, just slip it in that handhold there, no biggie. Injection points give us a plug-and-play framework to easily make changes on while supporting a series of small automations to bridge small gaps that are also easily changed in isolation.

So You Mean We Can Build an Evolvable and Scalable Complex Automation…

Gentlemen James, I can practically see you gaining wisdom! Keep at this and you’ll be well on your way to designing the magnum opus of all automations: Continuous Integration and Delivery systems!

More seriously though, it’s absolutely correct. Architecting complex process automations in such a way gives us the flexibility we need at the granular level (by now you realize that everything changes…) while providing the efficiency gains of an automated process and the security of being able to put checkpoints in for manual or automated verification. That’s what a real work of automation art looks like.

Say Goodbye to Jimbo…

He’s served us well in terms of looking at the evolution and implementation of complex automations, but let’s get down to some brass tacks here.

 

Automation Targets: the Good and Bad

As we’ve seen, automations that target large and complex processes need to be handled with care.  But that’s not all automation, and some are easily implemented and bring excellent value with fairly low risk:

• Anything where duplicate entry is required is usually a good candidate to be automated.

• Any sort of transfer of data between systems is usually a good candidate to be automated.

• Manual processes such as cleaning up data dumps in Excel can usually be automated fairly easily.

Automatons Beware…

• Resource Usage: A downside of automations is that they seem to have this desire to run wild if someone leaves the cage door open. Something happens in a pre-automation process that throws a kink in the works and you wake up the next morning to an infinite loop automation that racked up a few tens of thousands of dollars worth of cloud compute time… let’s try to avoid that!
• Resilience & Retry Capability: What’s the impact if an automation doesn’t work? Sometimes these things can take a critical role in business operations, and failures can be catastrophic. Let’s make sure to assess the risks, and take the time to build in resilience when needed.
• Failure to Monitor: Any automation, but especially those critical ones, should have some means of monitoring and alerting so failures can be recognized. Automations have a tendency to fade into the shadows once they work as expected (that’s the point, right?), and if we’re not careful about it we could get quite the bite if one fails and we don’t know it. Worse yet, sometimes failed automations can go on failing for a long time before it becomes noticed, leaving us in a less-than-desirable situation of trying to deal with the fallout (typically at the time of noticing, it’s then an urgent need with no clear fix for a months’ worth of failed processes… not a fun place to be).

 

Automated processes have a great appeal and can really push efficiency through the roof. They’re certainly something to strive for, but maybe not something to rush into without fully understanding the risks involved.  Martin Folwer once said of microservice architectures: “you need to be this tall to ride…,” (meaning that the ability to implement without it backfiring in your face has many operational maturity prerequisites) and while the same isn’t always true for automations – the simple stuff and low hanging fruit are usually simple enough – a highly complex process automation system first requires a well matured, battle-tested and proven process to reap the big benefits.