PDF Summary:The Phoenix Project, by Gene Kim, Kevin Behr, and George Spafford

Information technology is so deeply embedded into every facet of the modern world that businesses today live or die based on the strength of their IT departments. If a company fails to properly manage IT services, it risks losing its competitive edge and the goodwill of its customers. In The Phoenix Project, Gene Kim, Kevin Behr, and George Spafford present a fictional case study of a business that’s failing because it doesn’t align the work of IT services with the company’s goals. The authors then show how an IT department can turn itself around and get back in the game.

In this guide, we’ll dissect the three pillars of IT management that Kim, Behr, and Spafford recommend for any modern business. In addition, we’ll clarify the authors’ basic concepts and explore further ideas on management, leadership, workplace culture, and productivity.

(continued)...

Sources of Work in IT

The way the authors differentiate between “business projects” generated by the company at large and “internal projects” generated by IT might at first glance seem to imply that IT is inherently separate from the rest of the organization, an attitude that the authors discourage later in the book. A potentially more useful distinction is that business projects advance the company’s business goals directly, while internal projects serve those same goals indirectly (for example, by improving overall efficiency). To make sure projects stay on track and maintain buy-in from the people involved, it’s important that any project be clearly aligned with the company’s goals—whether that project is internal to IT or in support of another division.

From a layman’s perspective, the type of IT changes the authors describe can often seem perplexing—such as when a program you’re using or your operating system mysteriously updates itself with no discernible change in performance. Nevertheless, these seemingly constant updates are vital to keep up with advances in technology, fix software bugs, close security vulnerabilities, and stay in compliance with ever-changing legal regulations. From a business perspective, these changes also arise as a result of user feedback and market research to determine how to better match a company’s products with customer demand.

Unexpected work is almost always an emergency, or at least is made to seem that way, and it gets in the way of doing anything else. Unexpected work includes fixing a nonstop flood of technical problems, many of which aren’t prioritized by importance, but rather by how vocal the person complaining about the issue is. What’s worse, unexpected work creates even more work by taking time away from the system testing and preventive maintenance that would stop such problems from arising in the first place. To correct unexpected problems quickly, the solutions implemented are often untested patches and workarounds that build up technical debt in the system, laying the groundwork for more unexpected problems.

(Shortform note: In the authors’ fictitious company, as well as in many real-world organizations, people have learned that the quickest way to get IT’s attention is to treat every problem like an emergency. However, this makes prioritizing difficult. One solution may be to implement a priority matrix, as described by Steven Covey in First Things First. While those tasks that are truly urgent will require immediate attention, Covey argues that the most important work is that which will have a significant impact but hasn’t yet risen to the level of an emergency. Making this category of work a priority will ideally prevent problems from becoming emergencies at all.)

The Bottleneck

The authors say that the other work stoppage in any system is the bottleneck (which the authors refer to as the constraint), defined as the one link in the chain that limits the speed of the entire production process. In the fictional example, the bottleneck is Brent, the lone engineer whose unique and exhaustive knowledge of Parts Unlimited’s computer systems makes him the indispensable go-to guy for every task IT tries to perform. As a result, IT can’t do anything without involving Brent in some way, and so no work gets done any faster than Brent is able to get to it. Because Brent is so overburdened, he doesn’t have time to document his work, which means his knowledge isn’t shared and IT becomes even more reliant on him as a crutch.

(Shortform note: One danger of the particular type of bottleneck Brent represents is that employees overloaded with so many tasks risk burning out from the combination of high demands, insufficient resources, and little or no recovery time between switching from one emergency to another. Though The Phoenix Project doesn’t address this directly, 75% of workers experience burnout at some point. Preventing workplace burnout requires listening to employees’ needs, curbing excessive overtime, and building a workplace culture based on clearly defined roles and expectations.)

Brent is merely one example of the kind of bottlenecks that can constrict a system. The authors list other types of bottlenecks based on those described in The Goal: A Process of Ongoing Improvement by Eliyahu M. Goldratt. These include the creation of test environments for software, installing large amounts of new code, and getting approval for changes from committees. Whatever your system’s bottleneck is, the authors are clear that you can’t push work through your department any faster than your bottleneck will allow. Any attempt to do so will result in a traffic jam of work piling up at one station while the rest of the production line sits idle.

(Shortform note: Though the authors derive most of their ideas about managing bottlenecks directly from The Goal, Goldratt makes some recommendations that aren’t echoed in The Phoenix Project. One is to decrease workstations’ idle time, which Kim, Behr, and Spafford caution against, as we’ll see in the following section. Another is to outsource the bottleneck’s functions, which The Phoenix Project’s authors also don’t recommend. In The Goal, Goldratt doesn’t discuss pesky human factors such as personality clashes, bureaucratic hold-ups, and company values that also contribute to the problem of bottlenecks and trying to unstop them.)

The Pillars of Production

None of the problems of IT are insurmountable, but Kim, Behr, and Spafford argue that addressing them requires completely rethinking how IT work is done. In their fictional case study, they demonstrate how work management principles developed on factory production lines can be applied in an IT environment, where the production of software, databases, and networks can be likened to manufacturing physical products. The three foundational pillars of production can be summed up as 1) fast workflow, 2) quick feedback, and 3) continual improvement.

Because the authors’ point-of-view character is a vice president of IT operations, one might assume that their advice is intended for readers in corporate management positions. However, understanding the principles that follow will be essential for everyone in the production process, since implementing the authors’ recommendations will require buy-in from many people in a company and certainly everyone in IT.

(Shortform note: The authors’ core concepts (which we’ll discuss in the sections that follow) evolved from principles of ITIL, Lean, and Agile production methods, which are mentioned in The Phoenix Project but not explained in depth. ITIL (Information Technology Infrastructure Library) is a set of best practices for delivering IT services that were introduced in the 1980s and have continued to develop over time. Lean production is a manufacturing process first developed at Toyota that minimizes waste within a production system. Agile project management is a system that takes Lean principles and applies them to software development.)

Pillar 1: Fast Workflow

The first keystone, “fast workflow,” may sound like a goal and not a place to start. However, the authors lay out a series of procedures that can establish a faster flow of work through IT from the outset. These include creating a visual tracking method to monitor and schedule work through IT, reducing the paths by which work enters the pipeline, breaking projects down into smaller, independently manageable components, and opening up your bottleneck so that work flows through it as quickly as possible.

To begin with, managing workflow is impossible without a way to monitor its progress. The authors repeatedly recommend using a kanban board (a visual tool that shows the status of tasks as they pass through IT) both for tracking work and for scheduling tasks as they come in. A visual tracking tool lets you document how much time a task takes at each station, and therefore it indicates how much work you can afford to take on. If certain tasks are regularly repeated, then documentation will allow you to plan for exactly how much time they will take. It will also make you aware of how many time-consuming handoffs occur as a task is passed from one station to the next and whether you can reduce those handoffs in order to speed up workflow even more.

(Shortform note: The kanban boards the authors mention are a production tool first developed by Toyota to visually track issues on their car production lines. They have since become a standard tool for tracking work through any kind of production process, and while kanban boards can be digital, the simplest ones—as depicted in The Phoenix Project—are just whiteboards with vertical columns depicting each stage in the production process and sticky notes representing each project moving through the system. In Personal Kanban, Jim Benson and Tonianne DeMaria Barry describe how kanban boards can be used by individuals to manage tasks in their daily work and personal lives.)

As you monitor your department’s workflow, you’ll also become aware of all the different avenues by which work—often unexpected work—enters and confounds the flow of operations. People in many organizations have grown used to calling IT staff directly to fix what they perceive as “urgent” computer problems, interrupting and backlogging whatever projects your staff was meant to be working on. Identifying where and how unexpected work interrupts production is the first step in limiting the amount of work in progress. The authors stress that doing so is vitally important to speeding up IT production.

(Shortform note: Unexpected interruptions are doubly deadly to IT workflow. Not only do they back up the whole production process, as the authors describe, but they also hinder individual workers’ ability to focus on any given task. In Eat That Frog!, Brian Tracy details the cost of multitasking—or “task shifting,” as he more accurately describes it. After being interrupted, it takes 17 minutes to completely refocus and continue with your work. Multiply that by the number of IT personnel who are interrupted by one crisis after another, and it’s clear how badly unexpected disruptions can impact overall efficiency.)

Another key component of speeding up workflow is to reduce the size of the projects you take on. Huge projects meant to accomplish multiple business goals at once, such as the authors’ disastrous Phoenix Project, are so unwieldy that identifying design problems and errors is that much more time-consuming and difficult. Dividing large projects into small, discrete units can let each part move through design, testing, error-fixing, and rollout in a timely manner, sometimes on the order of days or weeks instead of months or years. If there’s any fundamental flaw in one of the smaller components of a project, it can be caught and corrected that much sooner, before it can have a devastating impact on the whole.

Individual Focus Applied to a Team

The authors’ suggestion to break large projects into small, discrete units is a common piece of advice that productivity experts usually prescribe to individual workers. For example, in Eat That Frog!, Tracy lists breaking tasks into actionable steps as both a cornerstone for making large goals achievable and an effective way to avoid distraction. In the context of The Phoenix Project, that distraction takes the form of nonstop demands on IT’s time.

At Google, software designer Jake Knapp devised a method for addressing that form of distraction by taking personal productivity tools and applying them to achieving team goals. In Sprint, he and co-authors John Zeratsky and Braden Kowitz describe a process for gathering a team and shutting out distractions such as email and meetings for a specific amount of time—such as a week—in order to complete the design of a tightly focused product with an incredibly rapid turnaround.

Open Your Bottleneck

The authors insist that to truly speed up workflow through IT, you should make the most efficient use of your bottleneck. By visually monitoring the work through IT, you’ll quickly identify where the bottleneck is—most likely one overburdened workstation. Though it may be counterintuitive, workstations need idle time in order for work not to pile up. The authors provide a simple formula to determine how much time a task will spend in the queue at any given station (the wait time) depending on how much time that station spends idle:

Wait time = Percent of time busy / Percent of time idle

According to this formula, wait times at a station that’s 80% busy and 20% idle will be four times longer than at a station that’s 50% busy and 50% idle.

(Shortform note: The authors’ suggestion that workstations and employees should have a certain amount of idle time goes against the grain of using efficiency as a way to cram in even more work. In addition to the authors’ mathematical reasoning, some studies have shown that downtime on the job is vital to mental health and productivity. Some research has revealed that the brain requires regular rest periods throughout the day to maintain its full cognitive function. Nevertheless, we must note that in order to be productive, downtime should be planned and not the result of unforeseen equipment breakdowns or poor communication.)

Once you’ve identified your bottleneck, you can arrange your workflow so that you don’t send work to the bottleneck faster than it can handle. Tracking work will also let you know if any of the bottleneck’s work can be automated. If the reason for the bottleneck is unshared skills or knowledge, as in the fictional example of Brent the software engineer, you should document everything the bottleneck does so that knowledge and skills can be shared among your team, eventually leading to sharing of the workload. Once one bottleneck has been opened up, you may find another in your production line. If so, apply the same steps as before.

(Shortform note: What the authors describe is a formalized process to convert individual knowledge into institutional knowledge. However, individual knowledge includes more than practices and procedures that can be written down. Individuals have skills and intuitions that can only be acquired and passed on through training and experience. Documentation is certainly vital, but to truly build institutional knowledge, a business must make its knowledge easily accessible to employees while encouraging seasoned workers to mentor new team members, even as early as the onboarding process.)

Pillar 2: Quick Feedback

In order for a streamlined, faster workflow to be beneficial, your system must generate and implement corrective feedback all along the production chain. Feedback cycles, like your projects themselves, should be small and efficient so that problems can be identified quickly, connecting both Development and Operations, and resolved in a way that generates new information.

The authors note that systems such as large software packages and sprawling computer networks are far too complex for any one person to fully understand. Therefore, work on those systems must be designed in such a way that errors can be detected and corrected quickly. Small project sizes let you catch and fix problems before they become disasters, but to enable this, your production process must generate feedback at every step in which work is performed. One example of a way to create this kind of feedback is to institute checklists for every task being done, especially at any point in the process where work is passed from one station to another.

(Shortform note: In The Design of Everyday Things, Don Norman describes the “small product” feedback loops recommended in The Phoenix Project as a cycle of iteration that’s intended to generate errors as fast as possible so that they can be quickly corrected. Norman favors this approach for small projects only, arguing that for large projects, implementing constant testing and feedback is cost-prohibitive. Kim, Behr, and Spafford don’t dispute this point, and it may be part of the reason why adherents to their design philosophy argue against large, resource-intensive projects altogether.)

Feedback cycles can’t be confined to Operations—they must include Development as well. Developers must know if their code is effective or problematic as quickly as possible, not months down the line after they’ve moved on to other projects. The authors say that feedback from Ops must be incorporated into the very beginning of the Development process, in effect merging the two departments into one cohesive DevOps gestalt. Ideally, the feedback and response time between designing and installing new systems needs to be fast enough to keep up with customer demand, whether those “customers” are clients of your company or other departments inside your organization.

(Shortform note: To shape the beginning of the development process, feedback loops must transmit information all the way back from the end of the process. In User Friendly, Cliff Kuang and Robert Fabricant explain that designers must understand the end-user by observing and empathizing with customers’ experience—what they enjoy about using the company’s software, and what frustrates them about it. An effective information loop provides feedback in the user’s direction as well. For example, an effective online store design will unambiguously let customers know if they’ve successfully completed a purchase—and if not, then why.)

The moment your feedback measures detect a problem, it should be fixed immediately, not patched with a workaround and put off until later. The authors insist that problems should be tackled by the people closest to them as part of their regular responsibilities. More than that, those people should bring in as much help as they can get from their department to resolve the issue. This way, the issue becomes a learning opportunity that generates new knowledge for the organization. By documenting the whole solution process, that knowledge becomes embedded in IT's procedures and toolkit for future problems.

(Shortform note: There’s a lot more to documenting institutional knowledge than simply writing down the steps people take to solve problems, though that’s certainly a place to start, as it is in the authors’ scenario. In order to be useful, that documentation has to be easily accessible to any employees who need it. In practical terms, it can take many forms, such as flowcharts, training manuals, and internal databases, all of which take time and effort to design, organize, and curate. None of this can be done haphazardly, but must be part of a company’s overall knowledge management strategy.)

Pillar 3: Constant Improvement

The last and perhaps most important component to harnessing IT’s—or any system’s—full productivity is to create a culture of continual improvement through practice, repetition, experimentation, and useful failure. Using the story of their fictional company’s ill-conceived “Phoenix Project,” the authors provide a negative example of a company with a toxic workplace culture and a blueprint for a better way to encourage growth, development, and innovation that benefits your business as a whole.

The defining characteristic of a toxic workplace is that employees’ behavior is guided by a constant fear of failure. In such a culture, administrators address mistakes and malfunctions by assigning blame and taking punitive action. As a result, people are discouraged from identifying errors and problems in a system. Feedback is silenced, because who will provide it when the default cultural response is to kill the messenger? Toxic work cultures stifle any improvement, and without constant improvement, a system will stagnate and problems will fester until they become catastrophic.

On the other hand, a productive company culture will encourage people to report problems at once. The authors show that if employees can trust that they won’t be reprimanded—and may in fact be rewarded—for identifying and helping solve issues detrimental to the company, then those employees will feel an ownership stake in seeking out ways to improve the whole system. If they’re encouraged to take risks in the process by trying solutions that may or may not work without fear of reprisal from on high, then your company will foster a culture of innovation in which even failed attempts at improvement are seen as a way to generate knowledge that can be shared across the organization.

The Value of Making Mistakes

A corporate culture in which mistakes and failures are rewarded as long as they move the company forward may not be as unlikely as it sounds. In Reinventing Organizations, Frederic Laloux says that such companies already exist and that they represent the next stage of organizational evolution. These companies run on the basic assumptions that workers can be trusted to make good decisions and that when employees feel they have an ownership stake in the company’s success, they’ll hold themselves accountable for their mistakes without any threat of administrative punishment. Meanwhile, that same culture of trust gives workers the freedom to unlock their full creative potential.

As a counterexample to the toxic workplace depicted in The Phoenix Project, consider that of Pixar, as presented by its co-founder Ed Catmull in Creativity, Inc. According to Catmull, embracing failure as positive is an essential ingredient to Pixar’s innovation and success in the animated film industry. The key is to normalize failure by removing its associated stigma of fear. That way, when mistakes are made, people come together as a group to address them with creative solutions, rather than retreating into personal silos to avoid receiving blame. To destigmatize the inevitable mistakes that arise from experimentation, managers must cultivate trust by taking risks themselves and admitting their own errors.

Finally, the authors state that businesses should formalize their systems of continual improvement. IT staff can hone new solutions and practices while refining them through repetition and practice. Teams within DevOps can root out flaws in their products by pushing their products’ limits, forcing errors to emerge before they crop up on their own. One way to accomplish this is to deliberately introduce faults and design flaws into the production line so that staff can practice identifying errors, providing feedback, and coming up with resolutions. Not only can you identify systemic deficiencies in this way, but you’ll also promote a culture in which experimentation, risk-taking, and learning become an institutional way of life.

(Shortform note: The idea of deliberately making mistakes to test a system predates The Phoenix Project or even the modern IT department. Inventors, advertisers, and even telephone companies have introduced deliberate errors into their practices to test their guiding assumptions. In order to make productive mistakes, it’s important to know what a system’s underlying assumptions are. Mistakes should be designed to confirm or dispute those beliefs, especially when those assumptions are the basis of regular, automatic decisions.)

DevOps, Past and Present

The ideas behind DevOps—the merger of software development and IT operations into one department—began to cohere about five years before being illustrated in The Phoenix Project. The term “DevOps” was coined in 2009 by Patrick Debois of the photography website Flickr. As a management and design philosophy, DevOps gained traction in the business world over the following decade, allowing products to get to market more quickly while fostering ingenuity and creative experimentation, especially in the software industry.

Currently, DevOps principles are fueling a trend toward applications based on microservice architectures in which individual functions and processes are handled by small, modular units of software. There has also been a shift toward cloud-based computing that lets developers build software without spending their own time and resources building the hardware to support it. Finally, DevOps practitioners are using artificial intelligence and machine learning tools to enhance system operations, speed up software deployments, and enable faster testing and analysis of new products.