Project Management

DevOps: Operational Disaster Strategies

From the Disciplined Agile Blog
by , , , , , ,
This blog contains details about various aspects of PMI's Disciplined Agile (DA) tool kit, including new and upcoming topics.

About this Blog

RSS

View Posts By:

Tatsiana Balshakova
Mark Lines
Mike Griffiths
Scott Ambler
Bjorn Gustafsson
Curtis Hibbs
James Trott

Past Contributors:

Joshua Barnes
Michael Richardson
Daniel Gagnon
Valentin Tudor Mocanu
Kashmir Birk
Glen Little
Klaus Boedker

Recent Posts

DA 5.6 is released

Disciplined Agile 5.5 Released

Choose Your WoW! Second Edition Is Now Available

Requisite Agility applied in Project Management

Disciplined Agile and PMBoK Guide 7th Edition

Categories

#ChoiceIsGood, #ChooseYourWoW, #ConsumableSolution, #ContinuousImprovement, #CoreAgilePractices, #experiment, #Experimentation, #GuidedContinuousImprovement, #Kaizen, #LifeCycles, #ProcessImprovement, #TealOrganizations, Adoption, agile, agile adoption, Agile Alliance, Agile Business Analyst, Agile certification, agile data, agile governance, agile lifecycle, agile metrics, agile principles, agile transformation, Agile2018, Agile2019, Agile20Reflect, AgileData, Analogy, announcement, Architecture, architecture, architecture owner, Articles and publications, Asset Management, Atari, Backlog, Barclays, being agile, benefits, bi, blades, book, Branching strategies, Browser, Business Agility, business intelligence, business operations, capex, Case Study, Certification, certification, charity, Choose your WoW, CMMI, cmmi, Coaching, Collaboration, Communications Management, Compliance, Compliancy, Conference, Construction, Construction phase, Context, Continuous Improvement, coordination, COVID-19, Culture, culture, Cutter, DA, DAD, DAD Book, DAD discussions, DAD press, DAD roles, DAD supporters, DAD webcast, DADay2019, Data Management, database, dependencies, Deployment, Development Strategies, DevOps, disaster, Discipline, discipline, Disciplined Agile, disciplined agile delivery, disciplined agile delivery blog, Disciplined Agile Enterprise, disciplined devops, Documentation, Domain complexity, dw, DW/BI, Energy Healing, Enterprise Agile, Enterprise Architecture, Enterprise Awareness, enterprise awareness, Essence, estimation, Evolving DA, Executive, Experiment, facilitation, FailureBow, feedback-cycle, finance, Financial, FLEX, Flow, foundation layer, Funding, GCI, GDD, Geographic Distribution, gladwell, global development, Goal-Driven, goal-driven, goals, Governance, GQM, Guideline, Hybrid, Improvement, inception, Inception phase, India, information technology, infosec, Introduction, iterations, Kanban, large teams, layer, lean, Lean Startup, learning, Legal Project Management, LeSS, Lifecycle, lifecycle, Manifesto, mark lines, marketing, MBI, Metaphor, Metrics, metrics, mindset, Miscellaneous, MVP, News, News and events, Non-Functional Requirements, non-functional requirements, Non-solo development, offshoring, Operations, opex, Organization, Outsourcing, outsourcing, paired programming, pairing, paper, People, People Management, phases, Philosophies, Planning, PMBoK, PMI, PMI and DA, PMI Chapter, Portfolio Management, post-format-quote, Practices, practices, Principle, Process, process improvement, process tailoring, Product Management, product owner, Product Owners, productivity, Program Management, Project Management, project-initiation, Promise, Quality, quality, rational unified process, Refactoring, Reiki, Release Management, release management, Remote Training, Remote Work, repeatability, requirements, Requirements Management, research&development, responsibilities, retrospectives, Reuse, Reuse Engineering, ride for heart, rights, Risk Management, Risk Management, Risk management, Roles, RUP, SAFe, sales, Scaling, scaling, scaling agile, Scheduled Workshops, SCM, scorecard, Scrum, ScrumMaster, SDLC, Security, security, self-organization, SEMAT, serial, skill, solutions software consumable shippable, Stakeholder Management, strategy, Support, Surveys, Teal organizations, team development, Team Lead, team lead, Teams, Technical Debt, Teleconferencing, Terminology, terraforming, test strategy, testing, time tracking, Tool kit, Toolkit, tools, traditional, Transformation, Transition iteration, transition phase, Uncategorized, Upmentors, Using PMI Standards, value stream, velocity, vendor management, Virtual Training, Workflow, workflow, workspaces

Date

linkedin twitter facebook Request to reuse this  


Burning Building

There are several disaster mitigation strategies that IT departments may choose to adopt:

  1. Disaster planning. Disciplined organizations will plan for operational disasters. Potential disasters include servers going down, network connectivity going down, power outages, failed solution deployments, failed infrastructure deployments, natural disasters such as fires and floods, terrorist attacks, and many more. This planning will include identification of potential problems, identification of strategies to address those problems, and putting mechanisms in place to hopefully mitigate the disasters. Potential strategies to address these disasters include building solutions that self-test and self-recover, building redundancies into your operational infrastructure, having disaster procedures in place, and practicing those procedures in simulated disasters.
  2. Scheduled disaster simulation. It is one thing to have disaster mitigations plans in place, it is another to know whether they actually work. Disciplined organizations will run through disaster scenarios to verify how well their mitigation strategies work in practice. For example, to test whether your power outage emergency plan works you would purposely simulate a power outage at one of your data centers and then work through your recovery plan. Like fire drills, these simulations should be done on a regular basis so that staff members build up the “body memory” required to act swiftly and appropriately in an emergency. The advantage of a scheduled disaster simulation is that you knowingly run it at a time where you will have minimal impact on your stakeholders.   A disadvantage, at least when people are informed of the simulation ahead of time, is that people are mentally prepared for the simulation and aren’t caught unaware and thereby you don’t simulate the real level of stress that people would be under during an actual emergency.
  3. Random disaster simulation. Very disciplined organizations will implement a service within their operational environment that causes problems such as server or service outages at random times. An example of this is the Chaos Monkey functionality in Amazon’s Web Services (AWS) offering, functionality that is being implemented within many organizations now. The Chaos Monkey injects random problems into production to verify that the IT operations organization is capable of overcoming them. This is done to verify that your solutions really are able to automatically recover from problems and failing that at least operators are alerted to the problem.

As you would expect, truly disciplined organizations have adopted all of these strategies.

Related blog postings:

 


Posted by Scott Ambler on: February 17, 2015 02:33 AM | Permalink

Comments (0)

Please login or join to subscribe to this item


Please Login/Register to leave a comment.

ADVERTISEMENTS

"Familiarity breeds contempt -- and children."

- Mark Twain

ADVERTISEMENT

Sponsors