Introducing Guest Blog Author:
Gene Komondor
As you know, my goal for this blog is to be a free resource to everyone tasked with keeping others safe. While I typically post school safety-related content, emergency preparedness is a universal topic and so I often reach out to Subject Matter Experts in my circle for additional insight and expertise. Gene Komondor is a FEMA-certified master exercise practitioner with more than 20 years experience in healthcare and emergency services. This article was published originally on Campus Safety Magazine, but is such a valuable resource that it deserves a home on my blog.
For more information on Gene and the work he does, visit www.OneStarTraining.com.
In sports there is a saying that “You play like you practice.” The same could easily be said for most agencies, organizations, businesses, schools, and other institutions. An organization that does not adequately train its personnel and does not evaluate the effectiveness of that training and the related capabilities attained (or not attained) by its personnel from that training has no right to expect its personnel will perform up to the standards required by the organization.
In the 9/11 Commission Report, investigators determined that the successful response at the Pentagon was due in part to the fact that “many fire and police agencies that responded had extensive prior experience working together on regional…training exercises.”
It’s not only the good guys who are investing time, money and resources in training and exercises. In order to test various assumptions about airport security and other operational aspects of their plans, the 9/11 terrorists “…used America as their staging area for further training and exercises – traveling into, out of and around the country…” claims the report.
Less than four years later, the White House investigation into the flawed response to Hurricane Katrina determined that “training and exercise programs did not prepare all levels of government.” Investigators found that “Federal, state and local entities were neither properly trained nor exercised,” thus contributing to the poor response to one of the most destructive natural disasters in U.S. history.
Why are the lessons learned from these and other incidents – lessons that are often learned at great cost in terms of lives and property – not incorporated into emergency plans and made part of the future planning, training and operations of emergency services agencies? Certainly there are ample legal reasons to do so, such as increased liability exposure.
Preparedness Begins With Planning
In DHS’ National Preparedness Cycle, planning is described as the process utilized to establish priorities, identify levels of performance and capabilities, and to set standards for evaluating the level of capabilities. An effective planning process should result in a comprehensive, all-hazards emergency operations plan (EOP) for a jurisdiction, organization or other entity. The EOP provides a framework that enables personnel to respond effectively to a wide range of incidents.
The planning process usually starts by identifying and analyzing the hazards and threats present in the area covered by the plan, and the likelihood of those hazards resulting in incidents that affect that area. When complete, the EOP identifies not only the incidents most likely to occur, but also the performance capabilities required by personnel responding to those incidents. These hazards and related responder capabilities will form the basis of the organization’s training and exercise program.
Training is the process within the National Preparedness Cycle that provides individuals with the knowledge, skills and abilities required to perform tasks necessary to achieve and maintain a response capability. Both planning and training require substantial investments of time, money and resources.
Exercises Test Preparedness Levels
A third component of the National Preparedness Cycle allows an organization to accurately evaluate the return it receives from its planning and training investments: exercises. An exercise is the activity of choice to assess an organization’s preparedness level. It is a structured learning environment in which an organization can evaluate the following aspects of its preparedness:
-
-
- The accuracy of its plans
- The clarity of its objectives
- The effectiveness of its training
- The level of capabilities achieved by its personnel
-
Exercises are not standalone events. They are a key component of a comprehensive approach to planning, training, evaluation and learning. Exercises play an important role in connecting an organization’s planning and training functions with the related improvement process upon which the organization bases its future planning, training and response efforts.
The U.S. Department of Homeland Security’s Exercise and Evaluation Program (HSEEP) divides exercises into two categories: discussion-based and operations-based.
Discussion exercises are more strategic and focus on planning and policy issues and validating plans, while operations exercises are tactical in nature, with participants actually demonstrating the procedures and activities outlined in their plans. The table top exercise is the most common of the discussion exercises. Operations exercises include drills, functional and full scale types.
Approach Gets Progressively Complex
HSEEP utilizes a progressive building-block approach to an exercise program, beginning with the discussion exercises and increasing in scope and complexity until culminating with a full scale exercise.
In order to provide value to participants, an exercise should possess two distinct and measurable qualities. It must be capabilities-based and objectives-driven. These qualities form the basis for the structure of the entire exercise and are directly linked to the development of the following exercise management tools:
-
-
- Core capabilities
- Exercise objectives
- Critical tasks
- Exercise evaluation guides (EEGs)
- After action report (AAR)
- Improvement plan
-
A capabilities focus allows participants to demonstrate the knowledge and skills required by the organization’s EOP. Assessing participant capabilities provides the organization with the information needed to improve its training programs. An objectives-driven focus ensures that the critical tasks to be performed by participants are consistent with the goals of the organization(s) participating in the exercise.
Objectives should be designed to allow participants to demonstrate the level of capability they have attained from their training. Comparing the exercise objectives to the actions exhibited (or omitted) by the participants is a clear way of evaluating the quality and success of that training. When designed correctly, an exercise allows participants to demonstrate targeted capabilities and achieve required objectives.
Have You Met Your Objectives?
Evaluation is the most important aspect of an exercise. It is the process that determines if capabilities are attained and objectives are met. Effective evaluation begins with the exercise design process and includes four key components:
-
-
- Planning for exercise evaluation;
- Observing the exercise and collecting exercise data during exercise conduct;
- Analyzing the data collected to identify strengths and areas for improvement; and
- Reporting exercise outcomes in an after action report (AAR).
-
Evaluators use exercise evaluation guides (EEGs) to observe and collect data related to the exercise objectives and the corresponding core capabilities. Exercise designers and evaluators customize these forms in order to focus on the specific critical tasks that must be displayed by participants to confirm that they have achieved the capabilities targeted in the exercise.
EEGs include four elements to facilitate observation and evaluation: the objective; the related overall core capability; the organization’s targeted capabilities and the critical tasks.
The objective is the specific outcome the organization wants to achieve during the exercise. It is usually based upon one or more of the 31 core capabilities listed in the National Preparedness Goal. Achieving these capabilities is based upon the performance of associated critical tasks. When linked to operating conditions and performance standards, critical tasks provide the primary source of learning objectives for training and exercises and provide input to planning and performance evaluation.
Critical tasks allow evaluators to assess the level of participant capabilities in two of the primary learning domains: cognitive and psychomotor. Participants demonstrate what they know (cognitive) by what they can do (psychomotor). For operations-based exercises, designers must ensure that conditions in which critical tasks are performed are as realistic as possible.
Targeted capabilities and critical tasks may be drawn from a number of sources, including the National Preparedness Goal and the five National Planning Frameworks, or from a threat and hazard identification and risk assessment product (THIRA). The best source, however, may be from an organization’s own plans and evaluations from previous exercises.
Fix Problems Revealed in After Action Review
At the conclusion of the exercise the evaluations are analyzed. This is accomplished through an after action review (AAR) process, the main focus of which is the analysis of core capabilities. The analysis should consider the following questions:
-
-
- Were the capability targets met? If not, what factors contributed to this result?
- Did participant discussion or activities suggest the critical tasks met capability targets? If not, what was the impact or consequences?
- Were participants familiar with current plans, policies, and procedures, and did these documents support critical tasks and capability targets?
-
The analysis should produce corrective actions to be implemented by the participating organization in order to enhance its overall capabilities. These corrective actions can assist an organization with making changes to plans, procedures, organizational structures, management processes, equipment, resources and the training related to these items in order to improve performance. It can also help to identify lessons learned that can be applied to similar situations in the future.
Unfortunately this process is not always implemented. Even after Katrina, investigators found that “too often, after action reports for exercises and real-world incidents highlight the same problems that do not get fixed.”
Paper Plans Can Only Go So Far
Until a comprehensive exercise program validates its assumptions and related concept of operations, even the most well written emergency operations plan remains only a theory. To ensure an adequate return on its investment in planning, training and its personnel, no organization should leave exercises out of its preparedness cycle.
