07.11/02

Foreword

Management practices and operational procedures for mine action are constantly evolving. Improvements are made, and changes are required, to enhance safety and productivity. Changes may come from the introduction of new technology, in response to a new mine or ERW threat, and from field experience and lessons learned in other mine action projects and programmes. This experience and lessons learned should be shared in a timely manner.

Technical Notes for Mine Action (TNMAs) provide a forum to share experience and lessons learned by collecting, collating and publishing technical information on important, topical themes, particularly those relating to safety and productivity. TNMAs complement the broader issues and principles addressed in International Mine Action Standards (IMAS).

The preparation of TNMAs follows a rapid production and approval process. They draw on practical experience and publicly-available information. Over time, some TNMAs may be 'promoted' to become full IMAS standards, while others may be withdrawn if no longer relevant or if superseded by more up- to-date information.

TNMAs are neither legal documents nor IMAS. There is no legal requirement to accept the advice provided in a TNMA. They are purely advisory and are designed solely to supplement technical knowledge or to provide further guidance on the application of IMAS.

TNMAs are compiled by the Geneva International Centre for Humanitarian Demining (GICHD) at the request of the United Nations Mine Action Service (UNMAS) in support of the international mine action community. They are published on the IMAS website at www.mineactionstandards.org.

Introduction

We cannot really understand and manage the work we do unless we measure it. This means using indicators, some of which may be important enough to be designated as Key Performance Indicators (KPIs).¹ KPIs help us understand performance. They are essential, not only for the day to day management of operations, but also for the monitoring and evaluation of operations.

Within mine action KPIs are primarily operational analysis tools. They provide a framework for analysing data. Without appropriate data subject to credible Quality Control (QC), a KPI will be of limited or negative value. Indicators are dependent on the collection of accurate and relevant data from the field. This requires well designed forms that capture data whilst not overburdening field operators. There is a finite amount of data that can practically be collected from the field. Invariably, not everything that is desirable to know is practicable to collect. Overambitious levels of data collection can result in lower quality data collected. What data is prioritised for collection is a choice. Operations managers should be clear about exactly what they want to measure and ensure that no superfluous data is collected. Field staff ought to fully understand that the data they collect is fundamental to their operational efficiency and effectiveness and not simply an Information Management (IM) concern.

Operational KPIs should be routinely used by operational staff. Furthermore, data should be subject to effective QC, by both operations as well as IM staff.

In the history of mine action use of KPIs has been limited. Often this has been justified by a concern that statistics, including KPIs, can easily be misrepresented or not presented in a fair context. This is usually summarised by the apocryphal phrase “lies, damned lies and statistics”. This is not a justification for not using KPIs. It is a justification for using them appropriately by making sure the supporting data is as accurate as possible, and that the KPIs are always viewed strictly in context.

1. Scope

This TNMA focuses on a suggested group of ‘Key’ Performance Indicators that are likely to be useful to mine action managers engaged in field operations.

2. References

A list of normative references is given in Annex A. Normative references are important documents to which reference is made and form part of this TNMA.

3. Terms, definitions and abbreviations

A complete glossary of the terms, definitions and abbreviations used in the IMAS series is given in IMAS 04.10.

In the IMAS series, the words 'shall', 'should' and 'may' are used to indicate the intended degree of compliance. This use is consistent with the language used in the International Organization for Standardization (ISO) standards and guidelines:

1. 'shall' is used to indicate requirements, methods or specifications that are to be applied in order to conform to the standard;

2. 'should' is used to indicate the preferred requirements, methods or specifications; and

3. 'may' is used to indicate a possible method or course of action.

The term 'National Mine Action Authority' (NMAA) refers to the government entity, in a mine-affected country, with the responsibility for the regulation, management and coordination of mine action.

Note: In the absence of a NMAA, the United Nations, or some other recognised international body, may assume some or all of the responsibilities, and fulfil some or all the functions, of an NMAA.

The term ‘Mine Action Organisation‘ refers to any organisation (government, military, commercial or NGO/civil society) responsible for implementing mine action projects or tasks. The mine action organisation may be a prime contractor, subcontractor, consultant or agent.

The term ‘Common Counting Rules‘ (CCRs) refers to a standardised way of defining and counting a given unit of mine action operational data. Use of CCRs minimises the risk of outputs being reported in an inconsistent way. CCRs allow data comparison through the application of a ‘like-for-like’ principle. CCRs exist for cancelled, reduced and cleared m2.

The term ‘Context Capture‘ refers to the collection of relevant contextual information concerning survey and clearance activities at the point of data entry. Such information, where appropriate collected by means of binary Yes and No questions, assists reasonable comparison of performance.

The term ‘Compliance’ refers to an adherence to the requirements of agreed standards or procedures. In mine action compliance typically involves adherence to standard operating procedures that comply with a National Mine Action Standard.

The term ‘Efficiency‘ refers to a measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results (outputs and outcomes).

The term ‘Effectiveness’ refers to the extent to which the intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.

The term ‘Performance’ refers to the achievement of a given task measured against an agreed standard.

The term ‘Progress’ refers to the proportion of a defined task that has been completed.

The term ‘Results’ refers to the extent to which interventions make a difference, sometimes termed ‘impact’ or ‘outcome’.

The term ‘Risk’ refers to combination of the probability of occurrence of harm and the severity of that harm [ISO Guide 51:1999(E)].

4. Background

4.1 Performance

Performance may be defined as the achievement of a given task measured against an agreed standard. The concept of ‘performance’ covers a range of formal, informal and perceived parameters. Performance relates to the extent to which an organisation or individual carries out their work, on time, on budget, safely, in an environmentally friendly way, and in accordance with contractual and legal requirements, (such as adherence to international or national standards).

Any aspect of performance can be measured, and an associated indicator generated. All indicators require robust and practical data collection, typically in the field. In every case it is the responsibility of managers to decide which indicators, alongside the underlying data, have the most value in terms of demonstrating ‘performance’. Those indicators are ‘Key’ Performance Indicators – the KPIs.

4.2 Risk management

Risk has been defined as the effect of uncertainty on objectives. Collecting data and then analysing it through the means of operational KPIs reduces uncertainty and is essential for operational risk management. The key is to make decisions based more on evidence rather than perception. This requires accurate and timely data, appropriately analysed through the lens of KPIs.

4.3 Types of performance indicators

Indicators can be developed for most aspects of performance in any organisation and in relation to any activity, product or objective. In most cases the indicators will relate to one or more of the following broad categories:

• Effectiveness: these indicators may be binary – an objective was achieved, or it was not, or they may be expressed as percentages, indicating the proportion of an objective that has been achieved.
• Efficiency: these indicators may be seen as the level of effort, resources or assets used in achieving objectives. Efficiency indicators are usually ratios – input to output; success to failure; acceptance to rejection; etc.
• Progress: these indicators may be set against time objectives and correlated with other time metrics. Progress can be represented in several ways e.g. using areas cleared or released against time and/or asset and method.
• Compliance: these indicators direct operations management’s attention to where additional effort should be focused.
• Results: these indicators may be seen in terms of more immediate outcomes, numbers of people benefiting from an intervention, or wider ranging and long-term outcomes and impacts. Developing KPIs linked to impact has historically been challenging for mine action, not least since it requires ongoing data collection after field operations have finished. This is often beyond the resources of most operators. This issue falls outside the scope of this TNMA but is an important aspect of understanding and improving mine action operations.
• Financial: financial indicators address similar aspects of activities and operations (efficiency, effectiveness, compliance, etc.) but do so in the specific context of money. Financial indicators include quantitative data analysis to derive the fiscal efficiency of activities, assets, methods, projects and programmes.

Some KPIs can provide information relevant to more than one of the above categories. An effectiveness KPI can also legitimately be counted as an efficiency KPI or progress KPI. For example, while m2/Explosive Ordnance (EO) item helps us understand the efficiency of clearance activity, it also provides an indication of the effectiveness of survey and operational planning that targeted the clearance. In many circumstances the KPI type and subsequent analysis are interdependent. Categorising a KPI as exclusively as an indicator of either efficiency or effectiveness is unlikely to be constructive.

5. Principles

KPIs rely on data. Collecting the required data often requires organisational effort. While the use of electronic forms allows more data, of a higher quality, to be collected, typically it is only feasible to collect a finite amount of data about operations. Choosing what data is collected should be done in conjunction with choosing what performance should be measured by means of KPIs.

The design of clear and usable collection methods is fundamental to the success of KPI data gathering. Forms should be carefully designed and be user-friendly to promote their use. The collection of quality managed and usable data should in itself be viewed as a KPI by mine action managers.

The selection of KPIs within this TMNA is not definitive or exhaustive, but provides a base to establish, maintain and use KPIs that are relevant to core aspects of survey, clearance and stockpile destruction operations.

5.1 Consultation and participation

Field personnel should understand that appropriate data collection is an essential function of their role. This understanding should be underpinned by field personnel making daily use of KPIs so that the important collection of the data that feeds such analysis is accepted as not only a necessity, but becomes a norm. In short, if KPIs are relevant and help field staff perform better, and if managers closely monitor data collection, field staff will take more care in collecting the data that feeds the KPIs. The importance of accurate and relevant data collection should be reflected in the job descriptions of all relevant field staff and regularly emphasised by managers.

5.2 Viewing KPIs in context

KPIs should be interpreted strictly in context. Taken in isolation they can give rise to misleading or invalid conclusions. For example, comparison of the m2/mine clearance figures between a pattern and a nuisance minefield could lead to a possibly unjustified interpretation that one was inefficient compared with the other, when one may just have presented a greater survey and clearance problem than the other. One common example is that of nuisance minefields, which typically give high m2/mine clearance figures, but which also need to be reduced and cleared in any case.

Different field conditions (i.e., with varying contamination and vegetation levels) may explain significant variation in performance figures at apparently similar sites. This is also the case with other environmental factors such as slope angles, rocks, metal contamination levels, magnetic characteristics of the soil, rock content of soil, and the availability of minefield records for survey.

Describing the context:

The following basic elements of context capture at the point of data entry should be considered for inclusion in operational forms by field operators. Capturing even elements of this basic list will allow those examining data with no connection to the operations on the ground, and thus no intuitive knowledge of a given context, an understanding of why a given performance is indicated in certain KPI figures. The unit of measurement for these context descriptors is usually a site. Where conditions alternate significantly within a given site, the operator may consider splitting the site for reporting purposes. Where possible simple Yes/No questions are recommended, although some context can only be captured with a numerical answer (e.g. depth, topography):

1. Significant vegetation coverage prior to processing Y/N?
2. Hard Soil Y/N?
3. Mechanically processed Y/N?
4. Water required for excavation? Y/N?
5. Significant metal contamination Y/N?
6. Method of detection, (if electronic detector – model and configuration. E.g. Minelab F3S Black
7. Endcap).
8. Topography (estimated average °slope on site)
9. Prone to adverse weather (floodplain, exposure to strong winds or extreme cold/heat) Y/N?
10. Clearance depth on site.
11. Depth of items found (cm to top of item).
12. Method of QC (Animal Detection System (ADS)? Manual? Combination Etc?)
13. Local interference –proximity to habitation/ proximity to usable public routes (meters).
14. Significant rubble Y/N?
15. Man made obstacles (AT ditches etc) Y/N ?
16. Minimum metal mine contamination Y/N ?
17. Category/Sub-Category and Model of EO found in accordance with IMAS 05.10.2
18. Potential for stand-downs due to insecurity Y/N?

Some mine action processes directly modify the context. For example, mechanical preparation of ground affects both soil structure and vegetation cover. Therefore, context should be recorded as per the condition of the site on the first day of Technical Survey (TS) or clearance.

5.3 Consistent and comparable

KPIs typically provide managers with the opportunity to compare performance: to compare it over time, between different locations and between different organisational elements. Such comparisons are only valid if KPIs generated at different times, in different places, and by different reporting entities, make use of data that is collected and reported in the same way, and analysed using the same methods. Therefore, data should be subject to CCRs.

Like-for-like KPIs:

Ensuring consistency requires the application of the ‘like-for-like’ principle. This means that a KPI describing asset performance over time, for instance, must use the same basic definitions of asset and time, if it is to yield comparable results from different reporting agencies.

Thus, a KPI called ‘area cleared/team/week’ presents ‘like-for-like’ problems. Team size may vary from one organisation to another and from one day to another (reflecting factors such as sickness and leave cycles). The length of the working week may vary between organisations and as a result of factors such as time lost due to weather, interruptions arising from training, security incidents and other factors. Too many of the data items included in such a KPI are open to interpretation and variation. Using this KPI 

reported by several different organisations or elements would not be a reliable way of comparing performance.

Instead, a KPI called ‘area cleared/deminer/hour’ enables ‘like-for-like’ comparability. A deminer is a deminer; there is no scope for misunderstanding about what this means. An hour is a scientifically defined unit of measure. All organisations have the same understanding of what is meant by an hour. Using this KPI brings a high level of confidence that it will be collected, reported and analysed in the same way by different organisations and over time. It will allow meaningful comparison.

5.4 Common counting rules for land release outputs

Even when there is confidence that the ‘like-for-like’ principle is satisfied, there may still be issues relating to how some of the data in a KPI is counted. Failing to define counting rules may compromise ‘like-for-like’ aspects.

Consider a square metre within a Suspected Hazardous Area (SHA). During the Land Release (LR) process that square metre could be subjected to a number of the following processes:

• Non-Technical Survey (NTS) – it will most likely have been subject to this process in order to have been included within an SHA in the first place;
• TS using one or more of:
  • Mechanical processing;
  • ADS processing;
  • Manual investigation; and/or
  • Other intrusive methods and tools.
• Clearance using a combination of; o Manual clearance;
  • Mechanical processing; and/or o ADSprocessing.

Leading to release through:

• Cancellation of some land (on the basis of NTS assessment);
• Reduction of some land (on the basis of TS processing); and/or
• Clearance.

The same square metre could be counted once (as an all up ‘area released’ figure for instance), or there might be good reasons to count it multiple times (e.g. to understand what proportion of ground was mechanically prepared prior to search with ADS and/or manual deminers).

An area that has been subject to a combination of TS and clearance might all be reported as entirely ‘cleared’ by one operator, but another might choose to report only the land physically investigated as ‘cleared’, while the rest is ‘reduced’. Dissatisfaction within the mine action sector about how some operators are perceived to ‘game the system’ by choosing potentially advantageous interpretations of how to count some data has been a recurring theme over many years. Clear CCRs (supported by appropriate attention within monitoring systems) are a way to address such situations.

The following CCRs for LR outputs should be followed:

Land Release Output	Common Counting Rules (CCRs) Definition
Cancelled	Areas designated as SHA/CHA, or part thereof, which have not been physically processed in any way, and meet set cancellation criteria. This includes areas re-designated as either SHA/CHA as the task progresses. Cancellation may be done at any stage of the LR process.
Reduced	The following areas should be reported as reduced after conducting TS: An area that has been physically processed using an appropriate technical intervention, be that clearance or an accredited TS process, to identify evidence of whether EO contamination is present or not; and Areas within the previously defined SHA/CHA, where it has been concluded with sufficient confidence after applying the appropriate technical intervention, that the unprocessed areas do not contain a mine/EO hazard, and that the appropriate technical intervention applied could reasonably have been expected to identify evidence of the presence of contamination if it had been present. If EO contamination is identified in an area processed during TS, and a subsequent clearance method is applied to the area around the EO, then the area to which the clearance process is applied should be reported as clearance only to avoid double counting, and an area that meets the cancellation criteria should be reported as cancelled not reduced. Within an area reported as reduced, organisations shall record clearly where is processed and where is not. Areas processed shall be further disaggregated into those subject to manual, mechanical and ADS processing, with multiple processing of the same area by different assets recorded in detail. Organisations should be fully transparent in the amount of effort used to process an area as part of TS.
Cleared	The area where the organisation has applied a process, or processes, to ensure the removal and/or destruction of all EO hazards from the specified area to the specified depth. Where multiple processes are applied to the same area to achieve the clearance standard, the area shall only be reported once, although the processes that achieved clearance may be recorded in order to reflect the accumulated effort applied. These CCRs apply to the situation on site at the point of completion. The combined figures of cancelled, reduced and cleared should not exceed 100% of the plan surface area of the site, albeit some discretion is permissible for topography. Areas processed multiple times in the course of TS or clearance may be recorded as a sub-detail of either reduced or cancelled in order to capture the sum of efforts involved. However, the headline combined cancelled, reduced and cleared figure shall only record an area once. Figures detailing multiple processes should be clearly labelled as such. In this way the full extent of effort required to release land is recorded, but the final figures do not misrepresent the surface area released.

 

5.5 Integration with information management systems

Although historically not widely used in mine action, KPIs can be an important component of any IM system. KPIs are dependent upon the integrity of the data in that system. Operations staff should lead in the identification, development and maintenance of any KPIs, with the support of IM staff.

5.6 Quality management

The value and reliability of KPIs depends on the quality of the underlying data and the analysis of that data. Both IM and operations staff should be trained in data QC so that they are able to identify 

anomalies or inconsistencies. Strict QC of data by both IM and operations staff is essential for the production of meaningful KPIs that give a true reflection of operational performance. Both should be accountable for the quality and accuracy of the data within respective databases.

5.7 Data collection roles and responsibilities

Indicators require accurate data. The first step in generating any meaningful KPI is the collection of accurate and relevant data. The use of electronic devices such as tablets and mobile phones should be considered as significant enabler in the collection of such data.

The key to KPI data collection is the design of the actual forms, whether they are electronic or paper- based. Forms are among the most important documents in mine action and careful attention should be given to their design. A form should encourage the capture, as accurately and completely as possible, of necessary data. Long forms are rarely filled out carefully and there is a finite amount of data that operators can practically extract from the field. The imperative is to select the most relevant data to capture. It is a choice, with data priorities preferably reflected in the forms. Ideally, an operator should decide the KPIs they wish to measure while designing the forms that will capture the all-important data in the first place.

5.8 Gender and diversity disaggregation

Selected KPIs should be disaggregated by sex, age and other diversity dimensions important in the specific context (for example, ethnic group, displacement status, etc.).

Understanding differences in all aspects of mine action, as it relates to different genders and ages, as well as to people with disabilities and other diverse groups, is necessary to ensure that KPIs are developed to ensure that intended outcomes meet the needs of the population.

6. Land release KPIs

LR may be deemed effective when it clears land that is actually contaminated. It may also be deemed effective if the land released is actively used by the local population and is efficient when it does so using the minimum time, cost and resources to do so.

6.1 Core land release KPIs

The following guidance is designed to aid in the selection and maintenance of KPIs related to the subject of LR.

M2/EO item:

KPI name	m2/EO item
Input data	Area measured in m2 Number of EO items (by ordnance category)
Guidance	The number of EO items should be disaggregated by ordnance category and model reflecting the significant differences in this KPI’s values depending on EO type and employment. For certain ordnance such as minimum metal mines, which are not disaggregated in IMAS 05.10, capturing detail will be important in understanding the KPI.
Interpretation	The KPI is strongly influenced by the type of EO present at the site and the method of employment. Landmines in large numbers in regular, well-recorded rows are likely to enable tight definition. Operators may expect low m2/EO item (perhaps in single figures) for this KPI under such circumstances. More irregularly used ordnance, such as nuisance/low density minefields, may be distributed in a wider, harder to define way. This KPI may yield high m2/EO item under such circumstances. Landmines in very small numbers, typically below 10, are likely to present some of the hardest circumstances to define and often yield KPI numbers in the thousands of m2/item. These are often referred to as nuisance or low density minefields. m2/EO item is also influenced by both the effectiveness of the survey and the efficiency of the clearance of a given hazardous area. While differing m2/item figures can often be explained by the type of contamination (e.g. pattern vs nuisance minefield), m2/item remains one of the most basic methods of identifying effective survey and efficient clearance.
Display	Charting this KPI as ‘area/item (Y axis) versus items found (X-axis)’ offers the potential for benchmarking of performance for survey and clearance of some explosive ordnance types (especially landmines and submunitions). The chart for m2/mine below shows a comparison of results between regions and countries (one country in blue, one in red in the chart below) and between different operating organisations. Use of different data sets to develop different trend lines during different time periods may also yield indications of change in performance over time. In the example below, the red trend line relates to more recent operations, the blue line to older ones. The fact that the red line sits below the blue line could imply an improvement in performance between the two time periods, or could indicate that easier sites, where the contamination is easier to find, were left until later in the project. The KPI also lends itself to other forms of charting such as using a bubble (sized in proportion to the m2/item figure) for each site, with the number of items found as the Y-axis, and the date on the X-Axis. In the illustrative chart below (showing data of the sort associated with landmine clearance), a number of features are shown – at first the programme focuses on sites containing larger numbers of items. Over time the size of bubble reduces for a given level on the Y-axis, reflecting improvement in understanding of the task. Over time sites with fewer mines (which may have been a lower priority) attract more attention, resulting in larger bubbles. The implication is that while performance at sites with a particular number of items may improve over the lifetime of a programme, the indicator may appear to get worse as the harder, lower density sites eventually work their way up priority lists as a programme approaches the completion stage.

m2/asset/time (disaggregated by activity – mechanical, ADS, excavation, metal detector, Battle Area Clearance (BAC), etc.):

KPI name	m2/asset/time
Input data	Area measured in m2 Type of asset and method (manual metal detection; manual excavation ADS; mechanical flailing; mechanicalads sifting; etc., etc.) Working time
Guidance	It is essential that common rules for measuring and reporting the data inputs to this KPI are agreed and followed. There is potential for multiple counting of the same m2 (the same piece of land being subject to mechanical preparation, follow-on clearance by manual deminers, internal QC check, external QC check, etc.). There may be occasions when users wish to include multiple counting in their analysis, but if so, requirements must be clearly defined to avoid invalid results. The unit of time should be selected to avoid ambiguity. Per hour offers a universally understood quantity. The working day or week may vary from one organisation to another and be subject to change in light of other external issues (weather, security situation, etc.). If analysts wish to use the ‘working day’ or ‘working week’ in this KPI then the day and week must be clearly defined, and data must be normalised to provide ‘like-for-like’ comparability.
Interpretation	This KPI is strongly influenced by external factors including: Soil and topography; Vegetation; Weather; and Metal contamination.
Display	A common charting approach for this KPI is to plot its value over time at a specific work site (as below). Note the unit of time used in this chart is a day rather than an hour. Other displays may be used to compare values between organisations or elements, but any such use should take care to recognise the sensitivity of this indicator to on-site conditions.

Asset time/EO item:

KPI name	Asset time/EO item
Input data	Type of asset and method (manual metal detection; manual excavation ADS; mechanical flailing; mechanical sifting; etc., etc.) Working time Number of EO items (by IMAS 05.10 ordnance category)
Guidance	Measurement units for ‘asset’ need to be clearly defined. It is recommended that base units should be used – deminer; animal; machine, etc. rather than aggregated units – section; team; etc. Measurement of time should use recognised scientific units – hour, minute, etc., rather than aggregated units that may different meanings to different individuals and organisations – working day, working week; etc. What constitutes an EO item should be defined – entire objects; fragments containing explosives; components; etc.
Interpretation	This KPI provides an indication of the level of effort required to find EO items. It can provide a useful ‘rule of thumb’ to support operational planning estimates. If it is known that the KPI for work in an area of regular minelaying is 1.5 deminer days/mine, then if there is an expectation of finding 200 mines at another site planners can expect 300 deminer days to complete work at the site. The KPI can be used in areas where average contamination levels are known – if the average incidence of EO/Ha can be estimated during planning then there is potential to use this KPI, from other similar areas, to support estimates of asset requirements. This KPI is associated with the m2/EO item KPI.

Cleared versus released ratios:

KPI name	Released m2/cleared m2
Input data	Area released Area cleared
Guidance	Definitions of area type need to be clear and unambiguous.
Interpretation	This KPI indicates the efficiency of the clearance process and the effectiveness of the survey that targeted the clearance. If only a small proportion of a released area required costly and time-consuming clearance efforts then that may be an indication of well targeted, efficient LR decision-making. Low clearance figures in proportion to the overall area released typically indicate clearance of a pattern minefield where the operator can demonstrate little or no mine movement. In some pattern minefields this figure can be as low 7-25% of the hazardous area. Some caution should be exercised with the released m2/cleared m2 KPI. If NTS has resulted in the inclusion of excessive or unjustified areas within SHAs, then this KPI may appear to be good (low clearance effort to release large areas), but may actually be indicative of poor earlier survey.
Other variations	Ratios of released m2/surveyed m2 can provide useful indications of the effectiveness of survey operations, indicating situations where the size of a hazardous area may have been misjudged. Monitoring such indicators across a programme and over time is recommended.

7. Planning and progress KPIs

The percentage of task objectives achieved against time is one of the simplest indicators, however, it requires managers to predict both output at a site and the expected duration of work, which is not always a straight-forward task. Even where no deadline is set, or where there may be some uncertainty about the volume of work expected at the site, mine action managers should be encouraged to set figures and monitor progress toward them, even if time and output targets need to be updated as experience in a given operating environment grows. Progress against time indicators tend to become more accurate as organisations gain experience in a given operating environment.

7.1 Core planning and progress KPIs

Comparisons between when tasks are expected to be implemented against when they actually take place, and of planned duration against actual duration, provide indications of the reliability of planning processes. As with any KPI, there may be good reasons for differences between planned and actual activity, but managers should generally expect the reliability of planning to improve over time under similar circumstances.

Basic progress line against objectives:

KPI name	Progress against objectives
Input data	Planned output Planned duration Actual start date Actual output
Guidance	As with all ‘before and after’ indicators, analysts should define which set of planned data provides the baseline for this indicator (see below).
Interpretation	This KPI is a specific ‘before and after’ comparison. It provides an indication of the reliability of planning predictions and provides an ‘early warning light’ for managers in case action is required to maintain progress. In the example below, when work started at the task site progress was below the target line and was falling further behind, implying that the task would be completed well behind schedule. Management action was taken (in this case by deploying more deminers to the site) so that the progress rate increased. Once the task was well above the target line (and by implication likely to finish ahead of schedule), managers were able to reduce the number of deminers on site to a level that would deliver the task on schedule.
Additional uses	The basic progress line is the simplest planning tool when used correctly and in context. The addition of set and defined objectives allows for the KPI to reflect the baseline of the programme and can indicate areas where more detailed KPI analysis should be undertaken.
Display	Progress rates can be displayed as numeric percentages or as time ahead or behind schedule (as is often done using GANTT charts), but the simplest way to display is the progress chart (as below). This has the advantage of showing the trend in progress, allowing managers to identify both any need for management action and the effects of that action once taken.

Before and after comparisons:

KPI name	‘Before and after’ comparisons
Input data	Details of what was expected (area, time, material quantity) Details of what was found/happened (like-for-like comparable to the details of what was expected
Guidance	‘After’ data is usually relatively easy to identify and define. m2 cleared is measured and recorded as part of site completion documentation. ‘Before’ data, constituting a prediction of some form, can be harder to identify. Predictions may be made at several stages in a project life-cycle: within a proposal; during one or more planning reviews during project implementation; immediately prior to the start of work at a site; revised predictions reflecting new information gained during work at a site. Analysts must decide which comparisons are most meaningful and then define which data constitutes the ‘before’ data.
Interpretation	‘Before and After’ comparisons are used to assess the reliability and quality of planning and decision-making processes. They compare ‘what was expected’ with ‘what happened’. Comparing the results of survey (which seeks to detail ‘what is expected at the site’) with the results of clearance (what was actually found), provides an indication of how well the survey produced a ‘true’ picture of the site. In effect this is one potential element of QC of the survey. Over a period of years, within an individual mine action programme, and to some extent across the mine action sector, it might be hoped that the alignment of such predictions would improve as survey capabilities improved.
Addtional uses	‘Before and after’ comparisons are useful in many aspects of mine action including: Financial – to understand the reliability of budget projections; Levels of effort – to understand how well planners predict operational demands; Progress – indicating whether actual rates of progress match expected ones; and Results – comparing expected results (e.g. land will be used for agriculture) with what actually happened (the land was not put to use).
Display	Variance charts show differences between predictions and actual results (see also date and duration variations and progress against objectives KPIs).

Date and duration variation:

KPI name	Date/duration variation
Input data	Planned start and end dates Actual start and end dates Working days policy
Guidance	Ensure that ‘planned’ data is clearly identified. It could be data extracted from a proposal or planning document; or it could reflect the plan immediately prior to work starting. The planned and actual durations will normally be based on the difference between the respective start and end dates. Analysts may wish to use calendar days for comparison, or working days, in which case CCRs need to be established. Working days could be calculated on the basis of a known working day rate (6 days per calendar week for instance), or it could be a summation of days when work actually took place (discounting those when weather, security or other factors intervened). Policies on how to handle part days, or whether to count person hours instead, as an indication of duration, may all need to be considered. Any policy must be applied to both planned and actual data.
Interpretation	This KPI is a specific ‘before and after’ comparison. It provides an indication of the reliability of planning predictions. Date variations focus on scheduling. In a ‘perfect’ plan everything would happen exactly when expected. In reality, most practical work is subject to change, reflecting external factors. Duration variations focus on level of effort projections. The decision to deploy a second team to a site may result in a shorter scheduling duration, while nevertheless requiring increased resources. Combinations of date and duration KPIs provide a balanced view of planning effectiveness. As with all KPIs, there may be good reasons why some variations occur. Irrespective of such reasons managers need to know how good planning processes are at predicting project and programme schedules, effort levels and durations.
Display	Date and duration variations are most easily charted as simple numbers (positive or negative, depending on whether activities happened sooner or later or lasted a shorter or longer time) as below. Indicator values can be associated with discrete project elements (most obviously a task). Individual or average values may allow useful comparison between different organisations or elements within a larger programme.

8. Open burning (OB) and open detonation (OD) KPIs

Where programmes and projects are required to conduct significant EO disposal operations the measurement of demolitions stores and techniques data alongside performance provides a useful planning guide and performance metric.

8.1 Disposal effectiveness

Disposal effectiveness:

KPI name	Disposal effectiveness
Input data	Number of EO items presented for OB/OD Number of EO items requiring further disposal action following OB/OD
Guidance	It is essential that common rules for measuring and reporting are agreed and followed. The number and type of EO items should be disaggregated by IMAS 05.10 ordnance categories. More detailed analysis will often require further disaggregation by individual ordnance model.
Interpretation	Most directly this KPI reflects the efficiency with which explosive destruction techniques perform against a specified ordnance type. The KPI is sensitive not only to method/technique, but may also be influenced by the competence of the individual Explosive Ordnance Disposal (EOD) technician.
Additional uses	This KPI lends itself to planning and logistics programming. It may also indicate trends in particularly effective, or ineffective, explosive stores or techniques.
Display	This KPI can be displayed as a basic ratio or percentage of total destruction figures against explosives stores used. KPIs values can be associated with: Different EO types; Different EOD techniques; and Different EOD elements/operators.

8.2 Use of materials

Disposal effectiveness:

KPI name	Disposal effectiveness
Input data	Amount of explosive stores used relative to devices destroyed
Guidance	It is essential that common rules for measuring and reporting are agreed and followed. The number and type of EO items should be disaggregated by IMAS 05.10 ordnance categories. More detailed analysis may require further disaggregation by individual ordnance model.
Interpretation	This KPI reflects the efficiency with which explosive stores are being used against specific EO targets. It can also indicate the appropriateness of the technique being used relative to the nature of contamination and the type of clearance task. For example, few detonators being used for many more armed submunitions found in separate spot tasks might indicate armed items are being moved that should not be. This KPI should be viewed in the context of other destruction KPIs to form a full understanding of the system in place.
Additional uses	This KPI lends itself to planning, risk management and logistics programming. It may also indicate trends in particularly effective, or ineffective, explosive stores or techniques.
Display	This KPI can be displayed as a basic ratio or percentage of total destruction figures against explosives stores used. KPIs values can be associated with: Different EO types; Different EOD techniques;and Different EOD elements/operators.

9. Safety KPIs

Accident and incident rates:

KPI name	Accident and incident rates
Input data	Number of hours worked (disaggregated by personnel category if necessary) Number of accidents and incidents (near misses) disaggregated by severity as: Fatal: one or more work related deaths; Major injury/ill health: fractures (other than fingers and toes), amputations, loss of sight, a burn or penetrating injury to the eye, an injury or acute illness resulting in unconsciousness, requiring resuscitation or requiring admittance to hospital for more than 24 hours; Serious injury/ill health: where the person affected is unfit to carry out his or her normal work for more than three consecutive days; Minor injury: all other injuries, where the injured person is unfit for his or her normal work for less than three days; or Damage only: damage to property, equipment, the environment or production losses.
Guidance	The various permutations of this KPI can be displayed for comparison as: Working time per disaggregated event – 250,000 working hours/major injury for instance; or Disaggregated event/number of workers/time period – fatalities/100,000 workers/annum for instance. One of the implications of this type of KPI is that mine action organisations need to collect, record and report data about working time. It may be desirable to disaggregate that data by type of worker (administrative, managerial, field, office, etc.), but in most other sectors it is the all up figure that is used. Authorities and managers should decide what level of accident/incident they wish organisations to report. An appropriate level may be that of fatal, major and serious injuries only (whether an accident or a near miss).
Interpretation	KPIs of this type are widely used in many industries and countries around the world. The specific way of constructing these KPIs does not matter greatly so long as all reporting parties make use of the same system. Rates are influenced to some extent by the operational context, including by the type of activities that different organisations engage in, but KPIs of this type are seen primarily as an indication of the effectiveness of an organisation’s risk and quality management systems.
Display	Safety KPIs can be displayed in tabular form, bar charts, etc.

10. Cost-related KPIs

The detailed compilation of financial KPIs falls outside of the scope of this TNMA.

Cost related KPIs can often be most straightforward when represented at the largest scale of the programme or project, e.g. total programme achievements against total programme budget. Fixed price contracts are usually the easiest to analyse for cost calculations. However the cost side of any ratio can be subject to manipulation.

When collecting and analysing financial data for cost KPIs it is essential that CCRs are established and agreed on. This is particularly important when allocating indirect costs during calculations of unit costs (e.g. cost/m2 cleared, or cost/EO item cleared, etc.).

Where CCRs are not well defined, especially the definition of direct and indirect costs, there is a large scope for the data to be distorted and invalid comparisons being made across programmes and between organisations.

11. Reporting KPIs

KPIs relating to reporting reflect the extent to which reports submitted to a recipient meet their requirements.

11.1 Proportion of reports accepted/rejected

Documentation acceptance rate:

KPI name	Documentation acceptance rate
Input data	Number of documents submitted Number of documents rejected/returned for further originator action
Guidance	Disaggregate by organisation/element
Interpretation	Reports, plans and records are important products of mine action systems. Submitting them accurately and in accordance with the recipient’s guidance is usually a requirement of the recipient (which could be a client, mine action centre, NMAA or donor). Submitting inaccurate or incomplete documentation is often an indicator of wider shortcomings in management systems. The reason for rejection of any documentation must be transparent and valid.
Additional uses	This KPI can be, and often is, combined with submission timeline KPI data as detailed below.
Display	This KPI can be displayed in tabular form (disaggregated by organisation/element/originator) or using other established charting methods (bar charts, etc.)

11.2 Timeliness KPIs

Submission timeliness:

KPI name	Submission timeliness
Input data	Required document submission date Actual document submission date
Guidance	Disaggregate by organisation/element
Interpretation	Reports, plans and records are important products of mine action systems. Submitting them on time is usually a requirement of the recipient (which could be a client, mine action centre, NMAA or donor). Submitting documentation late is often an indicator of wider shortcomings in management systems. Most organisations are likely to aim for a 100% timeliness rate.
Additional uses	Users may wish to combine this indicator with additional compliance aspects such as rejection rates, allowing data to be broken out into: On time, accepted; On time, rejected; Late, accepted; and Late, rejected.
Display	This KPI can be displayed in tabular form (disaggregated by organisation/element/originator) or using other established charting methods (bar charts, etc.)

12. Compliance KPIs

A number of simple quality management KPIs are of value to the sector. Typically these center on the occurrence and severity of nonconformities, identified by organizational unit (e.g., a team), date, and management system aspect; whether it relates to quality, safety, or the environment; which standard or Standard Operating Procedure (SOP) requirement has not been satisfied, etc.

Such KPIs have value when comparing teams in the same operating environment, looking for trends in performance, and when identifying aspects of standards or procedures that seem to be causing difficulties. However, there are potential problems when trying to compare differing quality management regimes in different countries.

There is also the issue that many National Mine Action Standards (NMAS) do not provide detail on the severity of different types of nonconformity and SOPs, while effectively listing working requirements, they too often do not specify clearly and consistently enough different levels of nonconformity.

Some contracts do specify nonconformity levels, and IMAS 07.40 provides general guidance. Managers developing and using compliance-related KPIs need to ensure that CCRs are in place if they are to avoid reaching misleading conclusions.

Compliance rates:

KPI name	Compliance rates
Input data	Number of nonconformities Severity of nonconformity (possibly including ‘observations’) Associated documented requirement Date of nonconformity Organisation, unit or element associated with the nonconformity
Guidance	The main value of compliance KPIs comes from the ability to disaggregate and associate nonconformities/observations with specific aspects and elements of a mine action system. Disaggregation categories include: Disaggregate by organisation/element; Disaggregate by documented requirement/source; and Disaggregate by severity.
Interpretation	Compliance KPIs need to be used carefully to avoid making operators nervous and potentially reluctant to be open about nonconformities. Anonymisation of specific data is often useful, allowing individual operators to compare their performance with an average, or benchmark, level, while avoiding the release of potentially sensitive data to all users. Compliance KPIs help establish benchmark performance levels and may indicate an organisation or element that is performing differently from that benchmark (either above or below). In either case management attention is likely to be merited to support improvement or to learn from success. Compliance KPIs may also indicate aspects of a management system (such as an individual NMAS or SOP) that appear to be causing problems for operators. If one operator generates nonconformities against a particular standard that may suggest an organisational issue; if all operators generate nonconformities against that requirement that may suggest that the requirement needs review, or that wider supportive action is required.
Additional uses	Nonconformity rates can be ratioed against other measures of the size of an organisation, or its deployed capacity, to allow valid comparisons between organisations or elements. Possible indicators could be: Number of nonconformities/team/month; Number of nonconformities/staff member/month; and Number of nonconformities/deminer/month. Other time intervals could be used, reflecting overall occurrence rates. While the bulk number of nonconformities may be useful to measure, compliance KPIs tend to yield most value when there is a degree of disaggregation, between different levels of severity of nonconformity for instance (critical, major, minor in IMAS 07.12 and 07.40), but also between nonconforming aspects (such as in relation to which standard, chapter or even specific requirement).

13. Other KPIs

The scope of this TMNA covers the main KPI metrics that are most broadly applicable over the majority of mine action operations. Additional KPIs specific to different operational contexts may be relevant. Such KPIs may include information or data relating to factors such as building search and clearance, Explosive Ordnance Risk Education (EORE), Victim Assistance (VA), advocacy and treaty compliance and victim/accident information.

A number of KPIs, especially those relating to compliance and nonconformity can be further disaggregated and analysed in relation to accidents (those events that give rise to harm including death). Analysis of the occurrence and prevalence of accidents, and the form of the resulting harm (injuries to hands, arms, face, torso, etc.) in relation to different techniques, methods and ordnance types provides important input in to continual improvement processes. Accidents may also be presented in terms of incidence relative to a number of working hours in order to give a quantifiable likelihood for risk assessment calculations.

Developing meaningful outcome indicators to improve understanding of the benefits that arise from mine action is a challenging task, and is the subject of wider investigation and development that falls outside of the scope of this TNMA. Even so, some indicators of immediate outcome, such as the comparison of the use of land following release, against the expectation of its use reflected in the prioritisation and planning process, can easily be adopted. If land is used in the way that was expected, or used in an alternative way that is demonstrably of benefit to the local community, it indicates that the overall tasking and LR process has been effective. Mine action operators should look for opportunities to measure and monitor outcome-related aspects of their operations.

14. Displaying and disseminating KPIs

Data for KPIs must be made available to operations managers, programme managers and other interested parties such NMAAs and donors. Information can be made available in ‘raw’ form – as tables of indicator values or, more usually, as charts, often combined to form dashboards.

KPIs should be displayed in easy-to-understand formats, ideally yielding additional trend or comparison information. KPIs displayed should include all those that are relevant, but should not be confused through the display of irrelevant information. For operations KPIs operations managers should decide how the information they need is presented. For more programme themed KPIs, it could be that programme managers lead on how such indicators are presented. It is possible that individual managers can tailor bespoke views in accordance with their information needs.

Dashboards are increasingly used across mine action and offer an effective way of easily monitoring selected KPIs. If more KPIs are standardised, dashboards showing comparative performance between operators at a national level will become increasingly easier to generate.

If appropriate, data may be disaggregated by gender and age in internal and external communication, dashboards, etc.. Regardless of how data in general and KPIs in particular are presented for analysis, the fundamental requirement remains the collection of relevant and accurate data in the first place. This requires not only well designed forms but also personnel, both operations and IM, to see the full and accurate completion of such forms as fundamental to their roles.

15. Responsibilities

15.1 National Mine Action Authority

The NMAA, or an organisation acting on its behalf, should:

1. identify mandatory KPIs;

2. establish and promulgate CCRs;

3. ensure that monitoring systems include IM in general and KPI aspects specifically;

4. identify and act on opportunities to establish benchmarking or other performance comparators; and

5. encourage the dissemination of appropriate KPI information.

15.2 Mine action organisations

Mine action organisations should:

1. collect and report data in accordance with the requirements of the NMAA; and

2. identify and maintain KPIs relevant to their own operations.

In the absence of a NMAA or similar authority, the mine action organisation should assume additional responsibilities. These include:

1. agree with the donor (or client, or customer) the mandatory KPIs; and
2. establish and promulgate CCRs.

15.3 Donors, clients and other stakeholders

Those organisations contracting or funding mine action operations should:

1. identify KPIs they require in MoUs, contracts and other relevant documentation; and

2. support and encourage NMAAs in the maintenance, use and dissemination of KPIs.

Annex A: (Normative) References

The following normative documents contain provisions, which, through reference in this text, constitute provisions of this part of the standard. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. However, parties to agreements based on this part of the standard are encouraged to investigate the possibility of applying the most recent editions of the normative documents indicated below. For undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC maintain registers of currently valid ISO or EN:

• IMAS 04.10 Glossary of mine action terms, definitions and abbreviations
• IMAS 07.11 Land Release
• IMAS 05.10 Information Management for Mine Action
• IMAS 07.12 Quality Management in Mine Action
• IMAS 07.13 Environmental Management in Mine Action
• IMAS 07.14 Risk Management in Mine Action
• IMAS 07.30 Accreditation of mine action organisations
• IMAS 07.40 Monitoring of mine action organisations
• IMAS 14.10 Guide for the evaluation of mine action interventions

Footnote

¹ Note that neither IMAS 04.10 or IMAS 14.10 defines performance. IMAS 14.10 distinguishes between the indicators used to measure the extent to which outputs have been delivered, termed ‘performance’ and those relating to subsequent outcomes and impacts.

² IMAS 05.10. Information Management for Mine Action.