Analysing safety data for an entire industry group – a case study from Queensland coal seam gas

Collecting data for years

Injury statistics were collected for the CSG industry for more than a decade before modern analysis tools would be used on them. The work of collecting data was done for the industry following a partnership between industry groups Safer Together and the Australian Energy Producers (formerly APPEA). Collecting and preparing quarterly statistics was handled by one operator within the industry who, for 14 years, took responsibility for collecting, preparing and presenting industry safety performance statistics quarterly at the Senior Leaders Group meetings of Safer Together.

Five measures were prepared on a quarterly basis for the industry and presented on a 12-month rolling basis. The measures were separated into two groups, there were four KPIs that reflected industry safety KPIs, and one TRIF: (1) an industry-wide total recordable injury frequency (TRIF), (2) a fatality rate, (3) a high potential incident rate, (4) a severe motor vehicle incident rate and (5) a process safety event rate. The four KPIs were meant to help the industry take a step beyond injury rates, which are known to be plagued with under reporting (Probst and Estrada 2010; Probst and Graso 2013); there is bias in what gets reported in injuries (Geddert et al. 2021); and injury rate focus can lead to some poor behaviours (Dekker 2017; Thomas 2020). The four key performance measures were designed to direct safety conversations away from a pure focus on injury rates, which were not yielding productive conversations as a stand-alone measure. The data collection was done via spreadsheets that were completed by all operators in the industry and submitted to one central operator who had volunteered as the data curator.

Injury rates would still feature heavily in industry conversations despite efforts to move away from a pure injury focus. One common thread in the quarterly data conversations was that the explanation behind higher TRIF rates in certain divisions was explainable due to the lower severity of injuries. It was also commonly stated that although the industry was seeing a higher than historical injury rate, overall, the average event was believed to be less severe than history. This explanation was presented anecdotally as months and quarters passed without the annualised TRIF rate demonstrating significant improvement. Nonetheless, assurances continued from the industry that we were seeing a higher volume of lower severity outcomes. The injury rate would only demonstrate the increase in volume.

A pilot project

There are academically suggested alternative measures to injury rates (Dekker and Tooma 2022; Erkal and Hallowell 2023), but the collection and tabulation of those measures were a considerable departure from what the CSG industry had collected for years and could reasonably share with each other. In 2023, Safer Together was presented with a model for reporting injury severity that could help address the limitations of the stand-alone injury rate. The suggested measure could also be trialled with minimal adaptation to their current data collection. This model, called the ‘Severity Adjusted Injury Frequency’ (SAIF), had the potential to change some of the conversation around a higher volume of injuries with a lower severity. It would require the collection of a small amount of additional information and the preparation of some additional presentation materials. The person responsible for the design of the SAIF rate was invited to participate in the preparation of the fourth quarter (Q4) industry statistics in late 2023. The pilot was deemed successful in its first iteration and carried forward into 2024.

It was not the intent of the SAIF rate pilot program to transition the preparation of industry statistics to a third party; however, through the preparation of the pilot data, it was identified that the individual responsible for the SAIF rate design could also manage the processes for injury data collection and analysis. The operating company that had been curating the data for more than a decade was providing resources from their own organisation who specialised in safety but not necessarily data analysis. The third-party organisation responsible for the SAIF rate specialised in data analysis, with key resourcing focused on the measurement of safety outcomes. The decision was made to change the relationship following the preparation of Q4 industry statistics. In 2024, starting with the collections and preparation of 2023 summary statistics, the task of preparing quarterly industry summaries would be handled by an organisation that focused on measurement and analysis.

The third-party organisation, after completing necessary confidentiality agreements, was provided with years of historical data that had been collected by the industry. There was some significant cleanup work required to make it consistent and correct for known errors. Once that cleanup work was completed, the data were prepared and analysed using a combination of considerable industry knowledge, experience in the measurement and analysis of safety information, and the latest in modern business intelligence tools. The examples that follow are a demonstration of what is possible with the combination of new approaches to measurement and new technical tools for analysis. They are also the result of trust, common sense, applied intelligence and curiosity. This combination of technical knowledge and personal intrigue allowed the data discussion from the past year to move from routine reviews to dynamic conversations about genuine insights.

Exposure trending and history

Prior to 2024, exposure hours were not a typical part of the analysis done in quarterly safety data presentations. As the denominator for TRIF, and other key measures, the data on hours seldom got direct attention. In addition to the summary of KPIs, the updated analysis package included a cumulative analysis of hours focused on significant increases and decreases in overall exposure hours across the four industry divisions: (1) construction, (2) drilling, (3) gasfield operations and (4) liquefied natural gas (LNG) facility operation.

Fig. 1 was presented with commentary around the percentage increase or decrease from the previous quarter. A comparison to longer-term history was completed outside of the presented materials but allowed for commentary and discussion of whether the industry was operating at higher or lower levels than normal for the period as compared to previous years.

Fig. 1.

Exposure by division.

The presentation of full-industry changes in exposure over a 12-month period led to conversations about the safety impacts caused by growth or shrinkage of the industry. Providing industry leaders with more detailed industry exposure data was met with conversations at an industry leadership level about the need to either maintain an experienced workforce or to ensure that ramp-up activities consider a largely new workforce, particularly if there is growth across the whole industry.

The key measures with history and detail

The four key measures of (1) a fatality and permanent impairment rate, (2) a high potential incident rate, (3) a severe motor vehicle incident rate and (4) a process safety event rate were continued in their application. These measures drove conversations about how the CSG industry is controlling fatal risks. The time and resources applied to the analysis led to additional detail and commentary beyond the basic numbers. The four key industry metrics were rebuilt from an excel document and are presented in Fig. 2.

Fig. 2.

Four key industry metrics. PSEF, process safety event frequency; HPI, high potential incident; FPI, fatality and permanent impairment; SMV, severe motor vehicle.

A single slide was produced to summarise all four measures; however, the introduction of analytical resources added additional commentary on the details behind each measure, sometimes calling out connections between reported low to medium severity injuries compared to high potential injury events seen in the quarter. Connections between events are not always obvious or immediately visible in the metrics; these were highlighted during the commentary on the provided metrics.

Additional to the performance for the quarter, there were tools that could access considerable history from the industry. For example, if several risk events from a quarter highlighted a topic for conversation, additional colours could be added based on a decade worth of history for similar events. The decade worth of high potential incidents could be analysed to provide detail on the correlation between events seen in the current quarter and events seen in the industry’s past. This would include historic causes of high potential events and the prevalence of certain events in the past.

The SAIF pilot program

The SAIF pilot program was included as part of the regular analysis of injury rates (TRIF). Injury rates continued to be reported as they had been, but the addition of a SAIF analysis radically changed the discussion about what those rates were telling the industry. The addition of a functional severity measure provided a necessary missing element to the discussion of injury rates.

The design of the SAIF is not the topic of this paper, but it will be discussed in upcoming publications. For a practical understanding, the SAIF rate takes all injuries included in the TRIF into consideration and rates them based on whether they would be considered as better or worse than 2-weeks lost time. By using this mathematical model, two measures can be demonstrated in parallel:

These complementary measures are taken together to present a more complete understanding of actual injury event outcomes.

Fig. 3 represents the first demonstration of the SAIF rate for the industry as presented in the third quarter (Q3) 2023 analysis. Both measures are technically accurate representations of the data on injuries but they tell different stories. The injury frequency alone tells a story that the industry was injuring more people than ever, which is the only story this stand-alone measure can tell. The addition of a SAIF, when reviewed together with the injury rate, tells the story that, although the frequency of injuries was increasing, the events were of a low enough severity that we were not seeing an increase in the relative number of people with a 2-week LTI. This suggests that although the rate is going up, the average severity of injuries is going down.

Fig. 3.

TRIF and SAIF comparison.

It is worth noting that this was not a revelation for the industry. For several years, the conversation around TRIF rates has been that the average event was less severe than in the past. What was new, with the addition of the SAIF rate, was that the concept could be proven empirically rather than just stated as an anecdote to a rising TRIF. One participant’s comment from an early presentation of the SAIF rate is worth noting. Upon seeing the first release of this methodology, it was hailed as innovative, but the comment followed: ‘The SAIF rate doesn’t suggest we’ve arrived; it suggests that there is still work to do’. Unlike previous conversations, the SAIF rate did not use severity to explain away the TRIF rate. Rather, the SAIF considered severity, then presented a more realistic reflection of actual outcomes considering both how many people were injured and how badly.

The SAIF rate assessment was continued into subsequent quarters, andthe standard presentation format for changes to TRIF was adapted into a version that analysed TRIF, SAIF and the average severity of injury events. The power of complementary measures was demonstrated through stories told by the data. The data stories were presented on a slide in the Q3 presentation, and the different stories from that slide have been included in Table 1.

The difference between the stories told by different approaches to data highlights the radically improved depth of standard reporting. The application of the SAIF rate created more dynamic conversations about performance that considered both how many, and how severe injuries were. However, the change in conversation on safety data was aided on a second front by the introduction of analyses unique to each quarter and tailored to the conversations that were driven by the quarter’s results.

Injury mechanism analysis

The introduction of a functional severity measure allowed for further analysis of injuries included in the data set based on mechanisms of injury. Seen below is an analysis of 12 years of injury data collected for the Queensland CSG industry. This analysis includes three measures for different mechanisms of injury: (1) the number of recordable incidents, (2) the overall contribution of those incidents to the SAIF rate and (3) the average severity of individual incidents.

Fig. 4 was included in the Q3 2024 analysis and demonstrates the calculation of an average injury severity for each mechanism of injury. This analysis highlighted concepts previously understood but not quantified. Injury rates alone could be used to tell that being hit by moving objects and hitting objects with a part of the body were among the highest contributors to injury rates. The addition of an average severity measure highlighted that those incidents, although common, were among the lower severity of outcomes. Meanwhile, falls from height, vehicle accidents and contact with electricity were all relatively uncommon events for the industry but had considerably more severe outcomes when they did occur.

Fig. 4.

Injury mechanisms with average severity.

Suggesting that certain mechanisms of injury have more severe outcomes was not an epiphany. The mechanisms of fall from height, motor vehicle accidents, and contact with electricity have long been recognised as fatal risks. This long-term representation of actual event outcomes provided a mathematically backed demonstration of the need to control higher-risk mechanisms rather than just focus on high-frequency events. None of these were new ideas, rather the change was the ability to demonstrate differences in failure mechanism outcomes in a clear, easy-to-consume manor based on actual industry events. Using years of historical data across an industry, this analysis demonstrates empirically that certain low-frequency events have significantly worse outcomes and are worth the effort to control despite their low frequency of occurrence.

Understanding the long-term performance of the injury rates in the industry

It was a long-held industry belief that while the overall frequency of events was increasing, the average severity of those events was decreasing. This no longer needs to be believed anecdotally; it can be tested mathematically. An 8-year trend was produced from the TRIF, SAIF and average injury severity data. This extended back to the period where the data could first be used to produce a reliable TRIF rate.

Fig. 5 applied linear regressions to the injury rate, severity adjusted injury rate and average event severity for the last 8 years of usable TRIF history. This analysis was able to prove the long-held belief that while the injury rate for the industry was increasing, the average severity of events was decreasing. However, it produced another insight: the average severity of events was decreasing but not fast enough to offset the increase in rate of events. Over the course of 8 years, the industry showed a slight increase in the relative number of people with a 2-week LTI. This demonstrated that the industry was correct in believing that the higher frequency of events was against a lower severity of event but that there was/is still work to do.

Fig. 5.

 Eight years of injury rate, severity adjusted injury rate and average incident data.

There is one significant missing piece of information from this analysis and that is the effect of under reporting. Over time, particularly in the last 2–3 years, the industry has reduced its focus on TRIF rates. As such, the reactions to recordable events in the later period of this chart would not be the same as the earlier part of the chart when injury rates were a considerable focus for the industry. There is not a practical way to test whether under reporting was more prevalent in the earlier years when focus on TRIF was heavier. This analysis didn’t suggest that everything was ideal. Rather, it highlighted that assumptions about severity were correct and moved the conversation forward from that point.

Shared understanding of exposure trends

Early analysis of 2023 exposure hours identified that exposure was higher in Q3 than other quarters. This was expected for LNG plants, which would schedule routine turnaround activities in Q3 that involved significant additional resources. Other business areas did not have such turnarounds in Q3 but were nonetheless seeing an increased volume of hours in the third quarter. Upon review, it turned out that the increase in exposure in Q3 was consistent over the past 4 years, and this was true for divisions outside LNG facility operation that did not execute Q3 plant turnaround.

Fig. 6 demonstrates that Q3 was, on average, 14.4% higher in terms of exposure hours compared to other quarters. This was calculated excluding the LNG facilities.

Fig. 6.

Average exposure hours by quarter (excluding LNG plants).

In discussion with the Senior Leaders Group, it was learned that the shutdown and turnaround activities performed by the LNG plants were resulting in additional flow-on work for the other divisions. Some leaders knew or anticipated this; for others, it was a revelation that the LNG plant shutdowns had a flow-on effect that increased exposure outside of the plants. Even for those aware of the flow-on effects of the turnaround activity, it was an insight that the affect could be quantified.

Detailed history at injury forums

Hand injuries were a common issue for the industry in 2023, leading to a second industry-wide hand safety forum hosted by Safer Together in 2024. This forum began with a presentation of injury statistics related to hand injuries from the previous 12 years.

Figs 7, 8 and 9 are excerpts from a larger data presentation prepared for the hand safety forum. The data from past years was able to demonstrate that hand injuries were the most reported injury every year since the start of injury tracking. However, trending was able to demonstrate that the distribution of injuries had changed. The analysis of injury frequency and SAIF was able to demonstrate that the previous year (2023) had not just seen an increased volume of low severity events, as some had predicted, but the previous year (2023) had seen a material change in hand injuries due to the cumulative effect of both more events and more severe events.

Fig. 7.

History of hand injuries.

Fig. 8.

Changing distribution of hand injury severity. LTC, Lost Time Case; RWC, Restricted Work Case; MTC, Medication Treatment Case.

Fig. 9.

Combined TRIF and SAIF for hand injuries.

Having data presented in a compelling manor to frame the problem facing the industry was an effective tool to kick of a day-long seminar on how the problem might be addressed.

Changing the conversation about injury frequencies

In the Q3 summary for 2024, a tailored analysis was included that highlighted different trends in injury severity and frequency for different mechanisms of injury. This presented changes in patterns of injury for events related to falls from height and falls on the same level (slips and trips). Fig. 10 includes two charts that were presented in the analysis of Q3 2024. These two charts demonstrated that, over the past few years, there have been significant changes in events recorded and the changes are quite different for different mechanisms of injury.

Fig. 10.

Examples of changes in reporting behaviours.

When falls from height were analysed, the trend demonstrated that events were somewhat sporadic in terms of frequency; however, the average severity of events followed a relatively consistent downward trend. When discussed at a senior level, some members of the discussion recalled a period where they had several events followed by introductions of engineering changes that reduced the likelihood of an event but also the severity should an event occur.

The conversation about falls from the same level took a much different turn. The identified data trend was that, over the past few years, the frequency of events increased considerably, but the average event severity remained relatively unchanged. A few ideas were shared as to why this may be, but one caught the attention of the group. One of the senior leaders pointed out that the increase in frequency of slips, trips and falls correlated with the approximate time they started focusing more heavily on fatal risk factors and stopped reacting heavily to slips and trips. The question was asked whether the industry may have always had that high frequency of slips and trips or, when they stopped reacting as heavily to them, did it become safer for workers to report them without significant backlash? The industry would not have the data or methodology to prove such a hypothesis, but it certainly would be a reasonable explanation for the resultant data that the industry is producing.

Dynamic conversation responses

The preparation of statistics changed from a standard set of reports to a more dynamic conversation about current statistical events. One example of the dynamic shift in the conversation came during the presentation of high potential rates. During the discussion of the second quarter (Q2) 2024 results, Fig. 11 was presented, which demonstrates high potential incident reporting for the quarter. It was noted that the rate of high potential events had decreased. The question was asked by one of the leaders what this might look like if the analysis were to extend further back. Would this drop represent a new low or a return to previous levels? The analyst was in the room for this discussion, and while the discussion ensued, the analyst was able to pull up that information and present the answer to the question during the discussion. It turned out that this was a new low in the rate of high potential events. In fact, all of 2024 demonstrated a considerable drop from prior years in terms of the rate of high potential events per exposure hours. This changed the conversation again from one of speculation about what the past might have looked like to what the lower reporting rate in 2024 actually means.

Fig. 11.

Reported drop in the rate of high potential events reporting.

One of the key capabilities of modern business intelligence tools is their ability to generate insights as quickly as you can create the questions. The ability to evaluate the information during the conversation moved the conversation forward.

Using data to support efforts on low-frequency events

Another example of dynamic reporting came in the analysis of Q3 2024. Several motor vehicle-related events took place in the quarter, including some very severe events. In response to this, an analysis was prepared to highlight the last decade of motor vehicle events.

Figs 12 and 13 were prepared in Q3 where significant motor vehicle events had occurred. In addition to the normal quarterly results, these charts highlighted to the senior leaders that motor vehicle events were the highest reported high potential incident for the past 10 years. Additionally, with the inclusion of Q3 2024 data, such events represented the highest average severity of outcome for any mechanism of injury.

Fig. 12.

Ten-year high potential events involving motor vehicles.

Fig. 13.

Ten-year actual injuries due to motor vehicles.

High potential events are defined at Safer Together as any incident or near miss that could have realistically resulted in one or more fatalities; this definition is in line with the International Association of Oil & Gas Producers definition, which is the international standard that the industry follows. In a quarter that involved significant vehicle-related events, this analysis drew attention to the history of such events in the industry. What this analysis demonstrated was that the actual number of injuries from motor vehicles in the last 10 years was quite low, but the severity of those individual events was higher than any other mechanism. This was meant to stress the importance of the considerable efforts that contractors and operators in the industry put into motor vehicle safety.

This analysis did meet some pushback, believing that it was meant to challenge the reporting accuracy of motor vehicle events, and it was clarified that the analysis demonstrated that efforts to control motor vehicle events were well placed. However, this pushback led to an important follow-up discussion: that reducing injuries is why safety is done, not how safety is done. Reporting on how many people were hurt in vehicles does not do anything to actually keep drivers safe. Lagging indicators, such as injury rates and the SAIF rate, represent the failure of programs to achieve their desired results; they also stress the importance of controlling different risk factors. However, lagging measurements do extraordinarily little to keep people from being injured; they only tell us if people were. Leading indicators are about how people are kept safe. In this case, measures such as in-vehicle monitoring activities, leader conversations, driver training programs and fatigue programs are the actual activities that contribute to how people are kept safe. The only thing that the lagging analysis could do was provide emphasis for the efforts behind the leading activities.