Real-Time Perception Enhancement in Obscured Environments for Underground Mine Search and Rescue Teams

1.

Introduction

After natural or man-made disasters, saving lives is a time-critical mission [1]. While racing with the clock, search and rescue teams tackle environmental difficulties [2]. Subterranean spaces such as tunnels, mines, or caves are among the most challenging locations. These locations are not only GPS-denied, but also have low to no visibility with uneven surfaces [3]. Moreover, due to its challenging nature, many devastating disasters occur in underground mines; hence, advanced indoor navigation solutions are needed [4]. In emergencies, the environment encountered by the mine rescue personnel is often visually occluded with dust and smoke to the point where visual perception is completely compromised, and the vision is restricted to 1–3 feet [5].3 °C

Traditional methods to guide emergencies in an underground mine include hand lines, pin-wheels, and strobe lights [6]. Using a green laser pointer, a more active approach has been shown to provide some situational awareness, as its beam can penetrate the theatrical smoke enough to indicate the back and rib locations [5]. Another attractive option is thermal imaging, which can be seen through dust and smoke, seen in firefighters and military operations, where thermal imaging is predominantly used in smoke-occluded situations [7].

Since the underground map changes, these traditional methods may be inaccurate after a tragic disaster, such as rock falls, roof collapse, or explosions. Therefore, real-time 3D world construction is necessary for safer and faster rescue operations [8]. The technology proposed for real-time construction is an extension of the robotics area of simultaneous localization and mapping (SLAM) [9]. This approach uses sensor data to determine the surrounding area appearance as well as where the sensors are in that area. SLAM solution has recently transitioned from laboratory to commercial sector; however, it has been generally tested under ideal conditions, such as light and structured spaces [10]. Most SLAM methods rely directly on the data gathered either by making the data the ground truth and fitting directly to it (which is computationally expensive) or extracting landmarks and incorporating them into the model to reason on, which is less effective in underground environments [11]. Even with the recent advances in sensor, battery, and processing power technologies, building an effective system capable of seeing through pitch-black conditions with real-time world constructions still has limitations [12, 13].

The Defense Advanced Research Projects Agency (DARPA) created a Subterranean Challenge (SubT) to overcome these limitations for mapping, navigation, and object detection in underground environments in 2021 [14]. DARPA aims to push the limits of technologies that could be ready to use sooner [15]. Although the main goal of the SubT was the autonomy of search and rescue robotics [16], this challenge helped us to understand sensor fusion and real-time mapping and navigation.

Reviewing the DARPA SubT challenge results, we noticed that underground search and rescue operations currently rely on humans and likely to be so in the near future. However, lessons learned and findings can be implemented in underground search and rescue operations, such as restoring visual information to emergency responders.

Exploring sensor fusion and mapping suggestions from various institutions and industry leaders can enable search and rescue teams in underground spaces to search the environment more safely and efficiently. They will be able to scan an area for hazards and provide personnel with better situational awareness [17]. In this respect, augmented reality (AR) interfaces have great potential to leverage the search and rescue efforts in an emergency evacuation. Yet, the AR interface for real-time perception enhancement in underground mine search and rescue operations has not been explored sufficiently [18, 19].

Given the shortfalls in mining operations, such as navigation in pitch-black and mapping in a changed environment, this work focuses on developing a hybrid human-machine system for solving situation awareness problems in pitch-black and smoke-filled underground mines with possibly faster navigation. The proposed methodology is to combine thermal imaging, LiDAR, and AR. This aims to penetrate smoke by leveraging thermal imaging while having real-time world construction with LiDAR and visualizing it on an AR device. By using this hybrid system, we aim to provide an AR image of the surrounding environment when pitch-black or visually occluded by smoke, dust, or other small particulates. This system is structured around an AR device, namely, Microsoft HoloLens, which will display an image to the user that depicts the surrounding environment meaningfully, but could be extended to any compatible AR platform. This map enables the user to explore the environment with continuous improvements in fidelity as they maneuver. Finally, the effectiveness of the proposed system is evaluated in an experimental underground mine.

Our study fills the necessity of having vision in pitch-black conditions during emergency evacuations in underground mines with a changing environment for search and rescue teams.

2.

System configuration

This system is composed of multiple components and, as such, is run across various hardware and software. The hardware includes a NUC computer, Omnicharger, Krisdonia battery, hardhat, HoloLens, LiDAR, and thermal camera. The software and algorithms include Linux, Windows, Unity, and Robot Operating System (ROS). Table 1 lists hardware and software information.

The high-level hardware and software arrangement is as follows. A portable computer (NUK) runs a ROS-based program to record, modify, and transmit live point data with the LiDAR, inertial motion unit (IMU), and thermal camera, which transmits data via web sockets. The LiDAR, thermal camera, and NUK require a power source; thus, a portable battery (Krisdonia or Omnicharger) is used. Sensor data collection in the form of LiDAR and thermal images collects and transmits 3D point data to scripts that manipulate and transform the data into another format by ROS bridge by NUK. Once the data has been converted, it is sent across a WebSocket on a local network to be visualized in Unity Editor and deployed to HoloLens. The images are color-coded with thermal data. Unity is the development platform that allows the application installation on the HoloLens. Figure 1 shows a high-level overview of the data flow.

3.

Approach

LiDAR scan information is too dense to display directly on the HoloLens in real time. So, the geometry of the surrounding environment is stored as a voxelated space using the Octree-based framework [20], which subdivides the current measurement volume into ever-smaller cubes with a final resolution of 5 × 5 × 5 cm. The final resolution is only necessary at the surfaces as large sections of rock or air can be represented as much larger voxels, which creates a memory-efficient storage method. The framework can statistically analyze the points that exist in each voxel. The mean and covariance describe not only where the surface is within the voxel but also its orientation and flatness. This additional information allows new LiDAR data to be compressed into shape primitives for display on HoloLens, which are then colored using either depth or thermal information.

To improve the integration of HoloLens; LiDAR, IMU and thermal camera are rigidly mounted on a hardhat with nested HoloLens (Figure 3). This rigid attachment allows to fine-tune the static transform between the LiDAR and HoloLens coordinate systems. This integration allows improved visual alignment and refined feature detection. Thus, LiDAR data is put into the Octree; then, mean and covariance are used to distort shape pixels to send a minimal set of triangles to the HoloLens to visualize. The Octree framework allows for efficient triangulation/tessellation, while the integration of hardware allows for improved visualization and feature detection.

4.

Results and discussion

The resulting system integrates LiDAR, IMU, Thermal Camera, and Microsoft HoloLens onto a wearable platform such as a hardhat and a belt or backpack. The processing and power storage are integrated into a waist-belt-mounted package with a cap-lamp-style cable connecting the two. This enables the user to look around with minimal impact on their motion.

Initial results yield possible visualization varieties. These visualization options are given in Figure 4 as (a) single color mapping, (b) thermal mapping, (c) depth mapping.

Among the visualization options, depth mapping with a thermal camera yields the best and robust results. Color schemes to provide depth perception information are added as a user option, where red schemes indicate closer objects and green schemes indicate farther objects (Figure 4).

The display capabilities of the system have been expanded to include several different shapes with tradeoffs in terms of point density and fidelity. Each of these display variants can be generated directly from the Octree data and evaluated as one of the following:

1. (1)

   Triangulated mesh, where the point-cloud is taken and the gd3 algorithm from the point-cloud library [21] is used to generate a tessellation.

2. (2)

   Square surface patch, where the point cloud is packed into Octree that tracks a Gaussian representation of the points measured in each voxel. A square patch is then generated that represents the average surface for that voxel.

3. (3)

   Triangle surface patch, where the point cloud is packed into Octree that tracks a Gaussian representation of the points measured in each voxel. A triangular patch is then generated that represents the average surface for that voxel.

4. (4)

   Cube occupancy grid, where the point cloud is packed into Octree, and the voxels that contain points are extracted and displayed to the user.

5. (5)

   Cuboid surface representation, where the point cloud is packed into Octree that tracks a 3D Gaussian representation, which is converted into a rotated and scaled cuboidal shape.

6. (6)

   Ellipsoid surface representation, where the point cloud is packed into Octree that tracks a 3D Gaussian representation converted into an ellipsoid according to the standard deviations in each direction for display.

The number of vertices and triangles with 2D performance of each display type is given in Table 2.

After choosing the visualization method and building the application into HoloLens, the proposed system is tested in an experimental underground mine, Edgar Mine, located in Idaho Springs, Colorado. The final run snapshots are given in Figure 5 with a complementary video [Video 1] that shows the user perspective.

While testing the end product, a few challenges were encountered. The first challenge was that, although the sensors and hardhat had fixed dimensions, the users’ head size and eye distance to the sensors were different. This made a small offset for the visual enhancement interface. Since the search and rescue operations are time-critical, we added a wireless gaming controller for real-time alignment for the visual enhancement stream. The alignment helped users to fit the enhancement in four degrees of freedom.

The second challenge was that the HoloLens was not initially intended for dark environments, as Microsoft software ceased to function as soon as the ambient lighting faded. As a workaround, the system allows the programmers to completely turn off Microsoft provided tracking and rendering, which allows the use of device at a lower level and project the images even if the HoloLens loses internal tracking.

Finally, the application must be run in Unity to collect test and debug information. To do this, we used an open-source library for Unity, ROS-sharp [22]. Using a network bridge, a Unity program can then subscribe to ROS topics and collect the transmitted data. This results in a 3-way communication (LiDAR->Linux ROS network->Windows). This was not a significant issue as, in the end, we would still have a 3-way communication of sensors communicating to ROS and that to the HoloLens. Furthermore, there was no visible change in the speed of visualizing a scene with moving objects (e.g., waving an arm or walking around).

Besides these challenges, we also observed few limitations. First, HoloLens is a battery-operated device. Due to the debugging and test runs, charging is unavailable while the device uses Unity and ROS bridge. Although the design can be updated to include charging while working, we can run it for approximately two hours.

The second limitation is due to regulatory reasons. Some mines, such as underground coal mines, do not allow battery-operated devices and exposed cables for safety reasons. This regulation is changing worldwide, but if this proof-of-concept device design is converted into a commercial build, some tests and permissions are required to be usable in all underground spaces.

Third, the LiDAR that is used in this build has a nine-meter range. Due to the tunneling effect of the visualization, we did not utilize the full nine-meter range. However, this might be a limitation for larger underground tunnels and openings by missing important crosssections or points of interest.

Lastly, the design is tested for proof of concept. It should be noted that the weight distribution on the hardhat is not equal in all directions. Even though it is designed in a more balanced way, the additional four-pound weight on the user’s head will not be comfortable in the long run. However, this system needs to be designed and recommended for daily usage. This is for search and rescue cases in emergencies.

To sum up, both challenges and limitations might make the utilization of such a system harder, but it is usable as is. Initial subject tests can be found in this publication [23].

5.

Conclusion and future work

In this work, we combined a thermal imaging camera with a LiDAR sensor and SLAM mapping and visualized a real-time world construction as an AR interface. Our proposed methodology yielded a proof of concept wearable device for underground mine search and rescue personnel. The feasibility and progressive nature of the device were tested in an experimental underground mine in Idaho Springs, Colorado. The results showed that the combination of technologies used in our study enable faster, safer, and more effective disaster response for mine rescue operations. Not only does it allow the responders to search the environment more rapidly, it also enables them to detect unexpected hazards before they become imminent threats. Moreover, the utility of the developed system is far-reaching, for example, for first responders searching smoke-filed burning structures. In the future, it may also enable autonomous systems to navigate these occluded environments effectively and enable disaster response to focus on the rescue in search-and-rescue.

As discussed in Section 4, the underground mine test yields some shortfalls. As part of future work, we will first focus on improving the visualization methods and then testing each method with human subject studies. This will help us determine how to create more effective visualizations so that search and rescue personnel can act faster, safer, and be more effective.

Conflict of interest

The authors declare no conflict of interest.

Acknowledgments

This study was sponsored by the Alpha Foundation for the Improvement of Mine Safety and Health, Inc. (ALPHA FOUNDATION) contract number AFC820-54. The views, opinions and recommendations expressed herein are solely those of the authors and do not imply any endorsement by the ALPHA FOUNDATION, its Directors and staff.

Source data

Source data (raw scientific data accompanying the research) for this article is available on Figshare: https://doi.org/10.5772/acrt.deposit.25688805