We are exploring the contributions of next-generation user interfaces in search and rescue operations in burning buildings.
Search and Rescue
During search and rescue operations, firefighters encounter a wide variety of hazards that potentially put their lives in danger. In a burning building, for instance, risks can quickly escalate. On the scene, a firefighter may encounter blocked passages, harmful gasses, high temperatures, toxic smoke, and various combinations of risk factors. Each incident is unique. Firefighters often navigate buildings blind, with a variety of floor plans, and must monitor the presence of hazards for the operation to succeed.
Moreover, firefighter heat-tolerant gloves limit their ability to perform fine-grained interactions. Along with that, the nature of their operation requires a high level of attention to observe hazards in a dynamic changing environment, constant use of their hands for navigation, debris removal and victim rescue, and non-traditional locomotion style for indoor navigation in smoke-filled rooms. All of these factors pose design challenges when interacting with the UI.
VR Simulation
We created a virtual simulation of a burning building with three operations in mind:
- Explore the building
- Rescue the victim
- Avoid and report hazards during the exploration.
NEXT-GENERATION USER INTERFACE
By adopting next-generation AR UIs, environmental perception is enhanced and uncertainties are reduced. For the search and rescue scenario, we designed AR UIs for three high-level functions: indoor guidance, teammate identification, and situational awareness.
Indoor GuidanceOur indoor guidance system includes two key functions: dynamically path generation and mini-map. The path generation function enables the request for directions for a remote team leader. When the path is received, it appears on the mini-map and HUD. The indoor guidance mini-map automatically updates a functional floor plan as firefighters explore the structure. The mini-map reveals the floor plan while firefighters explore the structure. When a new room or hallway is entered, the room is automatically scanned and added to the map. It also depicts real-time data about firefighter’s position, the presence of hazards, the victims’ location, and the path to a particular area. Real-time updates of the environment and the team help avoid repeated searches, ensuring that other teammates have instant access to the latest information. The mini-map is activated on-demand by eye-gaze. Gazing downward, and focusing on a virtual tab for a specified time, the mini-map rises into view. |
Team IdentificationFirefighting search and rescue operations require a high level of communication and cooperation among teammates. Next-generation UIs must provide firefighters with access to real-time information. This may include the names, distance, and status of nearby firefighters along with other crucial data. Team members should be able to share information and request assistance as needed. The system tags the firefighters and shows their location in the environment. We use icons floating over the teammates where we display their information. The tag contains the firefighter’s name and distance. Each firefighter’s tag has a different color to easily distinguish them. The tag is visible through walls and barriers, and its size is consistent with the firefighter’s distance. The mini-map also displays the same tag representing each firefighter and their accurate location in the building. |
Situation Awareness Interface
Most risks encountered in firefighting search and rescue operations are due to a lack of situational awareness. For example, the unexpected spread of fire could block the planned evacuation route putting the firefighters at risk. To improve awareness of environmental factors, our next-generation UIs implements a situation awareness interface, which consists of a hazard marker system, an alert system, and an environmental information display.
- Hazard Marker System: The system allows for the deployment of markers around the hazardous area to alert teammates. There are two marker deployment modes: precise deployment and quick deployment. In the precise deployment the hazardous region is explicitly specified by choosing the type of marker from the menu and placing it near the source of the hazard. In the quick deployment method, suspicious regions are quickly labeled while the firefighters exploring the space. The virtual markers placed on the environment are displayed both in the AR display and on the mini-map. The markers also have distance information similar to the team identification feature.
- Alert System: The alert system sends a color-code-based warning to the peripheral view if the firefighter is confronting some danger (Fig. 2(d)). For example, when the environment temperature is unusually high, the system can send an alert to notify the change.
- Environmental Information: Two fundamental information that firefighters need when exploring the building are oxygen level and room temperature. These information are displayed in the HUD display in the peripheral vision for quick access.
VR SEARCH AND RESCUE SIMULATION WITH NEXT GENERATION UIS
The video shows the search and rescue operation assisted by next generation UIs.
Typically, search and rescue are cooperative operations that require the coordination of several firefighters. While the scenario exemplified here count with an exchange of information with other firefighters in the team, the interactions with other team members and team leader are simulated and only allow for a limited set of options that are predetermined based on the current simulation phase. The advantage of doing that is that we can guide the user through the scenario and impose the use of the user interfaces.