Aaron Barth tries out the new head-mounted display in the research simulator.
Aaron Barth tries out the new head-mounted display in the research simulator. (Photo: Franz Viertler / TUM)

Augmented reality enables helicopter flight in degraded visual environmentsVision through the clouds

Fog, blizzards, gusts of wind – poor weather can often make the operation of rescue helicopters a highly risky business, and sometimes even impossible. A new helmet-mounted display, developed by researchers at the Technical University of Munich (TUM), may in the future be able to help pilots detect hazards at an early stage, even when their visibility is severely impaired: the information required to do this is created in an on-board computer and imported into digital eye glasses. A new study has shown that this augmented reality improves the performance of pilots.

Thick clouds hang over the Tegernsee. The range of sight is just a few hundred meters. Under normal circumstances, a helicopter would not be allowed to take off in such weather – the danger that the pilot would not be able to react in time to a construction crane, a power line or a mountain would be too great. But this does not stop Franz Viertler from entering the cockpit. Thanks to his head-mounted display - digital eye glasses attached to his head that display additional information - Viertler can detect hazards and navigate around them, even in thick fog.

The new technology is still in the research stage: the Alpine foothills are a projection, Viertler is not a pilot, but a TUM engineer and he is not really operating an actual helicopter, rather a flight simulator. Yet that is all you need to demonstrate the possibilities of augmented reality - a type of reality enhanced by additional information in digital form. The rotor purrs, the cockpit vibrates, clouds approach and obscure the view. Nevertheless, Viertler knows exactly where he is: the head-mounted display paves the way with green lines representing the outlines of mountains and houses. A red outline in between depicts wind turbines, construction cranes and high buildings.

Many realities, one goal: safety

"Our goal is to increase safety for pilots using augmented reality," explains the researcher. Together with his colleagues at the Institute of Helicopter Technology, he has developed software that combines terrain information with sensor readings that can be taken during a flight. Light Detection and Ranging, LIDAR for short, has proved especially useful here, stresses Viertler: the measuring instruments that can be attached to the helicopter’s skids emit radiation in the micrometer range and detect the waves reflected by hazards or obstacles.

All of the data is processed on-board and projected directly to the see-through head-mounted display. This allows the pilot to not only see what he can detect with his own eyes, but also the digitally-generated outlines of the landscape and potential obstacles. Furthermore, flight data, such as speed, altitude, position and course, can also be displayed. In addition, a head-tracking system ensures that the projections adjust to the pilot's line of sight – depending on whether he is looking forward, downward or out the side of the cockpit.

The right amount of information

Does all of this information really help the pilot? To answer this question, the TUM researchers conducted a study with 16 professional helicopter pilots. They were allowed to test the new head-mounted display during various simulator flights. The engineers recorded how the participants flew and queried them afterward on their stress symptoms.

Viertler recently presented the results to the members of the American Helicopter Society in West Palm Beach, Florida: for ranges of sight below 800 meters, the pilots benefited measurably from the terrain and flight data displayed. They not only flew more quickly and more safely than without the head-mounted display, but they also felt that the flights had been less demanding both physically and mentally. The advantages of the system really came to bear when visibility was extremely poor, e.g. in visual ranges of just 100 to 400 meters.

Fighting the white-out phenomenon

"The new technology can reduce the risk when helicopters are operated," Viertler is convinced. "The main problem is poor visibility caused by clouds or snow, or dust blown up when taking off and landing. Augmented reality can help to overcome this white-out or brown-out phenomenon."

However, it may take some time before pilots benefit from this in daily practice: the collection, evaluation and projection of the data must first be tested in research helicopters. "We cannot leave that to our simulators," explains Viertler. Here, support from the industry is required. Fortunately, the industry has already shown great interest in this technology.

Video on Youtube: www.youtube.com/watch

Highresolution pictures: mediatum.ub.tum.de

Franz Viertler
Technical University of Munich
Tel.:   +49-(0)89-289-16362

Technical University of Munich

Stefanie Reiffert

Article at tum.de

Landing approach to a ship in the helicopter simulator.

Safe landings in rough seas

Landing a helicopter on the deck of a ship in poor visibility is a big challenge even for the most experienced of pilots. Researchers at the Technical University of Munich (TUM) have developed an assistance system that...

The flight demonstrator of the project FLEXOP at the Special Airport Oberpfaffenhofen.

Super-efficient wing takes off

In a joint effort by the Technical University of Munich (TUM) and the German Aerospace Center (DLR), researchers have successfully developed new technologies for lighter aircraft wings that are still extremely stable. These...

Flight simulator for aerial taxis

A vertical take-off into the mobility of tomorrow

In the near future we may see electric vertical take-off aircraft taking passengers to their destinations as aerial taxis high above traffic jams. But the flight system controls for these aircraft still represent a major...

Mann mit Virtual-Reality-Brille entwirft eine Turbine

Radical changes through virtual reality

Virtual and augmented reality will have potential applications in countless business sectors as the technologies mature. A study by economists at the Technical University of Munich (TUM) shows that drastic disruption lies...

Mit dem "eRay" kreierte das siegreiche Team der TUM ein ultraeffizientes Flugzeug. (Bild: DLR, CC-BY 3.0)

Ultra-efficient aircraft "eRay" from Munich

The German Aerospace Center (DLR) and the National Aeronautics and Space Administration (NASA) in the United States have announced the winners of their joint design competition in Washington DC. As in 2017, students of the...

Radioteleskope am Geodätischen Observatorium Wettzell

New Department of Aerospace, Aeronautics and Geodesy

In response to epoch-making developments in aviation and space flight, the Bavarian Minister President Dr. Markus Söder announced the establishment of a new engineering department at the Technical University of Munich (TUM)...

Hubschraubereinsätze können gefährlich sein. (Foto: assetseller / fotolia)

New simulation software improves helicopter pilot training

Missions at sea, in mountainous regions or close to skyscrapers are extremely risky for helicopter pilots. The turbulent air flows near oil rigs, ships, cliffs and tall buildings can throw a helicopter off balance and cause...

Prof. Nassir Navab erklärt Moderatorin Alex Forrest, wie die neue Technologie funktioniert.

Augmented Reality: TUM develops new tools for surgeons

In operating rooms of the future, doctors will work with glasses – glasses with a special feature: They will make it possible to look right into a patient's body. In the current episode of "Horizons" on BBC World News,...

Versuch zum hirngesteuerten Fliegen im Flugsimulator am Lehrstuhl für Flugsystemdynamik.

Using thoughts to control airplanes

Pilots of the future could be able to control their aircraft by merely thinking commands. Scientists of the Technische Universität München and the TU Berlin have now demonstrated the feasibility of flying via brain control...