Air Forces all around the world want more pilots. The latest figures from the US Air Force, the largest in the world, are that it is around 2,000 pilots short. There are also not enough aircraft. The UK’s RAF has around 300 trainees on hold for training because they don’t have enough airplanes, despite urgently needing new pilots, an admission made in a report that leaked at the beginning of this year.
The fundamental challenge is that fighter pilot training is almost unimaginably expensive and takes a long time – up to US$11 million and three to five years per pilot. Ground simulators are useful to a certain extent. But they cannot fully recreate the physiological and psychological effects of flying. Eventually, a pilot has to fly live training missions.
Nowhere is the problem more apparent than with combat training in fast jets, which require extra pilots and aircraft pretending to be the enemy. The cost in terms of extra aircraft is high enough, but factoring in the already scarce use of pilots as enemy actors make it painfully expensive.
“If two pilots are flying, it’s not two people training, only one of them is getting training. And all the time an F-22 costs US$100,000 an hour to fly,” says Daniel Robinson, founder and CEO of Red 6. “Put simply, there is a massive demand for adversary training and a massive undersupply, which we will never solve with physical assets.”
Augmented reality outdoors
This seemingly intractable problem is one which Robinson set out to solve in 2018. His proposed solution is to accurately recreate aerial combat situations while in an aircraft using augmented reality (AR) technology. AR superimposes computer graphics in front of people’s eyes using a head-mounted display (HMD).
Robinson first realized AR could bridge the gap between ground simulators and live flight training while watching an example of virtual reality (VR) being used to race cars, while the drivers were in separate locations. The demonstrations used technology that had been developed by Glenn Snyder.
In 2018 Robinson co-founded Red 6 with Snyder with the purpose of adapting AR technology into a system that could be used during live flight training. By 2019 the company had conducted its first demonstration.
The Advanced Tactical Augmented Reality System (ATARS) has been in development and testing since then. Last year the company hit a key milestone when it conducted the first ever live training flight using AR to simulate a mission with multiple aircraft.
A blended background
Just as his product blends reality, Robinson is a mix. Part tech entrepreneur and part fighter pilot, part Brit and part American, he has a nerdy knowledge of AR and aviation and displays a dogged determination to bring ATARS to market. He also knows which sport football really is, having lived in the UK and flown Tornadoes for the RAF before moving to the USA, where he became the first foreign national to fly the F-22 Raptor.
The first thing that becomes apparent when talking about AR for fighter pilots is how tough a technical problem it poses. Or according to Robinson – posed. ATARS is now moving from being an advanced prototype to being a product.
This year the system should be integrated and used for the first time with the US Air Force’s T38 Talon training aircraft. This will be the first of many aircraft types that will use ATARS, Robinson believes.
(Photo: UK MOD)
Early challenges
AR applications so far, using devices like Microsoft’s HoloLens, are somewhat more pedestrian than recreating aerial dog fights in fighter jets. For example, providing information about parts to remote maintenance technicians.
Robinson, who is the company’s chief test pilot admits the R&D program has had its tough moments – early versions left burn marks on his face after testing: “It was an amazingly tough technical challenge but it is a disruptive technology. Yes, there were times in the early days when it seemed too hard.”
Robinson explains how Red 6’s enabling technology works by describing the hardware and software aspects: “For it to work, we need to understand where the pilot is looking at any point in time in six degrees of freedom, relative to the real world,” he says. “In other words, tracking. The AR system then puts synthetic entities into the world around the pilot.”
An AR system accomplishes tracking with a technique called simultaneous location and mapping (SLAM), which combines data from optical and inertial sensors. However, once an AR system moves outside SLAM becomes a massive computational challenge, because of the size of the area. It becomes even more difficult when flying upside down at 400mph (640km/h) pulling multiple gs.
The second main challenge for the hardware was developing the optics for the HMD. The display has to be bright enough – in terms of thousands of nits – with a sufficiently high resolution and low latency. It also has to have a field of view that mimics human vision at 200˚.
Robinson says, “We figured out the tracking and then quickly realized we had to design our own optics. The result is without a doubt the most capable AR system in the world, able to present the user with high resolution daylight-capable images, rendered in a sufficiently low latency so that it makes sense to a human being.”
The software side of ATARS involves rendering graphics, processing data for SLAM and other elements of the system, and transporting data around a network. It includes a standalone computer linked via telemetry to the aircraft.
Perhaps unsurprisingly, Red 6 has used video game designers and software engineers to develop the graphical models and AI algorithms that control their behavior. The high amount of computer processing necessitates the remote computer, but this was found to present challenges.
“When you start sending things across the network latency becomes a problem,” says Robinson. “So we created a protocol to compress the latency out and only throw small packets of information across the network – where you are and how you are orientated at any point in time. All the rendering is done locally.”
Testing times
The development team has conducted more than 1500 hours of flight testing for ATARS using three Berkut 540 test aircraft from Santa Monica Airport in California. Robinson is the chief test pilot and built the first Berkut himself.
The experimental Berkut 540 has a top speed of 300mph (422km/h) and features a bubble fighter’s cockpit. The company, which employs just under 100 people including around 70 engineers has also delivered more than 300 external demonstrations of the technology to members of the defense industry.
“The team is a blend of hardware and software engineers, mechanical and electrical engineers and highly experienced fighter pilots who provide operator input,” says Robinson. “We have a lot of fifth generation aircraft experience, covering every platform.”
ATARS is designed to be platform agnostic, but space, power, heat and vibration is different for every aircraft, so the system has to be customized for each platform.
The immediate focus is on integrating and testing ATARS for us with the US Air Force’s T38. Then it will go into Boeing’s TA4 and later this year the Hawk in the UK. “It’s going to be a really busy year for us,” says Robinson. “We also aim to achieve the MIL-SPEC standard with ejection seat testing this year. We’ve signed deals with Boeing, BAE Systems and Lockheed Martin and we’re about to sign another deal.”
Robinson believes the future value of the market for ATARS to be as large as US$24 billion and does not see ground simulators as competition.
“Simulators are great for repetition-based tasks, but there is a lot of difference between making decisions on the ground and making them in a high-g environment. The cognitive pressure is massive,” he says.
“There is no doubt you have to fly live aircraft, but they can be flown cost effectively and in a relevant and effective way at scale against peer adversaries in complex environments. We need to get the most and the best we can out of live flying. In the future we will enter the training airspace and all adversaries for each pilot training will be synthetic.”
Data benefits
ARCADE is a debriefing system supplied with ATARS which allows pilots to refly missions in VR on the ground. The system includes biometric information such as eye tracking. This enables pilots to analyze to an “excruciating extent” decisions and ways to improve.
Beyond individual training, Robinson sees massive potential for operational applications using AR in the future: “Fifth generation aircraft were game-changing in the way they took multiple data sources and fused them together,” he says.
“The next big leap will be the fusion of complex datasets and taking it further out into the battlespace, representing that visually in 3D to the pilot. A picture paints a thousand words.
“It is a dramatic difference from the Tornados I started off my career flying.”
