What is the definition of Augmented Reality (AR)? For lack of linking to an official definition from Wikipedia, it’s overlaying digital objects on the real world. It started with an early version of the iPhone or Android phones, where you could fight Star Wars Tie Fighters by moving your phone around the 3-dimensional space of the room you were in.
More recently, AR manifested itself in one of the most popular games of all-time, Pokémon Go. It overlayed digital critters over the real world you were pointing your phone’s camera at. Throw a digital ball at a digital animal, capture it, get more points than your friends, and win.
Microsoft invented the Hololens, which we got early access to and later followed up with what would make AR a truly magical experience.
Apple’s 2nd worst-kept secret behind their self-driving car is for the Apple Glasses. Tim Cook can’t stop gushing about it to reporters. This isn’t the Jobs we’ve been accustomed to. We’ve been writing about what their patents mean for features, where their whole product strategy is headed, and how the AirPods are really just superhero ears.
Snapchat even came out with their version of special spectacles.
But it all pales in comparison for a project that’s been going for over a decade, won’t get fully delivered for another two decades, and already has a price tag over $1 trillion.
What is it?
It’s a $400,000 augmented reality helmet for an F-35 Lightning II stealth fighter jet. It’s built for United States military and manufactured by Lockheed Martin.
F-35 Lightning II augmented reality combat helmet for fighter pilots.
II. Details on the Helmet
You can read all about the F-35 Lightning II on Wikipedia, so we can focus on the helmet. First a bit on how the latest, Generation III helmet works.
- It’s custom fitted to each pilot’s head. They use 3D laser scanning technology to make sure the inner lining of the helmet is exactly form-fitted to the human who wears it. This is the exact opposite of one-size fits all.
- It weighs about 4 pounds and when you’re rattling around at supersonic speeds, it’s important that the helmet doesn’t slip around your head because the target you’re aiming at could be off if you do. They’ve tried to reduce the weight by making it of carbon fiber.
- You can look down below you or around you and see “through” the metal of the plane onto the earth and horizon below and in front of you. It will point out high value targets at a distance and in close range. Obviously it’s connected to a camera and sensor feed located around the plane, but is connected to the helmet with a series of cables. WiFi and Bluetooth aren’t real-time enough for these folks.
- It has a digital night vision mode projected directly onto the lens of the helmet so you can see in the dark. But of course, dear Bond.
- Last but not least, it has noise cancelling so the pilot can hear himself think. These ain’t your boss’s Bose headphone though. It produces “white noise” frequencies that exactly offsets and cancels the wind and jet engines.
- Some of the “situational awareness” sensors include: real-time video, thermal heat, enemy targeting and prioritization at varying distances, as well as stats like altimeter, horizon, level, air speed, etc.
Here’s the official Rockwell Collins video from a few years back:
My favorite part? When they call part of the digital display “Highway-in-the-Sky”. Sounds a lot like how an augmented reality self-driving car helmet would look like. But what are the basics for how this helmet works?
III. How To Detect Orientation & Speed
Many of the technologies outlined above were all made possible by a very simple piece of technology that found its way a decade ago into the original iPhone 1. It’s called an accelerometer and it, among other things, helps detect the orientation of the device.
It can be used in vehicles to determine if you’re driving uphill, whether your car is about to fall off a cliff, or if you’re flying horizontally or at an upward or downward angle. You can see how such a simple sensor can mean the difference between life or death.
And can you believe it costs less than $8?
What you see in the image above is a 3-axis accelerometer from the amazing adafruit. It can detect motion and, as you might have guessed, acceleration and deceleration from sensitivities of +/-2 Gs to +/-8 Gs. It also records dropping, vibration, shocks, bumps, and the like. If you want something more precise, you can go all the way up to +/-16 Gs for only about $18.
In later phones, they added a gyroscope for more accurate sensing.
What’s the benefit of a gyroscope? Quite a lot actually. Because you have 3 discs that spin, it helps determine a center point for maintaining level-ness. This is especially important for flying objects like drones that, if tipped too far to one side, will crash back to earth.
Another especially important benefit for space exploration is use as inertial navigation. Said differently, they replace magnetic compasses as a more accurate measurement device. They work in space, in the air, and on the ground. You’ll find gyroscopes in self-driving cars, self-flying drones and planes, self-navigating space rovers, and self-sailing ships.
Some of those words you haven’t heard very many times because much of the industry is focused on the $1 trillion market size of future car market. But it will become important for every type of transportation medium, whether that’s land, air, water, space, or on extraterrestrial planets or asteroids.
Two cameras for eyes, a gyroscope of 3-dimensional orientation like your inner ear. Both are important for any augmented reality mechanism.
It’s incredibly important for biological animals to have this mechanism working properly. Ever experience vertigo? Or dizziness? Likely an inner ear problem.
IV. What’s Next
As you might imagine, it’s still very early days in the AR and self-driving industry’s but I have a premise that both will become incredibly important to one other. Two sides of the same coin if you will.
If you lend out your self-driving car to Uber’s fleet, you’re probably going to want to check in on it and see where it’s headed or how its passengers are behaving.
If you’re the US Military and just spent a trillion bucks on a big ole smart plane, odds are you’re gonna want someone looking after it.
Same goes for that self-driving drone you’re about to get for Christmas when it goes out of eyesight (watch out FAA).
Last but not least, if you’re in the SpaceX rocket on the way to Mars and want to make sure the habitat that was supposed to be set up by the time you get there is making good progress, odds are you’re going to invest in your own set of glasses and video stream from a self-driving rover that lets you inspect whether that puppy is air tight.
Autonomy is great, until you want to take control. And then you need an extra pair of eyes. AR isn’t just for catching critters. It can be for saving highway lives as well as watching shooting stars.