Dutch SKies

Air traffic over The Netherlands
Approaches to Schiphol Airport
Plane in the sky with MR overlay
User interface windows

Inspiration

Living in Amsterdam, with Schiphol Airport nearby, the sky here is always full of planes. Some pass quite close by when landing, some can be seen climbing after having taking off, while others criss-cross high over everything else.

There's a few services (such as OpenSky or FlightRadar24) that can show air traffic on a 2D map, but it would be great to have a 3D mixed reality view of that same map. It would allow intuitively understanding the flight tracks of all those planes and see the patterns and organization that underlies modern air traffic control. And to be able to inspect what plane just flew over your head, and how fast and low it was going, just by looking at at would be a great use for mixed reality!

What it does

Dutch SKies runs on a Microsoft HoloLens 2 and provides two simultaneous views of real-time air traffic data:

A map view, with planes at their correct position and (relative) altitude, and,
A virtual overlay of real-time plane data (roughly) aligned with the actual planes in the local sky region of the HoloLens user.

Dutch SKies has a few maps of The Netherlands built in, but it is fully possible to use custom map images and locations. Such custom scenarios can be loaded using a QR code, to fetch the necessary configuration data and map images by URL.

How we built it

The basis of the application was built using C# UWP with the awesome StereoKit, as application framework, for 3D rendering and UI interaction. The real-time air traffic data is retrieved from the OpenSky Network. Microsoft's QR scanning library was used to be able to use QR codes to control the map being used, or the observer location for the sky overlay.

An initial prototype showed enough promise to further develop it, in total over a period of around 3 (intense :)) weeks. The use of plain C#, without any kind of extensive game engine or 3D GUI development environment actually made this so much easier, as it allows you to focus on only those elements you need in the code. The code structure also stays quite simple, as it is basically a render loop, with some worker threads for background tasks. Also, StereoKit provided just the right amount of features for this project and feedback from the lead developer during the project was quick and very useful.

Challenges we ran into

There is an inherent challenge in overlaying real-time data in mixed reality over planes flying through the sky. The overlay will not be perfect, due to latency and inaccuracy of the air traffic data used, together with the challenge of aligning a HoloLens's 3D virtual world with the physical world on a scale of tens of kilometers and to within a few degrees of rotation. But we can come close! Certainly close enough to unambiguously augment real planes with mixed reality data. But even when using fairly precision location information for the observer (e.g. GPS lat/lon coordinate and altitude), the overlay was not precise enough. So manual trimming, based on landmarks, had to be added as a feature.

Using QR codes for easy switching between different map and observer configurations works and allows an arbitrary URL to be used through which the configuration data is retrieved. But it certainly isn't a perfect solution, as it is a bit cumbersome. In general, getting data interactively to/from the HoloLens is still a bit of a struggle. I did a bit of testing in embedding an HTTP server on the HoloLens side, for easier control, but that was a step too far for this hackathon (possible future extension).

A somewhat frustrating and unique challenge was the weather! There were lots of days with a low cloud cover and/or rain, making it hard or impossible to see planes in the sky, and thus not many opportunities to test the code (or film).

Finally, although hand interaction on the HoloLens 2 is vastly improved over version 1, it is still a bit suboptimal, leading to frequent misgrabs or misclicks (or a UI window getting stuck to your hand). This hampers both development as well as the application experience, and is one of the reasons for not adding too many parts that can be interacted with (e.g. the map or planes themselves), other than the UI windows.

Accomplishments that we're proud of

Having (virtually) looked at air traffic for the past weeks I've learned a lot of things on traffic patterns that I didn't know before, even though I've lived near an airport for 15 years and frequently look at plane tracking websites. So a mixed reality view (both map and sky) showed more added value than I had expected, which is really great.

The map view is not based on an existing interactive map service, such as Bing Maps. It is rudimentary in functionality, but it's easy to add your own maps and the full code (which isn't a lot) is included. In principle, all kinds of data could be shown in these types of maps and real-world views, not just air traffic. One reason for going with this custom approach was that I wanted enough detail to be visible in the map - place names, airports and their runways, etc. - even when using a country-scale map, without having to resort to interaction for zooming/translation. This leads to requiring quite a high resolution of the map image (around 4k x 4k) and I wasn't sure rendering the necessary high number of tiles would leave enough performance, compared to a single texture image that could be rendered in a single draw call.

The use of landmarks to fine-tune the alignment of the HoloLens coordinate system with the real world was a late addition, but a real "aha!" moment when the idea came. It is easy to use, while still providing a benefit to improve the mixed reality overlay. It is also a nice check of the consistency of the mixed reality world against the physical world.

What we learned

StereoKit is very easy to use to build XR applications, a real gem. Having previously worked with Unity and MRTK (and a bit in Unreal Engine for a non-XR project) using a plain C# framework with the necessary concepts a few API calls away was such a breeze. For smaller projects like this one, or for quick prototyping, I see no reason anymore to use a game engine.

There's all kinds of patterns to air traffic that are hard to spot from either looking at the sky or looking at a 2D map. Air traffic inherently travels in 3D space and it's only through depth perception with mixed reality (or VR) that you can really see how complex air spaces really are, and what 3D traffic patterns are needed to keep everything safe.

What's next for Dutch SKies

At SURF we like to keep track of technological developments that are interesting and useful for higher education and research. As such, we sometimes develop small tech demos to inspire and inform. DutchSKies is almost ready to serve as such as demo. Adding a synchronized multi-user view of the MR world would be a good next addition for that use case. Adding map interaction (e.g. zooming) and plane selection would be nice to have. Making the code work for VR would also be very interesting, although mostly for the map view (and increased FOV). We're also interested in what the application could bring to the Dutch air traffic control community in general.