AirAssist is a multimodal aviation awareness system that fuses live radio audio, weather, traffic data, and airport charts to provide real-time advisory information. The system performs reasoning through situational awareness by using aircraft system information such as position, altitude, outside air temperature, fuel information, weight in order to provide the pilot with potential hazards and hazard avoidance information. AirAssist simultaneouly monitors information from multiple sources. Every new piece of information is assigned a timestamp to establish chronological awareness. Data is processed and formatted differently depending on the type of information it contains. All information is presented in the pilot dashboard. A real pilot was involved throughout the development entire process. AirAssist's functions and pipelines are based on real experience and process pipelines.
The following screenshot on the left is the pilot dashboard, where the pilot-in-command interacts with AirAssist. Here, AirAssist successfully analyzed San Francisco airport's complex taxi routes, identified runway intersections, runway orientations, obstructions, hazardous areas (hot spots), and compliance-related information. The screenshot on the right is the control panel. Since AirAssist is currently not connected to avionics systems, the control panel is used to spoof aircraft information AirAssist would be able to obtain if integrated with an avionics system, ForeFlight, Garmin Pilot, etc.
Weather information (ATIS / AWOS / ASOS) are presented in a table formatted in the same way a pilot would. In this scenario, it is clear that at ground level the air temperature is close to the dew point. AirAssist utilized Gemini 3's reasoning capabilities to warn the pilot of fog and low visibility. The small difference in temperature and dew point indicates a very high likelihood of fog and poor visibility, which the AI system picked up on through reasoning from the information provided.
A collision scenario was set up to test AirAssist's situaltional awareness and planning capabilities. In this scenario, AirAssist is approaching to land and is 7 nautical miles away and at a bearing of 200 degrees from an airfield, at an altitude of 1500 feet. Another aircraft is making the same approach with the same bearing and altitude, 6 nautical miles from the field. AirAssist issued a critical warning to the pilot with the relative position of the incoming aircraft, recommended next steps to remain predictable, and information to lookout for. The AI system was able to deduce that the second aircraft is directly behind the pilot, with a separation of 1 nautical mile based off positional information.
If taxi instructions are detected, AirAssist summarizes the instructions in text and simultaneously generates an image of the airport diagram with the taxi route marked. The purpose for this dual mode of processing is the result of image generation latency. In a real-world environment, communication is expected to move fast, delays cause confusion and congesrtion. Text is generated quickly, giving the pilot something to work with while the image generates.
