AirAssist

A multi-modal reasoning aviation AI system built with pilot expertise. Fusing radio comm, weather, traffic, airport charts with aircraft data for hazard and situational awareness, recommend actions.

Comment

Inspiration

Every pilot has had more than their fair share of dangerous situations from information and task overload. One air incident that really stood out to us was a mid-air collision in Watsonville in 2022, where two small aircrafts collided while entering traffic pattern. The most striking part of this incident to us was the fact that it was such a simple scenario with no adverse weather, no congestion, and no mechanical issues. It was an incident that never should have happened and could have been easily prevented with better planning and awareness.

Airspace throughout the San Francisco Bay Area is extremely complex due to dense and complex airspace, lots of air traffic, and mountainous terrain. In addition, the nationwide shortage of air traffic controllers who are understaffed and overworked only exasperates the problem. The last line of defense against aviation accidents is the pilot's ability to make good decisions, but this can be hard when the amount of information from the radio, documents, and avionics systems is overwhelming. Meanwhile, an AI's ability to process massive amounts of information is unmatched. If "AI for all" is the future of humanity, then creating AI to assist pilots is the only logical next step in the future of aviation.

What it does

AirAssist is a multimodal aviation awareness system that fuses live radio audio, weather, traffic data, and airport charts to provide real-time advisory information. The system performs reasoning through situational awareness by using aircraft system information such as position, altitude, outside air temperature, fuel information, weight in order to provide the pilot with potential hazards and hazard avoidance information.

AirAssist simultaneously monitors information from multiple sources. Every new piece of information is assigned a timestamp to establish chronological awareness. Data is processed and formatted differently depending on the type of information it contains. All information is presented in the pilot dashboard. A real pilot was involved throughout the development entire process. AirAssist's functions and pipelines are based on real experience and process pipelines.

For information not available from avionics (such as weights and balances, fuel type), the pilot has the option of entering them manually. AirAssist is then able to perform weight calculations, taking into account fuel type and density.

AirAssist is a reactive system rather than a proactive one. This was a design decision that was directly the result of deep consultation with a real pilot. The motivation behind this decision is that the pilot-in-command must always be in control. The AI system provides analysis, points out details, advises, but it can never interfere with the decision making (provide only when asked).

How we built it

Gradio was used to build the application and UI. Two Gemini 3 models were used as the AI behind the reasoning. To link our application with Gemini 3, we used the google-genai Python library.

Since AirAssist is currently not connected to avionics systems, we built a control panel to spoof aircraft information as if obtaining it from an avionics system. AirAssist uses two Gemini 3 models: gemini-3-pro-preview and gemini-3-pro-image-preview, with internally context routing so the two models have consistent context and the same information to work with. This is critical to ensure consistent and accurate situational awareness.

In a fully integrated system, the audio from the communication radio would feed directly to AirAssist automatically. We implement the external hook by having AirAssist monitor a dedicated directory for new audio files. The expectation is that the external radio would write to an audio file to the directory. A thread monitors this directory and when new data is available, an ingestion worker queues it up for AirAssist to consume.

One important thing to note is that AirAssist may not be user-friendly to everyone. We designed the UI according to pilot feedback, tailoring every aspect based on a pilot's real workflow.

Challenges we ran into

  • Understanding airport operations, how pilots work, how aircrafts work, procedures in the air and on the ground.
  • Replicating real pilots' workflow and making sure Gemini 3 understands what pilots expect was the biggest and most critical challenge.
  • Understanding what information pilots use, what they are used for, where they come from. This allows us to integrate them into the system appropriately.
  • Working around model latency and creating fall-back measures to make sure it does not interfere with pilot and airport workflows.
  • For this project, we utilized two Gemini models: gemini-3-pro-preview as the main model and gemini-3-pro-image-preview as the secondary model. The reason is because gemini-3-pro-preview can take text, image, audio as input but only output text. Meanwhile, gemini-3-pro-image-preview can only take text and images as input but only output images. Radio, avionics, charts are given to gemini-3-pro-preview. When taxi instructions are detected, AirAssist needs to output a marked airport diagram, which requires gemini-3-pro-preview. AirAssist internally routes context information to gemini-3-pro-image-preview to generate the image of the marked airport diagram.

Accomplishments that we're proud of

Stressing the system through real situations pilots go through, we are very proud of the fact that AirAssist was able to catch critical details through reasoning. It was able to mitigate collisions, predict icing conditions, and anticipate low-visibility just to name a few of the things it was able to pick up. Throughout every step of the process, the expertise and feedback of a real pilot was embedded into every aspect from code architecture, to system design, to UI layout. The following are four scenarios we tested AirAssist against, and we are proud that it was accomplish all of them to a pilot's standards.

Scenario 1: Airport Analysis

AirAssist successfully analyzed San Francisco airport's complex taxi routes, identified runway intersections, runway orientations, obstructions, hazardous areas (hot spots), and compliance-related information.

Scenario 2: Weather Report

AirAssist presents weather information in a table formatted in the same way a pilot would. In this scenario, AirAssist was provided with a real radio of San Francisco International Airport's ATIS broadcast. From the report, the air temperature (9 degrees) is close to the dew point (8 degrees). AirAssist utilized Gemini 3's reasoning capabilities to warn the pilot of fog and low visibility. AI system reasoned that the small difference in temperature and dew point indicates a very high likelihood of fog and poor visibility.

Scenario 3: Collision Avoidance

A collision scenario was set up to test AirAssist's hazard awareness and planning capabilities. In this scenario, AirAssist is approaching to land and is 7 nautical miles away and at a bearing of 200 degrees from an airfield at an altitude of 1500 feet. Another aircraft is making the same approach with the same bearing and altitude, 6 nautical miles from the field. AirAssist issued a critical warning to the pilot with the relative position of the incoming aircraft, recommended next steps to remain predictable, and information to lookout for. The AI system was able to not only deduce that a collision is imminent, but also that the second aircraft is directly behind the pilot, with a separation of 1 nautical mile.

Scenario 4: Taxi Instruction Marking

Real radio communication was used to implement and test AirAssist's ability to identify taxi routes pilots need to take, given by air traffic controllers. Given an airport diagram of San Francisco International Airport, AirAssist was able to correctly mark the taxi ways given by the air traffic controller through radio (audio).

What we learned

Gemini 3's reasoning capabilities are incredibly powerful. We were impressed by its ability to understand a domain-specific field as complex as general aviation.

We also learned the importance of understanding the field thoroughly when implementing a usable solution. Working through countless hours testing Gemini 3, we really got to understand both gemini-3-pro-preview and gemini-3-pro-image-preview capabilities and limitations thoroughly. AirAssist was created by bridging Gemini 3's capabilities and limitations to domain expertise.

We also learned that Gemini 3 does not perform particularly well when audio is noisy, particularly when lots of static and abrupt segments of communication are cut out (both very common in aviation). In addition, Gemini 3 has a difficult time making the distinction between "1" and "7" on airport diagrams. In times where it was able to accurately interpret the audio taxi instructions as "D7", it incorrectly marked "D1" thinking it was "D7".

Understanding what is the appropriate level of automation was also another challenge. HCI research suggests that although automation can potentially improve situational awareness, excessive amounts of automation actually decreases situational awareness. This solved by consulting domain expertise.

What's next for AirAssist

Several areas that can greatly improve AirAssist:

  • Embedding more documents: aircraft spec sheets, regulations, noise abatement procedures, pilot checklist, etc.
  • MCP for computing aircraft performance: glide distance, path planning after engine failure, weight and balance calculations, "aircraft envelope", etc
  • Deeper integration with avionics systems to allow MCP system monitoring and debugging: engine temperature, oil pressure, air/fuel mix ratio
  • Run the model on-device model: currently AirAssist requires an internet connection to work with Gemini 3
  • We would like to create a version of AirAssist for drones. The development of delivery drones has come a long way. We want to embed an experienced pilot's planning and decision making abilities into drones. In this scenario, AirAssist would need to be a proactive agent instead of a reactive agent.
  • We can greatly simplify AirAssist's software architecture if a single model can combine both the features of gemini-3-pro-preview and gemini-3-pro-image-preview. This would eliminate the need of context routing and would eliminate any risk (however small) of inconsistent context information.

Built With

Share this project:

Updates