PHOBOS: Generation of Feature-Accurate Martian Horizons

We propose PHOBOS, a system that generates feature-accurate photorealistic imagery of unseen Martian surfaces-- built on a GauGAN2 model that receives terrain and geologic data.

Comment

Inspiration

There are many unexplored locations on Mars and we wanted to know what they may look like from our point of view.

What it does

PHOBOS is a novel system architecture, built on a GAN style model and extensive preprocessing pipeline to create feature-accurate Martian horizon images. This system uses existing Martian elevation data and geologic records as a constraint for image generation-- as far as we are aware, this approach has not been previously implemented. This creates the ability to not only see Martian textures, but to actually create scientifically valuable landscape imagery.

How we built it

  1. We scraped various open-ended databases to create a complete dataset of ~8,000 Martian horizon images. This process took a while, and we trained a sorting model to determine if a dataset contained "valid" landscape imagery.
  2. We hand-segmented 200 of these images using RoboFlow, an online tool for collaberative database annotation for teams.
  3. Using this annotated dataset, we trained a new header for a DinoV2 model, a modern tool for instance segmentation.
  4. The fine-tuned DinoV2 model was used to segment all 8,000 Martian landscape photos, segmenting each aspect of the image into classes (such as craters, hills, sand, rocks, etc)
  5. Image pairs were fed into a GauGAN2-style model. This model was trained over 150 epochs on 6 NVIDIA A100 GPUs. This model took around 6 hours to train, and consumed about 50 hours of distributed computing time.
  6. This trained model, now named PHOBOS, was cable of converting label maps into photorealistic Martian images.
  7. We took Martian elevation data and geologic data to create new, unseen label maps. These label maps represnt real landscapes in Martian areas that have never been explored.
  8. We created photorealistic images of Martian horizons, some of which in areas that have never been explored from the surface.

Challenges we ran into

  1. Finding relevant photography to use in the model's training. This took hours, and required us to siphon through dozens of databases, each with many thousands of images.
  2. Compositing data. It was difficult to standardize the data, getting all images to the same size, getting all of the labels to correspond to the correct classes.

Accomplishments that we're proud of

  1. Creation of PHOBOS, the novel architecture.
  2. Creation of a consolidated dataset of Martian imagery, a contribution to scientific endeavors
  3. Speedy application of many complex models to achieve a concrete, positive result.

What we learned

  1. Plan ahead, we started pretty late because we were debating ideas.
  2. The hardest parts are what you overlook, we were not expecting the dataset creation to take as long as it did.
  3. You only get one shot with model training like this in this time period. If the model had failed, we would not have had time to retrain it. Probably would not take that risk again.

What's next for Generation of Feature-Accurate Martian Horizon Imagery

  1. Development of the automated pipeline for elevation data.
  2. Currently, this is a supervised process. Being able to create an unsupervised model that can conduct this style of content generation would be highly beneficial.

Built With

Share this project:

Updates