TreeHack

TreeHack!
Map Data with overlays
Choice of nearby areas to analyze
Canopy Overlay

Inspiration

This July was the hottest month ever
Trees are effective in cooling the city (10˚ cooler than no-tree areas, 2-4x faster at cooling)
But the bottleneck is knowing specifically where to plant trees (how do we know? a city mayor said it in an interview)

What it does

After the user selects a location on the map, our platform performs three things:

Identify and outline urban heat islands in the neighborhood
Locate specific streets where a) temperature is high, b) trees are underutilized, c) the sidewalks are wide enough to plant trees
Propose a solution with a) visualization of what each street would look like with tree canopy, b) recommendations of native tree species that can be planted, c) estimate the benefits of this course of action

How we built it

Part I. Overlaying heat, canopy, and air pollution data on the map

Heat data: we used Mateo Blue's surface temperature API
Canopy (trees): there's no direct canopy dataset, so we bootstrapped with Google Maps' Satellite API. We scanned a 5-mile radius from a point of interest. To calculate canopy density, we built a lightweight computer vision algorithm from scratch to detect trees in aerial images.
Air pollution: we used Google's pollution API.
Map: @Jenny @Balaji carried us
Part I isolates specific small regions in a large radius that have potential to be improved via tree planting. Next:

Part II. Identifying specific streets suitable for planting trees

Getting street view: Google StreetView API
Assessing whether street 1) already has a canopy presence, 2) if not, does it have space for trees to grow: we used OpenAI's CLIP API @Andy

Part III: Recommendation and details for planting trees on this street:

Generating visualization of what the street could be like if it had canopy: we integrated OpenAI's Dall-E API
Finding suitable trees to be planted in this region (soil, climate, maintenance, and we want to promote native species): we applied OpenAI's GPT-3 API
Benefit estimation: we drew conclusions from research papers (here's a meta study)

Front and Back End:

Balaji and Jenny took care of arguably the nastiest part of this project - the front-end, which consists of building a map using the Leaflet library, overlaying data on it (drawing them polygons and coloring them shades), working with kml and geojson files, marking user input coordinates

Andy and Ryan took care of the algorithm and the backend, creating computer vision algorithms, applying LLMs, and ensuring the Python code used for the LLM works in the Typescript environment of our overall project.

Challenges we ran into

Bottlenecks from big to small

Adding trees to a street is surprisingly hard. Generative models aren't good at adding objects without altering the rest of the picture. We tried using Stable Diffusion, ControlNet, Dreambooth to finetune Stable Diffusion, Dall-E, creating pipelines of two foundation models, and countless variations of prompts and parameters. We gave it our best shot and eventually found a set of prompts that generated good results.
Assessing canopy coverage (which streets lack trees) took us a long time. Because 1) we needed fine-grained, street-level data; 2) the closest match, Google's canopy dataset is inaccessible as it has a waitlist; 3) tree classification is difficult because we needed to iterate through hundreds of satellites images. So, we created our own dataset from raw satellite images and an in-house computer vision algorithm (>95% accuracy, processing 50 images in <5 seconds).
Map. Perseverance. Patience.
Google's StreetView image defaults to a northern direction. Their photographer-pov API has been deprecated. So we created our own computer vision model to ensure that our street view faces down the street.
Finding the right datasets, reformating them, and parsing GBs of data to fit our application.

Accomplishments that we're proud of

We LEARNED so much! From how to fine-tune stable diffusion to working with Leaflet map library. Each one of us walked away doing something we had no idea how to do a day ago.