ONLINE SUBMISSION
Inspiration
The inspiration behind NESGPT is the need for a more intuitive and visual way to assess ESG (Environmental, Social, Governance) metrics. Traditional ESG scoring involves answering many questions and can vary from company to company, making it less accessible and harder to standardize. NESGPT aims to simplify this process by providing a 'New Earth Score' that leverages AI to analyze and annotate geographical images for better environmental insights. Automating months and millions of ML and Data Science tasks in minutes, reducing technical debt.
What it does
NESGPT takes a screenshot from Google Maps and uses DINO + GPTV models to auto-annotate the image. It then sends this annotated image to GPTV, identifying potential carbon sinks or carbon sources and extracting other relevant data through advanced prompt technology. This system is designed to provide actionable insights for improving ESG scores by suggesting solarpunk solutions and other environmentally beneficial practices.
How we built it
The system was built by integrating various advanced technologies. The DINO + ChatGPTV model is annotated, while Gradio is used for Hugging Face hosting. The GPT model serves as the main interface through assistants, connecting to the model hosted on Hugging Face. MindsDB is employed to query relevant business information for ESG scoring.
Challenges we ran into
A significant challenge we faced was aligning the outputs from our AI models with the expectations of our system's ontology. Specifically, the detailed and descriptive responses from our image annotations did not match the more straightforward prompts we had initially defined. This discrepancy led to errors when processing images, highlighting a need for better alignment between the model outputs and our system's framework.
Additionally, we encountered a pivot point in our project's direction, which required us to reassess and realign our objectives with the evolving understanding of our user's needs and the business landscape we are aiming to serve. This pivot was not only a challenge but also an opportunity for growth and refinement of our project's goals.
Accomplishments that we're proud of
The team is proud of creating a novel approach to ESG scoring that goes beyond traditional methods. By using AI to provide high-level, actionable insights specific to geographical images, NESGPT can potentially revolutionize the way local governments, environmental agencies, and businesses assess and improve their environmental impact.
What we learned
Our project was a deep dive into the practical application of cutting-edge AI technologies. We expanded our knowledge on the use of Gradio, which proved to be a powerful tool for building interactive web interfaces for our models. Hugging Face's hosting capabilities allowed us to seamlessly deploy and manage our models, and OpenAI's assistants provided an advanced user interaction layer that was crucial for our interface. Last but not least, we really pushed our limits with how well we could do prompt engineering and learned some new tricks along the way.
Moreover, we gained valuable insights into leveraging GroundingDINO, an innovative approach to object detection and image annotation. This experience taught us how to better integrate visual data processing with language models, enhancing our ability to extract meaningful insights from images. We also became more adept at tackling the complexities of ESG scoring, learning to create a more accessible and actionable framework for these important assessments. These learnings are not just technical but also strategic, as they guide us in making ESG scoring a more standardized and insightful tool for stakeholders.
What's next for NESGPT
The next steps for NESGPT include refining the business case and continuing to develop the model's capabilities. This includes expanding the range of actionable insights provided, enhancing the accuracy of annotations, and exploring partnerships with local governments, environmental agencies, and businesses interested in ESG metrics and environmental impact.
Log in or sign up for Devpost to join the conversation.