Inspiration
Philadelphia Legal Assistance (PLA) operates a hotline to help tenants who are facing eviction, but many tenants have to wait on hold for a long time because PLA does not have enough funding to fully staff the hotline. This is not fair to tenants. While some tenants need to speak with a human, others might be able to receive effective assistance from a web-based chatbot, which can do many of the things a human hotline operator can do, like answer specific questions, look up information about a tenant's situation, or schedule a tenant for an appointment. The purpose of this project is to explore to what extent ChatGPT can be trained to do this, and see what we can learn about the prospect of providing one-on-one assistance through a chatbot interface.
What it does
The web app, called EVITA, provides a simple, user-friendly chat interface to the user. The user can ask questions about the eviction process in Philadelphia and the chatbot answers them, not based on ChatGPT's background knowledge, but based on specific information that we gave ChatGPT about the eviction process in Philadelphia.
Based on our testing, EVITA performs well. It can figure out what the user is asking and it is good at basing its responses on the information we gave it, rather than using information from other jurisdictions or hallucinating facts that do not exist.
Although we did not have time to integrate them into the web app, we tested a variety of additional features:
- If the user has an eviction case in court, the chatbot can gather the user's name and address and then look up the user's court case and summarize what is happening in the case.
- The chatbot can also look up and summarize the code violation on the property, using information from OpenDataPhilly.
- If the user needs a housing counseling appointment, the chatbot can schedule the user for an appointment, from a list of available appointments maintained in Google Calendars.
How we built it
We built a front end in React and a back end in Python. The Python back end uses the OpenAI API to call ChatGPT. The user does not interact with ChatGPT directly; the web app processes the user's input, relays it to ChatGPT along with specific instructions, and processes the output that ChatGPT returns.
In order to control what ChatGPT knows and says, we developed "prompts" for ChatGPT. The user does not see the content of these prompts. Through "prompt engineering," we trained ChatGPT how to act like an operator on a hotline operated by Philadelphia Legal Assistance.
Challenges we ran into
One challenge is that the amount of information ChatGPT can work with at one time is limited. So we had to make our prompts concise.
We had to adapt the prompts through a process of trial and error to make sure that ChatGPT "stayed in character."
The response time is slow and we will need to do something about that, like give the user an animation to look at while ChatGPT is processing the input.
ChatGPT responds differently each time; there is randomness. Sometimes the responses would be good and other times the responses would contain something confusing.
Accomplishments that we're proud of
After a lot of iteration, we were able to develop a pretty good prompt with a lot of specific information about Philadelphia' eviction process.
What we learned
ChatGPT, particularly version 4, is incredibly good at following instructions written in plain English. For many of the special features we tried to develop, ChatGPT worked correctly on the first try.
We learned that it is only necessary to train ChatGPT about Philadelphia-specific aspects of eviction. For example, it was really good at explaining what mediation is and how a tenant should approach a mediation with their landlord. We didn't need to teach it that because it is not Philadelphia-specific.
What's next for Eviction Help Chatbot
If the technology is good enough, we would like to build a voice-enabled version of the chatbot so that users could use it over the phone.
We need to continue to do testing. We need to test it by trying to get it to "break character." If the user submits inappropriate input, we want to be able to end the conversation. We need to methodically test for incorrectness, bias, and hallucination.
We then need to take what we have learned and present it to the community of service providers, so that we are all better informed about the benefits and risks of deploying AI technology.
Log in or sign up for Devpost to join the conversation.