code-dot-org icon indicating copy to clipboard operation
code-dot-org copied to clipboard

Error on profane response from model

Open bencodeorg opened this issue 1 year ago • 0 comments

This PR applies the profanity filter to responses received from Sagemaker. If a message is found to be profane, we display a generic "There was an error getting a response. Please try again" message, and do not include the user message that preceded the profane response in the next attempt to make a request to the model.

image

From a logging perspective, I'm looking to log the profane content somewhere (last step in this PR, location TBD!), but not as part of our normal logging session, since I didn't want to forward the content back and forth between Rails and the front end (or expose it at all).

Testing story

Tested manually that a stubbed version of the model producing profanity was flagged and resulted in the error message in the screenshot below. Also tested that normal messages and logging continued to work as expected. Added a unit test to cover this case.

bencodeorg avatar May 01 '24 18:05 bencodeorg