Clipped

🛒 Goal

Prevent and stop shoplifting in stores through constant surveillance technology.

📹 What it does

Feeds real-time surveillance footage through our machine learning model, which then classifies individuals as suspicious or normal. Texts employees and alerts them of individuals deemed suspicious.

🤖 How we built it

Before putting the frames of a clip of a video into the machine learning algorithm, we pre-processed the input data by categorizing the video clips of shoplifters labeled as either normal or suspicious (on posture, movement, etc.) based on a popular Kaggle dataset. Libraries such as os and matplotlib helped us traverse multiple mp4 files in the directory of a computer and visualize prediction models before implementing a neural network. Additionally, we utilized Python OpenCV to break down the video clips into labeled image fragments that are passed as data for the machine-learning algorithm. After the pre-processing of the cumulative data given (over 800 clips), we passed the data through a convolutional neural network that is optimized for image classification alongside sequential long-short-term memory models (LSTMs) that account for both spatial and temporal data. During the implementation of the model itself, the neural network consists of many hidden layers and nodes, such as the ConvLSTM2D layer that utilizes filters and a tanh activation function to find patterns, the MaxPooling3D function that reduces the image size while increasing the number of filters, and TimeDistributed(Dropout() ) which drops 20% of the results at each layer to prevent overfitting. Finally, it uses the convlstm.fit() method to train the dataset by optimizing weights and biases each time the training data goes through the model (called an epoch --> was done 11 times to get an accuracy of 95% from the model with a loss function of only 15%).

🚀 Challenges we ran into

One challenge we ran into while working on the project was optimizing the amount of RAM processes such as creating a sample dataset of an even number of normal and abnormal video clips and training our convolutional neural LSTM network with all its hidden layers. Multiple times we would run the convlstm_model.fit() function in CoLab but would exponentially run out of system RAM (35,5 GB) before the notebook would crash deleting all of our trained model progress at that time.

🌟 Accomplishments that we're proud of

As our whole group is comprised of aspiring freshmen interested in the field of back-end AI programming, taking on the challenge of developing an accurate theft detection model system using a nuanced network was an obstacle, to say the least. From learning how to effectively break down seconds of video clips into classifiable image fragments from our laptop's camera to incorporating both CNN and LSTM models to determine suspicion of shoplifters, the last 24 hours helped us vastly grow our skills as future software developers, especially with working with narrow artificial intelligence. Furthermore, although we struggled initially to come up with ideas to feed a real-time camera into our machine-learning model, we were able to standardize the number of clips the longer clip was chopped up into (3-4 seconds to analyze 15 frames per epocha0 while maintaining a similar efficiency to that of prescribed, training data.

📈 What's next for Clipped

The next step for Clipped is working on integrating these theft detection models into stores such as Wakefern's branches of supermarkets and other retail stores that require a contemporary surveillance system capable of alerting the owner or a security guard of possible suspicion. In this context, it would have the ability to detect if a customer really checked out their product at self-checkout or if they are suspiciously hiding something in their pockets while leaving the store. Additionally, we want to sell a more refined and fine-tuned version of this model, with potentially an even lower validation loss function, to companies who may want to invest in not only the safety of both their products/inventory but also the customers who shop often at their stores.