Defectors

Defectors - Automate your data
Generation structure based on DefectGAN
Generated defects
Web Dashboard
Training Dashboard
Classification

Inspiration

Automated defect inspection is critical for effective and efficient maintenance, repair, and sustainable production in advanced manufacturing. On the other hand, automated defect inspection is almost always constrained by the lack of defect samples, instead of working on detection, we fix the underlying issue by replicating the defect creation process!

Defect-GAN Structure

What it does

While the goal remains to be able to identify anomalies as accurately and effectively as possible, it's not the detection itself that is the actual challenge. Our project has to be able to recreate every single possible damage type on golden-samples and then use that to generate a dataset of reasonable size for the actual detection training. In the current era of machine learning, training a model given the perfect data has become trivial, that's why detection is only the cherry on the anomaly-generation based cake. 🎂

Based on Stable Diffusion, we use the already pre-trained SD-in-painting models as a starting point. In-painting with Patchmatch is more than capable of removing important elements from a golden sample, which is the first step to both, defects through removal and defects through anomaly introduction to the image. If a defect can be put into the group of "missing elements" then the in-painting model does the job completely, removing a labeled element and then filling in with background information. If a defect can be put into the group of "unwanted elements" then the first step remains the same - first we 'fix' an image with a defect by removing the unwanted addition. With the element removed, and gold sample restored, we can then calculate the absolute difference between the restored image and the original defected image, finding out what type of diff-mask modification can turn the restored image back to a defect. The absolute difference masks are then used to train a generative adversarial network, which is a lot faster and easier than training the GAN to create a whole defect image from scratch.

Generated defects

How we built it

Having had a lot of experience with PyTorch Lightning, we used the wrapper library to write most of our machine learning tasks from GANs to the actual classification network. Our classifier uses a simple, but proven by experience structure, containing three convolutional layers and flattening down in the last two fully connected layers:

self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1)
self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
self.conv3 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1)
self.fc1 = nn.Linear(128 * 64 * 64, 128)
self.fc2 = nn.Linear(128, 2)

For Stable Diffusion we used InvokeAI framework to automate training data generation as it's one of the most intuitive and simple frameworks for SD, allowing our designers to help the developer team directly with data generation (something that would never be possible if the framework was code-heavy). To gather bonus information, we included Google's Tesseract OCR library to also read the text from the images.

Not to be forgotten is our front-end developer who takes all the processed data and presents it in our Dash Enterprise based web-dashboard. There we can preview all the detections, OCR results, available files, sample augmentations, and it's also what makes our data-science heavy project presentable for businesses and less tech-savvy people.

Challenges we ran into

Learning how to use new tools effectively in a short amount of time is a very common issue that we ran into in this Hackathon as well.

A challenge that was very specific to this challenge is the quantity and quality of provided and generated data, leading to training times of over 4 hours. In a competition where time is the most important factor, being able to see the results of trained models maximum after every 4 hours leaves very little room for error. Having our last training finish 3 minutes before the submission deadline, we believe we handled the challenge exceptionally well and are happy with the results.

Final model being trained

Accomplishments that we're proud of

Being the main focus of the challenge, we are very proud of being able to identify all types of defects (that were provided in the dataset). While the validation loss is insanely promising, we're getting access to the validation dataset only after the challenge ends, as such we can't rest on our laurels and need to still prepare for the worst. Having trained multiple models and testing them on both originally provided and artificially generated images, we however believe that we will be able to satisfy the <1% slip through limit for the errors that was set to us by our sponsors.

Things we can detect

The frontEnd was a success as well - having a proper designer on the team, the design and dashboard ideas were basically limitless. Sadly, having a competent designer, we quickly realized we were heavily limited by our knowledge and what design ideas we could implement in the first place. Using the Dash library still got us quite far quite relatively quickly. The heavily nested structures do unfortunately make the Python code become hard to read, though. Originally we wanted to make use of the Streamlit library, which, albeit being a lot easier to use (and us having more experience with it), could just not keep up with our intended design goals.

Last but not least, getting the text recognition to work and being able to extract the different parts of the text is also something we're very happy to have been able to add. It took us quite some tinkering, and being able to implement it nonetheless goes to show we had some time to spare while working on the project.

What we learned

The software we used so solve this challenge was new for some of us. Even though we were familiar with the different methods of image labeling, image augmentation and text recognition, not all of us knew the specific Python packages like Label Studio, Tesseract, etc. that we ended up using. Having learned how to work with them was a huge takeaway.

What's next for Defectors

By identifying defects early in the production process, companies can avoid wasting resources on products that would ultimately be rejected. This reduces the amount of raw materials consumed, along with the energy used in production, both of which have associated carbon emissions. As such, having some members of the team already working in the anomaly detection and computer vision industry, it's safe to say the project will remain a valuable tool for us in the future for similar tasks and with a little bit of luck, we hope the project to become a potential gateway to a professional solution specialized for our sponsors!