ExteriScan

A performant scanning software, made to detect surface anomalies on both micro scale (microscopic pictures) and macro scale (satellite pictures)

Comment

Inspiration

Noticing a microscopical fracture on a wing of a plane before it's too late, could mean the difference between someone's honeymoon and funeral. Whether today, or in the future years of technology - the stability of our enormous metal constructions, will remain as relevant as never before. We're a team of three students, who are interested in shaping a safer future for everyone, and plan on doing so in the most efficient way possible.

What does it do

The core of the project is data analysis. Multiple classification and object detection algorithms are mixed together to analyze visual data and provide insigts with minimal computing power. A picture of a solid, fed through our scanning software will be analyzed whether it contains any surface anomalies, and if any of them are present, the software will determine the approximate position and the count of all damage points.

Due to the sheer volume of data, generated by in-depth scans, we knew that the software had to be as efficient as possible - crunching through gigabytes of data in a matter of minutes, it would be able to deal with heavier datasets in a reasonable amount of time already. We have been able to achieve our own-set speed goal and proof test it by performaing crack detection with a live camera footage - the speed of image analysis can easily keep up with lower framerate video streams.

Speed however, while impressive on it's own, is not all that matters. Since the data that has to be analyzed could vary heavily, the software has to be able to adapt to different types of datasets. We believe that we have managed to achieve precisely that - going from micro to macro, analysing camera footage of roads, our software was easily able to identify road cracks and damaged pieces of asphalt without any problems, even though the dataset was vastly different.

The most important part, surpassing both speed and pertinence between datasets, is accuracy. We have expanded the provided data to multiple thousand pictures. We have also taken over 40 thousand pictures of cracked roads from the internet, and left the algorithm to improve for more than 30 hours throughout the whole hackathon. We're really happy with the quality that the software has achieved over the weekend, but we're even happier knowing the full potential of our algorithm, and the accuracy that it could achieve if trained without time constraints.

How we built it

Building such a software was a multi-step process. Just like all the data-driven projects, it started with the first and only possible stage - the data gathering and labeling. We felt that the data we recieved was insufficient and we were lacking pictures of objects without cracks. The work began by artificially increasing our sample size, creating pictures without any cracks for testing purposes, properly labeling and grouping similar data, then once again splitting everything up into multiple training and testing sets. This early time investment allowed us to begin as soon as possible with the second stage of development - actual training of our classifier. Using Keras, the first milestone would be identifying whether a crack is visible in an image. Parallel to our 1st classifier doing training, the 2nd approach was starting to manifest - actively looking for, and pin-pointing the position of any visible cracks using Tensorflow. To save time and increase quality as early as possible, we started training our model on top of a pre-trained Single Shot MultiBox Detector: MobileNet COCO - well known model in the machine learning industry, for its speed and object location times under 30ms. Just as the MobileNet COCO model was starting to show results, our 1st approach finished its planned training and beat all of our expectations with insanely accurate predictions and high overall confidence levels - while happy with the success, we worried about overfitting to this specific sample and started our hunt for more and varied data, thus arriving at the third stage of our development - data generalization. Satellite pictures of rivers, camera pictures of roads, telescope pictures of moon surface - everything even remotely similar to cracked solids was of interest to us - we knew that achieving high accuracy on such a variety of datasets would allow massive freedom when deciding what initial data should look like - precisely what we wanted to achieve.

One dataset after another, our model was starting to get astonishing results on multiple different "crack-classification" problems, while maintaining high accuracy on the initial data. By this time, our crack locating software became really good at pin-pointing cracks not only in images, but in real life as well. We knew we had the time to take the final step of our development - the merge. Both classification and location models were working flawlessly and instead of dropping one of them, we decided to join them into what later became the final version of ExteriScan - the communication between classifier and locator model allowed to wipe out the majority of our remaining problems. In images where no crack can be found to begin with, the locator won't even attempt to find something, not only saving some precious time, but also avoiding potential false-positives. In images where the classifier is extremely confident that a crack exists, the locator will try extra hard to find something even remotely representing a crack, allowing for some insane precision when multiple cracks can be seen at once in one picture. This, combined with a quick live camera feed, is more than enough to display the potential of ExteriScan.

Challenges we ran into

Set high standards and high expectations from our own software definitely created some extra challenges for us. Building on top of an older existing model, while interacting with our own brand newly written classifier, meant nothing else, but compatibility chaos. Different versions of Python, different versions of Tensorflow and different versions of Keras all working together meant rewriting thousands of lines of Tensorflow source code just to get the code to run. Even then, every new line of code, was a risk of hitting a new dependency that wouldn't be compatible with the rest of the codebase - nonetheless, with clever workarounds and combined experience, we were able to make all parts of the code work together for the greater good.

Accomplishments that we're proud of

The biggest challenges are also the most rewarding ones, once you manage to tackle the problem. As mentioned previously, getting all the small pieces to work together was something, that we can all be proud of as a team. But just as we're strong as a team, each of us had his own victories through the hackathon: Valentin, who was working on the classifier, was able to crunch mind boggling amounts of data and see the fruits of his labour immediately after. Mantas, while spending most of his time on object locating, was able to fix hundreds if not thousands of errors by completely rewriting a Tensorflow module for a Python version on which the module isn't even supported anymore. As to Fabian, him being as efficient as ever, goes all the flowers and credit for training our model to the point where it is now. Every single member of the team has had his small victories throughout the hackathon, but it's safe to say, that the final result of the project is something we're all jointly proud of together.

What we learned

The whole challenge was definitely a learning experience - a machine learning experience to be exact. Most of the time was spent on working with ML libraries, classifiers and models, and no matter the background or previous expertise in this field, every one of us learned something new. From training models, to sorting data, from optimal ways of labelling cracks to forcing AI to label the data itself. Everyone picked up a new skill or two, and we're sure these skills will come in handy soon enough.

What's next for ExteriScan

As proud as we are of our software, we can safely say it's far from perfect - the accuracy can always be increased, the code can always be better documented and most importantly, ExteriScan has so much to prove by being applied to specific tasks in the industry. We don't plan on leaving this project alone any time soon, and we hope to find people who would be interested on taking the software on for a test drive and throwing a task or two at it, to see how it performs.

This is not the end; for ExteriScan, it's only the beginning.

Built With

Share this project:

Updates