LaserVision

• 

  App Logo

• 

  Scanning an object (coffee mug) using LiDar

• 

  Rendering the DepthCloud of the object and its environment

• 

  ML running an object detection algorithm to separate the subject from the background

• 

  Final measurements captured by the ML model

Inspiration

Taking dynamic measurements of 3d objects is an incredibly tedious process which is either achieved through manual measurements (a tailor using measuring tape) or a large laser-based apparatus (mapping buildings or large environments for construction). LaserVision is designed to collect 3D measurements in realtime without the need to physically measure objects with either a large, expensive, industrial-based machine or a simple measuring tape. Thanks to Apple's inclusion of LiDar in the new iPhones, all you need to collect accurate measurements of 3D objects is your iPhone.

What it does

LaserVision scans an object (animate like a person or inanimate like a car or coffee mug), creates a DepthCloud by combining a pointcloud and depth map, and feeds the data into 1 of 2 different CoreML models: Fixed or Adaptive.

Fixed ML Model

• The Fixed ML model is excellent for capturing measurements of objects that don't change in size. The perfect use case for this model is in quality assurance in industrial manufacturing. Simply feed the correct measurements for an object and LaserVision will scan every object as it moves through the assembly line and ensure it meets the required specifications up to 10mm in accuracy.

Adaptive ML Model

• The adaptive ML model is perfect for capturing measurements of objects that change. A great use for this model is in custom tailored clothing. Customers can scan themselves and supply their measurements to any retailer who would then recommend clothing in their catalog that best fits the customer's body type. No need for shoppers to needlessy buy and return clothing that doesn't fit them.

Previous forms of digital measurements are incredibly tedious because they require the user to manually select points of the object they'd like to measure and are often limited to height and width measurements. Our innovative implementation of machine learning models eliminated the need for any user input after scanning. Simply define what you'd like to be measured and our models will capture them. This also allows us to capture volume, density, and many more metrics in addition to height and width. Using ML also allows us to capture measurements of obscure and non-static objects (curved surfaces like the human body).

How we built it

The frontend user interface is built in Swift and Xcode in order to leverage Apple's LiDar sensor. The ML is written in Python and OpenCV.

Challenges we ran into

LiDar data is extremely dense and we ran into many issues constructing a scene view of the object in its environment with the LiDar scan.

Accomplishments that we're proud of

We're extremely proud that we were able to construct the DepthCloud which we considered to be the most difficult part of the workflow.

What we learned

We learned a lot about processing different 3D object filetypes and rendering graphics using Apple's Metal kit.

What's next for LaserVision

Future improvements for LaserVision include writing the ML in PyTorch instead of OpenCV and translating the code into Apple's CoreML API.

Built With

• blender
• opencv
• python
• swift
• xcode