Advecting 3D Meshes using Point Cloud Interpolations

Project poster

Deliverables:

Final Write-up: https://drive.google.com/file/d/1RFlkyAIwKTwiGOsdW0lqhuOlSagOygf1/view?usp=sharing

Presentation video: https://www.youtube.com/watch?v=sgjUwudivk8

Poster: https://drive.google.com/file/d/1ul0lGPf169hrRa8jqw1HYolM4eW8WF7W/view?usp=sharing

Github: https://github.com/ArmanMaesumi/CSCI2470Final

Project Check In #2: https://docs.google.com/document/d/1okfyGDuAcS5arpXj9dDYblSxYJLOQ5XLpiYRTmzQ_Vo

Introduction: I will investigate deep learning models that facilitate 3D shape synthesis and deformation. This is a central problem in computer graphics and vision, as many applications require the creation of realistic 3D models of real-world shapes. For example, in robotics, perception systems are trained using synthetic scenes, which are populated with 3D shapes, and in entertainment, 3D shapes are used to populated virtual scenes.

Related Work: My work will be related to two particular works: ShapeFlow and SP-GAN. The ShapeFlow paper introduced a learned deformation space, which generates a continuous vector field that advects mesh vertices. SP-GAN is a point cloud GAN that learns to synthesize new shapes by using a "sphere prior." This prior essentially maps a spherical point cloud into the dataset shapes, which allows the model to learn an unsupervised dense correspondence mapping between all generated shapes.

ShapeFlow paper: https://arxiv.org/pdf/2006.07982.pdf SP-GAN paper: https://arxiv.org/abs/2108.04476

Data: My datasets will be entirely based on ShapeNet, a large repository of 3D shapes. This is the standard for shape synthesis, and it won't require significant processing (apart from partitioning the shapes into training/test sets).

Methodology: I will first investigate the viability of using SP-GAN's output as the driver of a deformation. More concretely, I will deform a given mesh by using a point cloud interpolation sequence as an advection flow. After this, I will attempt to modify SP-GAN's model to better facilitate this task.

Metrics: Evaluating shape synthesis (and any generative model) results is difficult do to the qualitative nature of it. Therefore, I will use modern quantitative evaluation metrics such as Frechet Inception Distance (FID), as well as Minimum Matching Distance, and Coverage.

Ethics: It is difficult to imagine an unethical use case of shape synthesis generative models. In the case of image synthesis (such as human faces), it is easy to see such use cases such as forgery and misinformation. However, for 3D shapes these use cases are a bit far fetched.

Deep learning is the perfect tool for this job because the space of possible 3D shapes is massive, and creating a handwritten program that generates novel shapes is extremely difficult. For generative modeling, deep learning is practically the only option currently.