Mesh-based 3D Reconstruction from Bi-planar Radiographs

Moritz Jokeit, Ji Hyun Kim, Jess Gerrit Snedeker, Mazda Farshad, Jonas Widmer

Keywords: 3D reconstruction, deep learning, mesh, medical imaging, X-ray, digitally reconstructed radiograph (DRR), style transfer, computed tomography (CT)

TL;DR: Concept for learning-based 3D reconstruction from bi-planar radiographs using anatomical mesh templates.

Abstract: Reconstruction of 3D surfaces from sparse 2D data is a challenging problem that attracted increasing attention also in the medical field where image acquisition is expensive and the patients often bear high radiation doses (CT, fluoroscopy). Further, advances in computer-guided surgical assistant systems and preoperative planning necessitate fast 3D reconstruction from scarce image data. Recent learning-based approaches showed notable success in reconstructing primitive objects leveraging abundant artificial data sets. However, quality 3D data in the clinical context is often scarce. This motivates the exploitation of domain knowledge in form of anatomical shape priors to simplify the reconstruction problem. Further, mesh-sensitive applications (e.g., finite element analysis of implant design) greatly benefit from pre-defined mesh topologies. Thus, we propose a concept for the implementation and training of a learning-based patient-specific 3D reconstruction from bi-planar radiographs based on altering anatomical template meshes.

Registration: I acknowledge that acceptance of this work at MIDL requires at least one of the authors to register and present the work during the conference.

Authorship: I confirm that I am the author of this work and that it has not been submitted to another publication before.

Paper Type: novel methodological ideas without extensive validation

Primary Subject Area: Application: Radiology

Secondary Subject Area: Image Acquisition and Reconstruction

Confidentiality And Author Instructions: I read the call for papers and author instructions. I acknowledge that exceeding the page limit and/or altering the latex template can result in desk rejection.

Code And Data:

https://github.com/MIRACLE-Center/CTPelvic1K (Data)
https://github.com/walsvid/Pixel2MeshPlusPlus (Encoder Network with Cross-view perceptual feature pooling)
https://github.com/jnypan/TMNet (Transformation network)
https://github.com/unibas-gravis/scalismo (Mesh registration)