Automated Landmark Detection for AR-Based Craniofacial Surgical Assistance System

Sanghyun Byun, Muhammad Twaha Ibrahim, M. Gopi, Aditi Majumder, Ross Lohrasb Sayadi, Usama Hamdan, Raj M. Vyas

Abstract:

The shape of the face of cleft lip patients varies significantly from a regular face due to the unique form and differing levels of severity of their condition. The first step in cleft lip repair requires surgeons to mark anthropometric landmarks that are used as a guide to conduct surgical incisions. These landmarks are different from the ones that are deemed important in a regular face and cannot be detected by existing facial landmark detection frameworks.

We propose a AI/ML based assistive tool that can automatically mark the anthropometric landmarks for cleft repair on the image of the cleft lip patient. We use a novel method for training a convolutional neural network that detects the anthropometric landmarks for patients with cleft lip without requiring a large number of images for training. By utilizing image ROI (region of interest) warp and direct regression, the proposed approach is able to accurately detect landmarks despite variation in the appearance of the cleft. Further, we show the significant improvement ROI warp has on the prediction of anthropometric landmarks used for cleft surgeries. We collaborate closely with reputed craniofacial surgeons to build our training datasets and validate the accuracy of our automated markings.

This tool is anticipated to have a tremendous impact on building surgical capacity for cleft repair surgeries, which has a huge shortage, in particular in rural areas, especially in emerging global areas of South America, Africa, and India.