Facial features, illnesses and computer vision

Article authored by Limor Wainstein

Listen also to Federico Sukno in Spanish National Radio talking on this topic (in spanish)

In this article, we will review work by Federico Sukno and others in the context of the correlation between specific facial features and various illnesses. We will review a few examples for such correlations, show Face3D project's model for detailed analysis of facial morphology, and discuss their proposed measurement methods and their efficiency. Finally, we will outline the potential social impact of identifying ailments through facial features.

Research correlating facial features with various disorders and ailments is not new. As early as 1997, facial diagnostic signatures were linked to speech delay in kindergarten children (approximately 7% had language-specific impairment; cited in Baynam et al., 2017). An article published on in 2007 discussed research conducted at Harvard's McLean Hospital, which found a genetic link between subtle facial abnormalities and specific abnormalities in brain structure. The research showed that genes associated with risk for schizophrenia are carried through supposedly healthy family members: It was found that both the schizophrenics and some of their healthy relatives displayed similar facial abnormalities.

In the article co-authored by Sukno, "The definitions of three-dimensional landmarks on the human face: an interdisciplinary view" (Katina et al., 2015), the researchers highlight findings that demonstrate the association between disorders that take place during early brain development and facial dysmorphology. For example, a characteristic facial topography is associated with schizophrenia (Hennessy et al., 2007, in Katina et al., 2015), and facial dysmorphology is associated with 22q11.2 deletion syndrome, a velocardiofacial syndrome, associated with high risk for psychosis (Prasad et al. 2015, in Katina et al., 2015).

Another study by Hennessy et al. also found a connection between significant facial dysmorphology in bipolar patients (such as common facial widening, increased width of nose, and more), some of which were similar to dysmorphologies found in schizophrenia patients (2010).

An article by Baynam et al., published in 2017, also highlights several studies linking between facial features and illnesses. For example, in 2016, the Western Australian Register of Developmental Anomalies found a correlation between Congenital anomalies (such as Noonan syndrome) and facial dysmorphology (cited in in Baynam et al., 2017). Another example is hundreds of “dysmorphic syndromes” or “developmental disorders” which were found to be characterized by facies (Fryer, 1991 in Baynam et al., 2017). Last, numerous rare disorders (such as Cornelia de Lange syndrome) associated with speech delay were also related to specific facial phenotypes (Adam, MP et al. in Baynam et al., 2017).

One of the basic explanations for the connections found between facial features or patterns and various disorders relate to the development of the brain. Because the anterior brain and the face are very closely connected in the early embryological stages of human development, it is thought that certain disorders in the brain are somewhat mirrored in the face. For example, one finding concludes that the fronto-nasal prominence has a characteristic topography associated with schizophrenia. This area of the face has the closest embryological relationship with the anterior brain, therefore pointing to possible anomalies in early development (Hennessy et al., 2007).

While some of the facial features linked to illnesses are more distinctive, others are too subtle for the human eye to detect. This raises the need for highly-accurate, automatic identification of facial abnormalities. The article "The definitions of three-dimensional landmarks on the human face: an interdisciplinary view" discusses a three-dimensional approach for analyzing anatomical shapes, specifically the human face. This approach is considered more efficient than a two-dimensional approach, where information is reduced to particular distances and angles between selected landmarks (Katina et al., 2015). Baynam's article supports this approach, and emphasizes the need to use precision (deep) phenotyping-specifically 3-dimensional facial analysis (3DFA)-to go beyond the information usually included in medical charts (Baynam et al., 2017).

In another study by Hennessy et al., they show how Three-dimensional laser surface imaging and geometric morphometrics resolve frontonasal dysmorphology in schizophrenia.

Sukno, Waddington, and Whelan also propose a practical model for automatically identifying facial landmarks. In "3-D Facial Landmark Localization With Asymmetry Patterns and Shape Regression from Incomplete Local Features", they propose and test their model, which proves both more efficient and more accurate than existing methods. Their model is unique in that it is able to deal with "missing points", meaning the calculations can be used to fill in incomplete information in scans, "making it possible to limit the number of candidates that need to be retained, drastically reducing the number of combinations to be tested with respect to the alternative of trying to always detect the complete set of landmarks" (2015).

In addition to proposing a model for automatic detection of facial asymmetries, in their article "On the Quantitative Analysis of Craniofacial Asymmetry in 3D", Sukno and fellow researchers look at different methods used to detect asymmetry patterns. They analyze 100 high-quality laser scans and compare their "known" asymmetries to asymmetries detected by a set of automatic algorithms. They challenge widely-used measurement methods by testing the various algorithms (landmark-, midline- and surface-based approaches), and demonstrating how a hybrid approach, which combines surface and midline points, proves most effective and accurate (Sukno et al., 2015).

As discussed in this article, geometrical analysis of facial landmarks supported by the right algorithms and 3D technologies is efficient in understanding various illnesses, namely linking specific diseases and types of dysmorphologies. Such analysis can potentially be used for early diagnosis and treatment of critical illnesses. Performing such calculations on large data sets may assist with identifying individuals with risk of ailment across large populations.


F.M. Sukno, J.L. Waddington and P.F. Whelan. 3D Facial Landmark Localization with Asymmetry Patterns and Shape Regression from Incomplete Local FeaturesIEEE Transactions on Cybernetics45(9): 1717–1730, 2015. 

F.M. Sukno, M.A. Rojas, J.L. Waddington and P.F. Whelan. On the quantitative analysis of craniofacial asymmetry in 3DIn Proc. 11th IEEE International Conference on Face and Gesture Recognition, Ljubljana, Slovenia, 2015.

G. Baynam, A. Bauskis, N. Pachter, L. Schofield, H. Verhoef, R. L. Palmer, S. Kung, P. Helmholz, M. Ridout, C. E. Walker, A. Hawkins, J. Goldblatt, T. S. Weeramanthri, H. J. S. Dawkins and C. M. Molster. 3-Dimensional Facial Analysis—Facing Precision Public Health. Frontiers in Public Health. 5. Accessed February 5, 2018. 2017.

Goldberg, Carey. "Could mental illness be written in a face?". The New York Times. Accessed February 5, 2018.

R. J. Hennessy, P. A. Baldwin, D, J. Browne, A. Kinsella and J. L. Waddington (2007). Three-dimensional laser surface imaging and geometric morphometrics resolve frontonasal dysmorphology in schizophrenia. Biological psychiatry61(10), 1187-1194. 

R. J. Hennessy, P. A. Baldwin, D, J. Browne, A. Kinsella and J. L. Waddington (2010). Frontonasal dysmorphology in bipolar disorder by 3D laser surface imaging and geometric morphometrics: comparisons with schizophrenia. Schizophrenia research122(1), 63-71.

S. Katina, K. McNeil, A. Ayoub, B. Guilfoyle, B. Khambay, P. Siebert, F.M. Sukno, M. Rojas, L. Vittert, J. Waddington, P.F. Whelan and A.W. Bowman. The definitions of three-dimensional landmarks on the human face: an interdisciplinary viewJournal of Anatomy, 228(3): 355–365, 2016.


The author:

Limor Wainstein