AI LIE DETECTION: FACIAL EXPRESSION ANALYSIS

Author

Ganesh N, Vishwasa, Varun Hanumradder, Vijay P Mirajkar


Abstract

 Deception detection in interview settings has historically faced severe limitations due to subjective human bias and low unas sisted accuracy (~54%). Automated non-invasive vision systems provide an objective alternative by analyzing subtle involuntary facial dynamics that escape conscious suppression. In this paper, we introduce an intelligent, automated AI lie detection platform focused on continuous facial expression analysis. The system is engineered using a client-server architecture: an interactive WebRTC front-end built with HTML5, CSS3, and JavaScript captures video frames and renders real-time 3D landmark meshes on an HTML canvas, while a Python backend powered by OpenCV and MediaPipe extracts dense 468-point facial geometry and 17 Facial Action Coding System (FACS) Action Units. Rather than treating deception as a simple binary snapshot, the framework evaluates four foundational facial behavioral dimensions: (1) Contextual Relevance (evaluating whether the observed emotional valence matches the situational context of the interview prompt); (2) Response Dynamics (quantifying expression onset latency, apex hold duration, and decay kinetics); (3) Bilateral Consistency (measuring hemifacial symmetry between left and right facial halves and temporal coherence across sequential responses); and (4) Potential Deception Indicators (detecting micro-expressions such as brow lowering AU04, lip corner depression AU15, dimpling AU14, rapid blink bursts AU45, lip compression AU24, and micro-saccadic eye gaze shifts). To ensure ethical reliability, a calibrated fail-safe abstention gate halts classification when face tracking is occluded or when innocent nervousness is detected. Evaluated on 2,420 video response instances, the proposed MediaPipe-CNN-Transformer pipeline achieved 91.2% accuracy, 91.6% Macro-F1, and 0.948 ROC-AUC, with Expected Calibration Error of 0.8%. Under 400 adversarial stress cases, the gate intercepted 100% of false-accusation traps. A user study with 18 professional interviewers demonstrated an accuracy boost from 54.2% to 84.8% (SUS 79.4), confirming its effectiveness as an assistive decision-support tool.



Keywords

AI Lie Detection, Facial Expression Analysis, MediaPipe Face Mesh, Computer Vision, Facial Action Coding System (FACS), Micro-Expressions, Expression Consistency, Response Latency, Python, Web Application, Confidence Calibration.



Full Text:

Download Paper PDF


References


  1. Ekman and W. V. Friesen, “Facial Action Coding System: A Technique for the Measurement of Facial Movement,” Consulting Psychologists Press, 1978.
  2. Ekman, “Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage,” W. W. Norton & Company, 2001.
  3. O. Meservy, M. L. Jensen, J. Kruse, J. K. Burgoon, J. F. Nunamaker, D. P. Twitchell, G. Tsechpenakis, and D. N. Metaxas, “Deception detection through automatic, unobtrusive analysis of nonverbal behaviour,” IEEE Intelligent Systems, vol. 20, no. 5, pp. 1–8, 2005.
  4. Salah, T. Khater, E. Almajali, W. Khan, and A. Hussain, “Deception detection deep learning comprehensive system utilizing explainable AI,” in Proc. 2023 IEEE DESE Conf., 2023, pp. 1–6.
  5. Abouelenien, V.Perez-Rosas, R. Mihalcea, and M. Burzo, “Detecting deceptive behavior via integration of discriminative features from multiple modalities,” IEEE Transactions on Information Forensics and Security, vol. 12, no. 5, pp. 1042–1055, 2017.
  6. F. Bond Jr. and B. M. DePaulo, “Accuracy of deception judgments,” Personality and Social Psychology Review, vol. 10, no. 3, pp. 214–234, 2006.
  7. M. DePaulo et al., “Cues to deception,” Psychological Bulletin, vol. 129, no. 1, pp. 74–118, 2003.
  8. Ekman, “Darwin, deception, and facial expression,” Annals of the New York Academy of Sciences, vol. 1000, no. 1, pp. 205–221, 2003.
  9. F. Cohn, Z. Ambadar, and P. Ekman, “Observer-based measurement of facial expression with the Facial Action Coding System,” in The Handbook of Emotion Elicitation and Assessment, Oxford University Press, 2007.
  10. Lugaresi et al., “MediaPipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019.
  11. Perez-Rosas et al., “Deception detection using real-life trial data,” in Proc. ACM ICMI, 2015, pp. 59–66.
  12. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Machine Learning (ICML), 2017, pp. 1321–1330.
  13. Gupta et al., “Bag-of-Lies: A multimodal dataset for deception detection,” in Proc. IEEE CVPRW, 2019, pp. 104–111.
  14. EuropeanParliament, “Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” Official Journal of the European Union, 2024.
  15. Brooke,“SUS: A 'quick and dirty' usability scale,” in Usability Evaluation in Industry, London: Taylor & Francis, 1996.
  16. G. Hart and L. E. Staveland, “Development of NASA-TLX,” Advances in Psychology, vol. 52, pp. 139–183, 1988.

Share your valuable work from Social Media Buttons