The NVISO Neuro SDK provides facial action unit recognition capabilities for researchers, developers, and manufacturers of human machine interfaces (HMI). The SDK can automatically analyze facial movements by coding them into the Facial Action Units (FACS) taxonomy as a part of a real-time system. The SDK uses selected components of computer vision, machine learning and state-of-the-art graph based algorithms that work on an embedded system or can run on a standard PC.

Facial action units (FAUs) are the basic building blocks of facial expressions. FAUs include eyebrow raiser, eyebrow lowerer, eye opener, lip pressor, lip corner depressor etc. Each FAU is associated with a specific muscle in the face. FAUs are essential for our ability to communicate non-verbally. For example, if you want to ask someone what they think about something you can raise your eyebrows and lower your eyelids and that tells them exactly what you mean without having to say anything at all.

Facial action units are key indicators of emotions that play an important role in nonverbal communication. They can detect drowsiness and fatigue in drivers (and other people), which is critical for safety applications such as automotive, healthcare and security industries. Another use for this technology is for digital avatars in video games and virtual reality applications (VR). These characters need to look as realistic as possible so that players feel like they're interacting with another human.

Facial Action Unit Detection Technology

Science Based - Facial Action Coding System (FACS)

Facial Action Coding System (FACS) is a detailed way of describing the facial expressions. It was developed by Paul Ekman and Wallace Friesen in 1978. FACS is used to describe the facial expressions, which are very important in emotion recognition.

Facial action units (FAU) are the basic building blocks of facial expressions. They are readily identifiable movements of the muscles in the face and they can be used to express a range of emotions. For example, if you smile or frown, you are using the muscles around your mouth to achieve that effect.

FACS provides a list of 68 different FAUs, which NVISO groups these into three categories: 

1) Upper face movements : Eye and brow

2) Lower face movements : Lip, mouth, nose, chin, and jaw

3) Asymmetrical movements : Left and right

Improved Accuracy - Graph Convolutional Neural Networks

Graph convolutional neural networks (GCNs), a type of neural network that uses graphs as its representation rather than the standard feedforward networks used in most previous work. While it was previously thought that these models were unsuitable for the task of facial action unit detection (for example, due to their large number of parameters), work by researchers at Google Brain has shown that GCNs can outperform traditional neural networks for this task.

NVISO's GCN based approach is now considered the state-of-the-art as it can be used to model the relationships between AUs by learning the shared spatial structure. This approach is able to learn a low-dimensional embedding for each AU that captures the spatial relationship between them. This greatly outperforms the previous approach of using only  convolutional neural networks (CNNs) to classify an image only which suffered from high false negative rates due to the complexity of AU recognition.

Reduced Bias - Large Scale Emotion Databases

In recent years, there has been a surge of interest in the use of deep learning methods to classify facial expressions. While these techniques have shown great promise, they are often limited by their inability to adapt to new subjects and contexts. This problem is particularly pronounced when the classifier is applied to new age groups or ethnicities. The problem is further complicated by the fact that emotional expressions can be similar across individuals even though they have different emotions. For example, both happy and sad smiles may involve contraction of the zygomaticus muscles and raising of the corners of the mouth.

NVISO overcomes this these problems, by using very large scale scale databases of emotional expressions that have been annotated with a large number of facial action unit (AU) labels. These databases allow us to train models that are robust across different ages, genders, and ethnicities, while still maintaining high accuracy on a single individual. 

Facial Action Unit Recognition Applications

Digital Avatars

Facial action units (FAU) are the basic building blocks of facial expressions. They are readily identifiable movements of the muscles in the face and they can be used to express a range of emotions. For example, if you smile or frown, you are using the muscles around your mouth to achieve that effect.

Digital avatars are computer-generated images or animations that represent a real person, such as an actor or celebrity. Facial expressions are important for creating realistic digital avatars because these expressions help the viewer connect with the avatar and understand what it is feeling. NVISO FAU detection software can be used to detect the facial expressions of a user and then animate their face accordingly using artificial intelligence algorithms. This allows digital avatars to be more lifelike than ever before!

Pain Assessment

Facial expressions can also be used to detect pain in patients who cannot communicate verbally due to illness or injury. This type of technology is especially useful when it comes to elderly patients who may have cognitive problems preventing them from telling doctors how they feel physically or emotionally.

Pain is one of the most common symptoms experienced by elderly people, and it can be difficult for doctors and nurses to accurately assess this type of pain. This is because they may not be able to get close enough to their patients, or they may not have adequate training in how to interpret facial expressions. Facial action unit detection software has been found to be able to make this process easier, by identifying specific facial movements associated with pain and providing an output which shows what kind of pain is being experienced.

Drowsiness and Fatigue

The need for automated facial action unit detection software is growing because human drivers are getting more and more drowsy and fatigued. According to the U.S. Department of Transportation, there were 5,987 fatal accidents in 2018 that involved drowsiness or fatigue as a factor in the accident.

The use of facial action unit detection has many advantages over other approaches:

1) Facial muscle movements are involuntary and cannot be controlled consciously by someone trying to hide his/her emotions;

2) The number of facial muscles involved in expressing emotions is relatively small compared to the movement of body parts;

3) The face is visible from many angles;


NVISO Neuro Models™ are purpose built for a new class of ultra-efficient machine learning processors designed for ultra-low power edge devices. Supporting a wide range of heterogenous computing platforms ranging from CPU, GPU, DSP, NPU, and neuromorphic computing they reduce the high barriers-to-entry into the AI space through cost-effective standardized AI Apps which work optimally at the extreme edge (low power, on-device, without requiring an internet connection). NVISO uses low and mixed precision activations and weights data types (1 to 8-bit) combined with state-of-the-art unstructured sparsity to reduce memory bandwidth and power consumption. Proprietary compact network architectures can be fully sequential suitable for ultra-low power mixed signal inference engines and fully interoperable with both GPUs and neuromorphic processors


NVISO Neuro Models™ use proprietary datasets and modern machine learning to learn from billions of examples resulting in an extraordinary capacity to learn highly complex behaviors and thousands of categories. Thanks to high quality datasets and low-cost access to powerful computing resources, NVISO can train powerful and highly accurate deep learning models.


NVISO Neuro Models™ store their knowledge in a single network, making them easy to deploy in any environment and can adapt to the available hardware resources. There is no need to store any additional data when new data is analysed. This means that NVISO Human Behaviour AI can run on inexpensive devices with no internet connectivity providing response times in milliseconds not seconds.


NVISO Neuro Models™ are scalable across heterogeneous AI hardware processors being interoperable and optimised for CPUs, GPUs, DSPs, NPUs, and the latest neuromorphic processors using in-memory computing, analog processing, and spiking neural networks. NVISO Neuro Models™ maximise hardware performance while providing seamless cross-platform support on any device.


Consumer Robots

Human–robot interaction plays a crucial role in the burgeoning market for intelligent personal-service and entertainment robots.

Automotive Interior Sensing

Next generation mobility requires AI, from self-driving cars to new ways to engage customers. Build and deploy robust AI-powered interior monitoring systems.

Gaming and Avatars

The gaming industry (computer, console or mobile) is about to make extensive use of the camera input to deliver entertainment value.


Get Our Whitepaper