In paper: MULTIMODAL TRANSFORMER FUSION FOR CONTINUOUS EMOTION RECOGNITION
- Feature-level fusion 2strategy
-
Traditional feature level fusion directly feeds the concatenated features into a classifier or uses shallow-layered fusion models [5], but it has the difficulty to learn mutual relationships among different modalities
-
Another alternative strategy of feature level fusion is multimodal
representation learning
The main approach is to learn joint representations from shared hidden layer connected with multiple modalities inputs
The models are usually based on deep learning frameworks, like deep autoencoder and DNN [6]. Kim et al. [7] proposed four Deep Belief Networks (DBNs) architectures to capture complex non-linear multimodal feature correlations for emotion recognition
- Model-level fusion
model level fusion learns multimodal interactions inside the models and makes better advantages of deep neural networks
The attention mechanisms were proposed to learn the alignment between audio-visual [10] and audio-text streams.
proposed temporal fusion model to dynamically pay attention to relevant modality features through time, which made the improvements over traditional fusion strategies
