Informatica

Traditional approaches primarily rely on conventional machine learning algorithms. For example, Moraes et al. (2013) conducted a comparison using SVM and ANN for document-level sentiment classification tasks. They used the bag-of-words model for feature selection and weighting, and discussed the classification accuracy of both methods across different contexts within a standard evaluation setting. Shunxiang et al. (2023) proposed a model based on sentiment intensity and PU learning. They divided reviews into different subsets based on sentiment intensity and used SCAR and Spy techniques to extract initial positive and negative samples. A semi-supervised PU learning detector was then constructed to iteratively detect fake reviews from incoming streaming data. Some studies focus on improving dictionaries and statistical methods to better understand text for sentiment analysis. For instance, Kang et al. (2012) addressed the issue of inadequate sentiment lexicons in restaurant reviews by proposing a new sentiment lexicon (senti-lexicon) and improving the Naive Bayes algorithm. They combined unigrams and bigrams as features, narrowing the accuracy gap between positive and negative sentiment classification, and outperforming SVM and the original Naive Bayes in recall and precision. Wang et al. (2023b) proposed an automatic method for generating fine-grained sentiment lexicons by constructing a seed lexicon through sentiment-sentiment transfer, extending the lexicon using graph propagation, and performing multi-information fusion based on neural networks. The resulting FGSL demonstrated strong performance across a range of sentiment analysis tasks. Pashchenko et al. (2022), using the NRC sentiment lexicon and unsupervised learning, explored the relationship between sentiment and star ratings in hotel and tourism reviews, finding that customer feedback on different sentiment aspects varied with ratings. However, some studies suggest that relying on a single method may limit the model’s expressive power; therefore, a combination of methods may be more effective (Rodríguez-Ibáñez et al., 2023). For example, Bibi et al. (2022) presented a framework for unsupervised learning that utilizes concepts and hierarchical clustering to analyse sentiment on Twitter. The results showed that unsupervised learning was comparable to supervised learning techniques. Wang et al. (2019) explored Twitter sentiment analysis by combining textual information with sentiment diffusion patterns. They proposed the SentiDiff iterative algorithm, which considered the interaction between both elements to enhance sentiment analysis performance, and used sentiment diffusion patterns for the first time to improve Twitter sentiment analysis. While these methods have made significant contributions to sentiment analysis tasks, the powerful expressive capabilities of deep learning have led many studies to adopt deep learning to address issues in this field (Tai et al., 2015; Tang et al., 2015; Dai et al., 2021). For example, Chen (2015) proposed two convolutional neural network models, Parallel CNN and Deep CNN, based on word2vec word embeddings, to identify question-answer relationship candidates in QA systems. They used convolutional layers to extract semantic features, and pooling and fully connected layers to summarize them. Usama et al. (2020) introduced a model based on RNN and CNN with an attention mechanism. They first used CNN to extract sentence features, then applied an attention mechanism to compute contextual weights for these features, and finally input the features and weights into an RNN for sentiment analysis. Tian et al. (2020) proposed the SKEP model, which incorporates sentiment masking and three sentiment prediction targets to embed sentiment information into pre-trained representations. The model significantly outperformed baseline models, achieving new best results on most test sets. The widespread use of attention mechanisms has provided new perspectives for sentiment analysis of text. Several studies have combined attention mechanisms to understand sentiment in text. For example, Zhai et al. (2020) proposed the Multi-AFM model, which generates contextual representations using gating units, this model was applied to sentiment analysis of educational big data, improving classification performance. Parveen et al. (2023) proposed the GARN framework, which integrates RNN and attention mechanisms to extract sentiment features, perform feature selection, and conduct sentiment classification on Twitter. More recently, Zhang et al. (2025b) proposed a textual graph representation with syntactic weighting that combines word-position graph structure, syntactic weights, attention, and external knowledge to model implicit sentiment. Beyond sentiment analysis, Zhao et al. (2025b) demonstrated that hyperbolic graph attention can integrate hierarchical semantic representations with long-context information for technical keyphrase extraction, illustrating the broader value of graph-based long-context modelling for text understanding.

Despite the significant achievements of deep learning in text sentiment analysis, the widespread use of social media means that sentiment expression is often not limited to text alone. Therefore, extending the powerful capabilities of deep learning to multimodal sentiment analysis by integrating information from different modalities to more accurately identify and understand sentiment is the primary focus of this study.

Multimodal sentiment analysis poses a greater challenge than traditional text-based sentiment analysis, as it requires integrating information from multiple modalities to uncover underlying sentiment cues. Traditional methods predominantly rely on attention mechanisms to enable models to focus on key information across different modalities, thereby enhancing sentiment analysis performance (Truong and Lauw, 2019; Xue et al., 2022). For instance, Zhang et al. (2023) utilized the Transformer model to extract both image and text features, incorporating LSTM and attention to better emphasize important information. Li et al. (2023), Liang et al. (2025) introduced tensors to extract and fuse features from different sources, such as text, images, and audio, in order to capture the complex data patterns in multimodal sentiment analysis. Additionally, an improved bilinear fusion method was employed, in which the model dynamically adjusts the fusion weights in real time based on the features of both images and text to achieve weighted fusion. Hu et al. (2024) proposed a three-channel multimodal fusion framework where the second channel removes redundant information from auxiliary modalities, while the third channel enhances the significance of the primary modality, and the integration of features from the three channels is managed by a multi-channel information fusion gate. Wang et al. (2025b) introduced an adaptive attention module that dynamically adjusts the contributions of text and image features by leveraging cross-modal attention to extract shared representations, thereby improving the performance of both modalities. Additionally, sentiment information is used to guide the extraction of features relevant to sentiment. Several studies have focused on effectively fusing features from different modalities to leverage their complementary information (Zhao et al., 2019; Wang et al., 2022). For example, Mai et al. (2022) employed intra-modal, cross-modal, and semi-contrastive learning concurrently, thoroughly exploring cross-modal interactions and learning the relationships between samples and categories, thereby reducing the modality gap. Moreover, they introduced refinement terms and modality intervals to learn single-modal pairs better. An and Zainon (2023) enhanced sentiment analysis accuracy by integrating semantic information and image colour cues from image-text pairs. The model included a feature extraction module to extract semantic and colour features, a feature interaction module to facilitate information exchange between features via cross-attention mechanisms, and a label prediction module to consolidate features and strengthen multimodal sentiment analysis. Cheng et al. (2023) proposed an Attention Temporal Convolution Network (ATCN) to enhance the representation of temporal features in a single modality, alongside a Multi-layer Feature Fusion (MFF) model to improve multimodal fusion performance, fused features of different levels based on feature correlation, and used cross-modal multi-head attention to explore relationships among low-level features.

In addition, semantic correlations and sentiment consistency between different modalities are crucial for sentiment analysis, leading some studies to incorporate contrastive learning to enhance model performance. For instance, Zhang et al. (2024) used a translation network encoder to capture shared concepts between vision and text, addressing the issue of missing modalities. Based on this framework, a single-modal weight adaptation strategy was introduced to leverage meta-learning from a few labelled samples to learn single-modal weights. Wang et al. (2025a) explores modality correlations by a multi-layer cross-modal interaction module. Subsequently, feature fusion was performed using a multimodal fusion module that incorporated contrastive learning to uncover key sentiment features across different categories and samples.

With the growing use of cutting-edge methods, including generative and graph models, more research is focusing on these areas. For example, Huan et al. (2023) proposed a UniMF model, which includes a translation module and a prediction module. The translation module generates missing modalities using existing modal information through Multimodal Generation Masking (MGM) and a Multimodal Generation Transformer (MGT). The prediction module integrates multimodal information via an attention mechanism to generate predictions, using a Multimodal Understanding Transformer (MUT) that incorporates Multimodal Understanding Masks (MUM) and Multimodal Sequence (MMSeq) representations for unified multimodal understanding. Zhao et al. (2025a) generated virtual modalities to replace missing ones and aligned the semantic space of virtual and missing modalities through contrastive loss. Wang et al. (2023a) addressed sentiment information in multimodal data from both global and local fine-grained perspectives. The global perspective obtained overall sentiment representations from text-image caption pairs, while the local perspective explored fine-grained information in text and images via two graph structures. Ultimately, combining global and local sentiment information provided aspect-level sentiment polarity. Wang et al. (2025d) captures sentiment correlations within and across modalities through a graph-based architecture. More recently, Zhang et al. (2025a) proposed a Multimodal Semantic Fusion Network that uses gated attention for cross-modal alignment, graph convolutional networks to model interactions among aligned features, and the integration of explicit and implicit sentiment semantics.

Although these methods have made progress in multimodal sentiment analysis tasks, they have not fully exploited the global structural information in images and have overlooked noise interference during cross-modal interactions. Given that multimodal data often contains substantial redundancy and noise, these interferences can disrupt the model’s training and degrade performance. Thus, the effective differentiation of image processing and noise mitigation between image and text data to improve the accuracy of multimodal sentiment analysis is the primary focus of this study.

Multimodal sentiment analysis primarily aims to detect the sentiment conveyed across different modalities. The multimodal data in this study consists of image and text. Given a text-image pair $({X_{t}^{j}},{X_{i}^{j}})$ along with the corresponding task label ${y^{j}}$, the dataset is defined as follows: where ${X_{t}^{j}}$ represents the j-th text sample in the dataset, ${X_{i}^{j}}$ is the corresponding image associated with ${X_{t}^{j}}$, and ${y^{j}}$ denotes the task-specific label. Specifically, for our core task of multimodal sentiment analysis, ${y^{j}}$ is a binary label for the HFM dataset, with 1 indicating positive sentiment and 0 indicating negative sentiment, whereas it is a 3-dimensional one-hot vector for the MVSA datasets, indicating negative, neutral, and positive sentiment, respectively. To further improve the generalization ability of our proposed method, we also conduct experiments on multimodal sarcasm-detection datasets, where the labels are split into sarcasm and non-sarcasm. The mapping between these sarcasm detection labels and sentiment labels follows the approach of Wei et al. (2023): sarcasm labels are mapped to negative sentiment, and non-sarcasm labels are mapped to positive sentiment. N refers to the total number of samples in the dataset, and the text-image pair $({X_{t}^{j}},{X_{i}^{j}})$ is fed into the model F to predict the task-specific label, which realizes the corresponding multimodal classification goal: This formulation describes the prediction process of the model for each text-image pair in the dataset.

To accurately capture sentiment-related features within an image, our method employs the Discrete Wavelet Transform (DWT) to decompose the image data into specialized frequency domains, thereby preserving both global structure and fine-grained textures (Zhao et al., 2021). Specifically, we utilize time-frequency analysis techniques—including wavelets (Mallat, 1989; Daubechies and Heil, 1992)—to minimize noise while isolating distinct visual components. Through a series of low-pass and high-pass filtering operations, our model decomposes the input into four sub-bands: $LL$ (Low-Low), $LH$ (Low-High), $HL$ (High-Low), and $HH$ (High-High). In our technical framework, the $LL$ sub-band constitutes the Low-Frequency (LF) domain, which retains the overall contour and large-scale global information necessary for understanding the primary sentiment tendency (Huang et al., 2021). Simultaneously, the combination of $LH$, $HL$, and $HH$ sub-bands forms the High-Frequency (HF) domain, which focuses on extracting horizontal, vertical, and diagonal edge details. By decoupling the image into these domains, the model can effectively capture subtle visual cues—such as textures and corners—that are often overlooked in spatial analysis but are vital for precise sentiment judgment. The formal decomposition process, illustrated in Fig. 2, is expressed as follows: where ${\mathbf{Y}_{b,{c^{\prime }},l}}\in {\mathbb{R}^{\ast }}$ is a three-dimensional tensor, where b is the batch index, c is the number of patches in the image, ${c^{\prime }}$ denotes the adjusted number of image patches after decomposition, and l is the embedding hidden layer dimension. The wavelet transform decomposes the input image into low-frequency feature ${\mathbf{Y}_{b,{c^{\prime }},l}^{\varsigma }}$ and high-frequency feature ${\mathbf{Y}_{b,{c^{\prime }},l}^{h}}$.

In this module, we implement a Global Semantic Sentiment Component (GSSC) and a High-Frequency Filtering Component (HFFC). After processing the GSSC and HFFC components separately, we use the inverse wavelet transform to synthesize the processed LF and HF parts back into the original feature space. The processed LF component, denoted as ${\mathbf{Y}_{b,{c^{\prime }},l}^{\varsigma ,\mathrm{\Im }}}$ with the superscript ℑ indicating the output of GSSC, and the processed HF component, denoted as ${\mathbf{Y}_{b,{c^{\prime }},l}^{h,\ell }}$ with the superscript ℓ indicating the output of HFFC, together reconstruct the final image. This process can be expressed by the following formula: which defines the corresponding representation.

In multimodal tasks, text data often contains rich information, and appropriate processing of text is essential for these tasks (Wang et al., 2024; Sun et al., 2024). However, text often contains various types of noise that manifest at different granularities, negatively affecting the accuracy of sentiment analysis. To effectively address this issue, we designed a Multi-Grained Semantic Purification Module. This module aims to refine text data by filtering out noise at different granularities, thereby extracting more precise sentiment information. To further enhance temporal feature representation, we first apply a causal convolutional operation to extract local temporal dependencies. where s is the sequence length. The causal convolution operation $\mathcal{K}$ enhances the representational capacity of temporal features. Subsequently, we generate multi-level feature representations using a multi-grained feature-extraction method that combines multi-head self-attention, convolution, and mean pooling. The formula for calculating the weight distribution for the input text features ${\mathbf{T}_{b,s,d}^{e}}$ is as follows: where $\sigma (\cdot )$ denotes the sigmoid activation function, and $\operatorname{LN}(\cdot )$ denotes layer normalization (Ba et al., 2016). $\mathcal{A}({\mathbf{T}_{b,s,d}})$ denotes the multi-head self-attention operation, $\mathcal{C}({\mathbf{T}_{b,s,d}})$ denotes the convolution operation, and $\mathcal{M}({\mathbf{T}_{b,s,d}})$ denotes the mean pooling operation. Then, ${\mathbf{G}_{b,s,d}}$ is used to weight and adjust the features processed by causal convolution, yielding the dynamically adjusted feature representation as follows: This operation produces the dynamically adjusted feature representation after applying the learned gating mechanism.

After processing, crucial information in both image and text features is preserved. To capture the correlation between image and text more precisely, we employ bidirectional cross-modal attention for interaction. Bidirectional cross-modal interaction is modelled in two directions: ${E_{\alpha }}\to {E_{\beta }}$, where $({E_{\alpha }},{E_{\beta }})\in \{({\mathbf{Y}_{b,c,l}^{\textit{final}}},{\mathbf{T}_{b,s,d}^{s}}),({\mathbf{T}_{b,s,d}^{s}},{\mathbf{Y}_{b,c,l}^{\textit{final}}})\}$. This approach improves the complementarity between image and text information, enhancing the model’s understanding of deeper relationships between the two modalities. where ${\mathbf{E}_{\alpha }}$ and ${\mathbf{E}_{\beta }}$ denote the feature representations of two modalities (e.g. image and text), ${\mathbf{W}_{\alpha }}$ and ${\mathbf{W}_{\beta }}$ are learnable projection matrices, ${d_{k}}$ denotes the scaling dimension, and ${\operatorname{Co}_{\textit{atts}}}(\cdot )$ denotes the cross-modal attention operation. Simultaneously, to enhance sequence position awareness, we introduce rotary positional embeddings for both the input query and the key (Su et al., 2024). Next, we concatenate the processed image features ${\mathbf{G}_{img}}\in {\mathbb{R}^{b,c,l}}$, text features ${\mathbf{G}_{txt}}\in {\mathbb{R}^{b,s,d}}$, and cross-attention features ${\mathbf{C}_{\alpha }}\in {\mathbb{R}^{b,c,l}}$ and ${\mathbf{C}_{\beta }}\in {\mathbb{R}^{b,s,d}}$. The concatenation operation joins these features column-wise, yielding a new combined feature: where $\mathcal{C}$ denotes the concatenation operation, and ${\mathbf{F}_{\gamma }}\in {\mathbb{R}^{b,{s^{\prime }},d}}$ represents the concatenated feature. We then compute the mean of the merged features, fusing the multimodal features into a more compact representation: where ${\mathbf{F}_{\textit{fused}}}\in {\mathbb{R}^{B\times {d_{f}}}}$ is the final fused representation, ${S^{\prime }}$ is the total number of concatenated positions, and ${d_{f}}$ is the common projected feature dimension. This removes the previously undefined normalizing constant $4D$.

All our experiments were conducted on the publicly available datasets MVSA-Single, MVSA-Multiple (Niu et al., 2016), and HFM (Cai et al., 2019). MVSA-Multiple is an enhanced version of MVSA-Single, featuring a larger number of image-text pairs suitable for sentiment analysis. We adopted the same preprocessing strategy as Xu and Mao (2017). The HFM dataset is primarily used for multimodal sarcasm detection, a task with two sentiment polarities: positive and negative. For a fair comparison, we used the consistent data preprocessing method described in Cai et al. (2019). A statistical analysis of the datasets is presented in Table 1.

MVSA-Single and MVSA-Multiple are available at.1 The HFM dataset can be downloaded from.2

In deep learning, the choice of hyperparameters significantly impacts model performance. For the Multimodal Sentiment Analysis Model with Frequency-Domain Decoupling and Semantic Filtering proposed in this paper, the order of the Discrete Wavelet Transform (DWT) is a crucial hyperparameter. To thoroughly understand its influence on model performance, a series of experimental analyses was conducted, as shown in Fig. 3. The results indicate that the model performs best when using the db2 wavelet basis, while db1 performs significantly worse, and the performance of db3, db4, and db5 decreases sequentially.

Specifically, the advantage of the db2 wavelet basis lies primarily in its balanced frequency-domain decoupling, which enables it to effectively process both LF global semantic information and HF noise in an image. db2 can not only precisely extract important semantic features from an image but also avoid excessive filtering that could lead to the loss of sentiment information, thereby enhancing overall model performance. In contrast, while the db1 wavelet basis is relatively simpler, its frequency-domain decomposition ability is weaker, failing to effectively handle complex image details. Our method provided insufficient noise suppression, leading to poorer model performance. Furthermore, although db3, db4, and db5 may offer advantages in processing certain frequency-domain details, their overly detailed decomposition methods can lead to excessive filtering and loss of image information, especially in finding a suitable balance between noise reduction and feature retention, ultimately causing a gradual decline in model performance. Therefore, db2 performs optimally in this model because it can suppress unnecessary noise while preserving effective information, thus maximizing the model’s effectiveness.

To gain a deeper, more intuitive understanding of how our model processes multimodal data, we employed various visualization methods to observe it from different perspectives.

To further explore the performance of our proposed model in sentiment analysis, we selected several representative cases for in-depth analysis, as shown in Fig. 7. This aims to comprehensively evaluate the model’s robustness and accuracy. The analysis results indicate that when image information is unclear or somewhat ambiguous, the model struggles to determine sentiment polarity, primarily because it lacks the ability to extract and understand the global sentiment tone. For instance, in the first case, despite explicit sentiment cues such as “feeling dizzy lol” in the text, the model’s confidence in its judgment was low due to the ambiguity of the image’s sentiment.

In contrast, in the other two cases in the table, the model demonstrated higher confidence in its judgment. For example, in the third case, the prominent flowers in the image mostly convey positive sentiment. Concurrently, the model effectively suppresses noise information outside the target object in the image, preventing it from interfering with sentiment analysis. The results above indicate that our proposed model has strong capabilities for extracting sentiment information from both images and text, thereby enhancing its overall performance in sentiment analysis.