1. Home
  2. To appear
  3. Frequency Domain Decoupling and Semantic ...

Informatica


Frequency Domain Decoupling and Semantic Filtering Network for Multimodal Sentiment Analysis
Pub. online: 18 August 2026      Type: Research Article      Open accessOpen Access
Received
1 June 2025
Accepted
1 August 2026
Published
18 August 2026

Abstract

Multimodal sentiment analysis, due to its comprehensive ability to capture user sentiment, has significant application value in areas such as public opinion analysis. Existing research, however, falls short in several aspects: (1) it inadequately models the global structural information of the image, and (2) it overlooks the potential noise impact within each modality. These limitations hinder the accurate extraction of sentiment cues from individual modalities. To address these issues, we propose a Frequency Domain Decoupling and Semantic Filtering Network for Multimodal Sentiment Analysis. This network primarily integrates frequency-domain decoupling with semantic filtering to process high- and low-frequency image information separately, thereby enhancing model performance. Specifically, we designed a Dynamic Frequency Domain Decoupling Module that applies discrete wavelet transforms for differentiated image processing. This module, combined with a Dual-Domain Loss Function, constrains consistency between the text semantic space and the frequency distribution of the optimized image features, preventing sentiment information loss from excessive filtering. The module also incorporates two key components: a Global Semantic Sentiment Component (GSSC) and a High-Frequency Filtering Component (HFFC). In the GSSC component, we designed a Hybrid Mamba to leverage text in capturing global semantic information from low-frequency image data. Furthermore, our HFFC component generates a dynamic weight matrix guided by text, enabling quantitative noise suppression. Additionally, we developed a Multi-Grained Semantic Purification Module to filter noise at the word, phrase, and sentence levels. Experimental findings from publicly accessible datasets indicate that our proposed model achieves competitive performance compared with existing methods on multimodal sentiment analysis under the adopted experimental settings and sarcasm detection tasks, validating the effectiveness of our noise-suppression method in cross-modal sentiment analysis. A key limitation of the model is its use of DWT: downsampling-related resolution loss in low-frequency subbands and independent subband partitioning compromise the capture of large-scale global structural correlations and local-global feature modelling, which advanced transform techniques can alleviate.

References

 
An, J., Zainon, W.M.N.W. (2023). Integrating color cues to improve multimodal sentiment analysis in social media. Engineering Applications of Artificial Intelligence, 126, 106874.
 
Aziz, A., Chowdhury, N.K., Kabir, M.A., Chy, A.N., Siddique, Md.J. (2025). MMTF-DES: a fusion of multimodal transformer models for desire, emotion, and sentiment analysis of social media data. Neurocomputing, 623, 129376.
 
Ba, J.L., Kiros, J.R., Hinton, G.E. (2016). Layer Normalization. arXiv preprint arXiv:1607.06450.
 
Bibi, M., Abbasi, W.A., Aziz, W., Khalil, S., Uddin, M., Iwendi, C., Gadekallu, T.R. (2022). A novel unsupervised ensemble framework using concept-based linguistic methods and machine learning for twitter sentiment analysis. Pattern Recognition Letters, 158, 80–86.
 
Cai, Y., Cai, H., Wan, X. (2019). Multi-modal sarcasm detection in twitter with hierarchical fusion model. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2506–2515.
 
Chen, D., Su, W., Wu, P., Hua, B. (2023). Joint multimodal sentiment analysis based on information relevance. Information Processing & Management, 60(2), 103193.
 
Chen, Y. (2015). Convolutional Neural Network for Sentence Classification. Master’s thesis, University of Waterloo.
 
Cheng, H., Yang, Z., Zhang, X., Yang, Y. (2023). Multimodal sentiment analysis based on attentional temporal convolutional network and multi-layer feature fusion. IEEE Transactions on Affective Computing, 14(4), 3149–3163.
 
Dai, J., H., Yan, T., Sun, P., Liu, X., Qiu (2021). Does syntax matter? A strong baseline for aspect-based sentiment analysis with RoBERTa. arXiv preprint arXiv:2104.04986.
 
Das, R., Singh, T.D. (2023). Multimodal sentiment analysis: a survey of methods, trends, and challenges. ACM Computing Surveys, 55(13s), 1–38.
 
Daubechies, I., Heil, C. (1992). Ten Lectures on Wavelets. SIAM, Philadelphia.
 
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186.
 
Do, H.N., Phan, H.T., Nguyen, N.T. (2024). Multimodal sentiment analysis using deep learning and fuzzy logic: a comprehensive survey. Applied Soft Computing, 167, 112279.
 
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., Gelly S., Uszkoreit J., Houlsby N. (2020). An image is worth 16×16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
 
Feng, Z., Han, D., Li, J., Shi, S., Li, K.-C. (2026). Intrusion detection system for shipping communication networks based on federated distillation learning. Expert Systems with Applications, 299, 129966.
 
Fu, D.Y., Dao, T., Saab, K.K., Thomas, A.W., Rudra, A., Ré, C. (2022). Hungry Hungry Hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052.
 
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., Hussain, A. (2023). Multimodal sentiment analysis: a systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Information Fusion, 91, 424–444.
 
Ge, Y., Chen, Z., Yu, M., Yue, Q., You, R., Zhu, L. (2024). MambaTSR: you only need 90k parameters for traffic sign recognition. Neurocomputing, 599, 128104.
 
Gu, A., Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.
 
Gu, A., Gupta. A., Goel. K., Ré. C. (2022). On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems 35.
 
Gu, A., Goel, K., Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396.
 
He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
 
Hu, R., Yi, J., Chen, A., Chen, L. (2024). Multichannel cross-modal fusion network for multimodal sentiment analysis considering language information enhancement. IEEE Transactions on Industrial Informatics, 20(7), 9814–9824.
 
Huan, R., Zhong, G., Chen, P., Liang, R. (2023). UniMF: a unified multimodal framework for multimodal sentiment analysis in missing modalities and unaligned multimodal sequences. IEEE Transactions on Multimedia, 26, 5753–5768.
 
Huang, S.-C., Shen, L., Lungren, M.P., Yeung, S. (2021). GLoRIA: a multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3922–3931.
 
Kang, H., Yoo, S.J., Han, D. (2012). Senti-lexicon and improved Naïve Bayes algorithms for sentiment analysis of restaurant reviews. Expert Systems with Applications, 39(5), 6000–6010.
 
Kim, K., Park, S. (2023). AOBERT: all-modalities-in-one BERT for multimodal sentiment analysis. Information Fusion, 92, 37–45.
 
Kim, W., Son, B., Kim, I. (2021). ViLT: vision-and-language transformer without convolution or region supervision. In: International Conference on Machine Learning. PMLR.
 
Kim, Y. (2014). Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1746–1751.
 
Li, J., Han, D., Shi, S., Xin, X., Li, K.-C., Chang, C.-C. (2025a). An active client selection scheme based on blockchain for federated learning in shipping. IEEE Transactions on Intelligent Transportation Systems, 26(11), 20669–20684.
 
Li, J., Han, D., Weng, T.-H., Wu, H., Li, K.-C., Castiglione, A. (2025b). A secure data storage and sharing scheme for port supply chain based on blockchain and dynamic searchable encryption. Computer Standards & Interfaces, 91, 103887.
 
Li, Y., Liang, W., Xie, K., Zhang, D., Xie, S., Li, K. (2023). LightNestle: quick and accurate neural sequential tensor completion via meta learning. In: IEEE INFOCOM 2023-IEEE Conference on Computer Communications. IEEE, pp. 1–10.
 
Li, Z., Xu, B., Zhu, C., Zhao, T. (2022). CLMLF: a contrastive learning and multi-layer fusion method for multimodal sentiment detection. arXiv preprint arXiv:2204.05515.
 
Liang, C., Liang, W., Chen, Y., Li, K., Zomaya, A.Y. (2025). FedTCTF: tensor completion-based federated learning for device heterogeneity. IEEE Transactions on Sustainable Computing, 10(6), 1227–1239.
 
Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P. (2017). Focal loss for dense object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
 
Liu, Y. (2019). RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXivurl1907.11692.
 
Lu, Q., Sun, X., Long, Y., Gao, Z., Feng, J., Sun, T. (2024). Sentiment analysis: comprehensive reviews, recent advances, and open challenges. IEEE Transactions on Neural Networks and Learning Systems, 35(11), 15092–15112.
 
Mai, S., Zeng Y., Zheng S., Hu H. (2022). Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Transactions on Affective Computing, 14(3), 2276–2289.
 
Mallat, S.G. (1989). A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7), 674–693.
 
Moraes, R., Valiati, J.F., Neto, W.P.G. (2013). Document-level sentiment classification: an empirical comparison between SVM and ANN. Expert Systems with Applications, 40(2), 621–633.
 
Niu, T., Zhu, S., Pang, L., El Saddik, A. (2016). Sentiment analysis on multi-view social data. In: Proceedings of the 22nd International Conference on Multimedia Modeling. Springer, Cham, pp. 15–27.
 
Pan, Z., Li, C., Plaza, A., Chanussot, J., Hong, D. (2025). Hyperspectral image classification with Mamba. IEEE Transactions on Geoscience and Remote Sensing, 63, 5602814.
 
Pandey, A., Vishwakarma, D.K. (2024). Progress, achievements, and challenges in multimodal sentiment analysis using deep learning: a survey. Applied Soft Computing, 152, 111206.
 
Parveen, N., Chakrabarti, P., Hung, B.T., Shaik, A. (2023). Twitter sentiment analysis using hybrid gated attention recurrent network. Journal of Big Data, 10(1), 50.
 
Pashchenko, Y., Rahman, M.F., Hossain, S., Uddin, K., Islam, T. (2022). Emotional and the normative aspects of customers’ reviews. Journal of Retailing and Consumer Services, 68, 103011.
 
Rodríguez-Ibáñez, M., Casánez-Ventura, A., Castejón-Mateos, F., Cuenca-Jiménez, P.-M. (2023). A review on sentiment analysis from social media platforms. Expert Systems with Applications, 223, 119862.
 
Shunxiang, Z., Aoqiang Z., Guangli Z., Zhongliang W., KuanChing L. (2023). Building fake review detection model based on sentiment intensity and PU learning. IEEE Transactions on Neural Networks and Learning Systems, 34(10), 6926–6939.
 
Singh, U., Abhishek, K., Azad, H.K. (2024). A survey of cutting-edge multimodal sentiment analysis. ACM Computing Surveys, 56(9), 1–38.
 
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y. (2024). RoFormer: enhanced transformer with rotary position embedding. Neurocomputing, 568, 127063.
 
Sun, H., Li, W., Liu, J., Chen, H., Pei, R., Zou, X., Yan, Y., Yang, Y. (2024). CoSeR: bridging image and language for cognitive super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
 
Sun, H., Niu, Z., Wang, H., Yu, X., Liu, J., Chen, Y.-W. (2025). Multimodal sentiment analysis with mutual Information-based disentangled representation learning. IEEE Transactions on Affective Computing, 16(3), 1606–1617.
 
Tai, K.S., Socher, R., Manning, C.D. (2015). Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075.
 
Tang, D., Qin, B., Liu, T. (2015). Document modeling with gated recurrent neural network for sentiment classification. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
 
Tian, H., Gao, C., Xiao, X., Liu, H., He, B., Wu, H., Wang, H., Wu, F. (2020). SKEP: sentiment knowledge enhanced pre-training for sentiment analysis. arXiv preprint arXiv:2005.05635.
 
Truong, Q.-T., Lauw, H.W. (2019). VistaNet: visual aspect attention network for multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 305–312.
 
Usama, M., Ahmad, B., Song, E., Hossain, M.S., Alrashoud, M., Muhammad, G. (2020). Attention-based sentiment analysis using convolutional and recurrent neural network. Future Generation Computer Systems, 113, 571–578.
 
Wang, D., Liu, S., Wang, Q., Tian, Y., He, L., Gao, X. (2022). Cross-modal enhancement network for multimodal sentiment analysis. IEEE Transactions on Multimedia, 25, 4909–4921.
 
Wang, D., Tian, C., Liang, X., Zhao, L., He, L., Wang, Q. (2023a). Dual-perspective fusion network for aspect-based multimodal sentiment analysis. IEEE Transactions on Multimedia, 26, 4028–4038.
 
Wang, H., Du, Q., Xiang, Y. (2025a). Image–text sentiment analysis based on hierarchical interaction fusion and contrast learning enhanced. Engineering Applications of Artificial Intelligence, 146, 110262.
 
Wang, H., Ren, C., Yu, Z. (2025b). Multimodal sentiment analysis based on multiple attention. Engineering Applications of Artificial Intelligence, 140, 109731.
 
Wang, J., Yang, Y., Jiang, Y., Ma, M., Xie, Z., Li, T. (2024). Cross-modal incongruity aligning and collaborating for multi-modal sarcasm detection. Information Fusion, 103, 102132.
 
Wang, L., Niu, J., Yu, S. (2019). SentiDiff: combining textual information and sentiment diffusion patterns for Twitter sentiment analysis. IEEE Transactions on Knowledge and Data Engineering, 32(10), 2026–2039.
 
Wang, X., Wang, M., Cui, H., Zhang, Y. (2025c). Manifold knowledge-guided feature fusion network for multimodal sentiment analysis. Expert Systems with Applications, 280, 127537.
 
Wang, Y., Huang, G., Li, M., Li, Y., Zhang, X., Li, H. (2023b). Automatically constructing a fine-grained sentiment lexicon for sentiment analysis. Cognitive Computation, 15(1), 254–271.
 
Wang, Y., Jian, H., Zhuang, J., Guo, H., Leng, Y. (2025d). SSLMM: semi-supervised learning with missing modalities for multimodal sentiment analysis. Information Fusion, 120, 103058.
 
Wei, Y., Yuan, S., Yang, R., Shen, L., Li, Z., Wang, L., Chen, M. (2023). Tackling modality heterogeneity with multi-view calibration network for multimodal sentiment detection. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 5240–5252.
 
Wu, H., Kong, D., Wang, L., Li, D., Zhang, J., Han, Y. (2025). Multimodal sentiment analysis method based on image-text quantum transformer. Neurocomputing, 637, 130107.
 
Xie, X., Cui, Y., Tan, T., Zheng, X., Yu, Z. (2024). FusionMamba: dynamic feature enhancement for multimodal image fusion with Mamba. Visual Intelligence, 2(1), 37.
 
Xu, N. (2017). Analyzing multimodal public sentiment based on hierarchical semantic attentional network. In: Proceedings of the 2017 IEEE International Conference on Intelligence and Security Informatics (ISI’17), pp. 152–154.
 
Xu, N., Mao, W. (2017). MultiSentiNet: a deep semantic network for multimodal sentiment analysis. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 2399–2402.
 
Xu, N., Mao, W., Chen, G. (2018). A co-memory network for multimodal sentiment analysis. In: Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 929–932.
 
Xue, X., Zhang, C., Niu, Z., Wu, X. (2022). Multi-level attention map network for multimodal sentiment analysis. IEEE Transactions on Knowledge and Data Engineering, 35(5), 5105–5118.
 
Yang, X., Feng, S., Zhang, Y., Wang, D. (2021). Multimodal sentiment detection based on multi-channel graph neural networks. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 328–339.
 
Ye, J., Zhang, J., Shan, H. (2025). DepMamba: progressive fusion mamba for multimodal depression detection. In: ICASSP 2025–2025 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, pp. 1–5.
 
Zhai, G., Yang, Y., Wang, H., Du, S. (2020). Multi-attention fusion for sentiment analysis of educational big data. Big Data Mining and Analytics, 3(4), 311–319.
 
Zhang, B., Yuan, Z., Xu, H., Gao, K. (2024). Crossmodal translation based meta weight adaption for robust image-text sentiment analysis. IEEE Transactions on Multimedia, 26, 9949–9961.
 
Zhang, S., Liu, J., Jiao, Y., Zhang, Y., Chen, L., Li, K. (2025a). A multimodal semantic fusion network with cross-modal alignment for multimodal sentiment analysis. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(10), 1–22. https://doi.org/10.1145/3744648.
 
Zhang, S., Li, J., Li, S., Duan, W., Wei, Z., Li, K.-C. (2025b). Textual graph representation with syntactic weighting for implicit sentiment analysis. IEEE Transactions on Emerging Topics in Computational Intelligence, 9(6), 4288–4299. https://doi.org/10.1109/TETCI.2025.3550515.
 
Zhang, Y., Wang, L., Wang, H. (2023). Cross-modal sentiment analysis based on transformer and image-text collaborative interaction. In: 2023 International Conference on Computer Engineering and Distance Learning (CEDL). IEEE.
 
Zhao, S., Yao X., Yang J., Jia G., Ding G., Chua T.-S., Schuller B.W., Keutzer K. (2021). Affective image content analysis: two decades review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6729–6757.
 
Zhao, X., Poria, S., Li, X., Chen, Y., Tang, B. (2025a). Toward robust multimodal sentiment analysis using multimodal foundational models. Expert Systems with Applications, 276, 126974.
 
Zhao, Y., Li, K.-C., Zhang, S., Ye, T. (2025b). Hyperbolic graph attention network fusing long-context for technical keyphrase extraction. Information Fusion, 120, 103061. https://doi.org/10.1016/j.inffus.2025.103061.
 
Zhao, Z., Zhu, H., Xue, Z., Liu, Z., Tian, J., Heng Chua, M.C., Liu, M. (2019). An image-text consistency driven multimodal sentiment analysis approach for social media. Information Processing & Management, 56(6), 102097.
 
Zhou, P., Shi, W., Tian, J., Qi Z., Li B., Hao H., Xu B. (2016). Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 207–212.
 
Zhou, S., Chen, Y., Liang, W., Li, K., Zomaya, A.Y. (2026a). MultiSecDFL: multi-metric filtering-based secure aggregation method for decentralized federated learning. IEEE Transactions on Consumer Electronics. https://doi.org/10.1109/TCE.2026.3688316.
 
Zhou, S., Chen, Y., Liang, W., Li, K., Meng, W. (2026b). TrustHFL: an efficient aggregation method for trustworthy hierarchical federated learning. IEEE Internet of Things Journal, 13(11), 24618–24631.
 
Zhou, S., Li, K., Chen, Y., Yang, C., Liang, W., Zomaya, A.Y. (2024). TrustBCFL: mitigating data bias in IoT through blockchain-enabled federated learning. IEEE Internet of Things Journal, 11(15), 25648–25662.
 
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., Kong, X. (2023). Multimodal sentiment analysis based on fusion methods: a survey. Information Fusion, 95, 306–325.
 
Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., Wang, X. (2024). Vision Mamba: efficient visual representation learning with bidirectional state space model. In: International Conference on Machine Learning.
 
Zhu, T., Li, L., Yang, J., Zhao, S., Liu, H., Qian, J. (2022). Multimodal sentiment analysis with image-text interaction network. IEEE Transactions on Multimedia, 25, 3375–3385.

Biographies

Liu Yundong
szxylyd@126.com

Y. Liu received the master of science degree in educational technology from East China Normal University, China. He is currently an associate professor in the Computer Science and Technology program at Suzhou University, China. His research interests include image recognition and artificial intelligence.

Tan Chengfang
874036730@qq.com

C. Tan received the master of science degree in educational technology from Nanjing Normal University, China. She is currently an associate professor in the Computer Science and Technology program at Suzhou University, China. Her research interests include image recognition and artificial intelligence.

Zhang Shunxiang
sxzhang@aust.edu.cn

S. Zhang received the PhD degree from the School of Computer Engineering and Science, Shanghai University, Shanghai, China, in 2012. He is currently a professor with the School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan, China. His research interests include intelligent information processing, data mining, big data analytics, and sentiment analysis.

Zhang Yulei
2230817302@qq.com

Y. Zhang received the master of science degree from Anhui University of Science and Technology, Huainan, China. He is currently with the School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan, China. His research interests include multimodal sentiment analysis, multimodal sarcasm detection, deep learning, and multimodal information fusion.

Li Kuan-Ching
edge4xt@outlook.com

K.-C. Li received the PhD degree in electrical engineering from the University of São Paulo, Brazil. He is currently a distinguished professor with the Department of Computer Science and Information Engineering, Providence University, Taiwan. He is also affiliated with the School of Mathematics and Big Data, Anhui University of Science and Technology, Huainan, China. His research interests include cloud computing, GPU computing, big data, and parallel programming.

Crespo Rubén González
ruben.gonzalez@unir.net

R. Crespo received the PhD degree in computer science engineering from Universidad Pontificia de Salamanca, Spain. He is currently a vice-rector and full professor of Computer Science and Artificial Intelligence at Universidad Internacional de La Rioja (UNIR), Spain. His research interests include artificial intelligence, Industry 4.0, project management, and accessibility.


Full article PDF XML
Full article PDF XML

Copyright
© 2026 Vilnius University
Open access article under the CC BY license.

Keywords
sentiment analysis multimodal frequency domain semantic filtering

Metrics
since January 2020
51

Article info
views

9

Full article
views

11

PDF
downloads

4

XML
downloads


Share


RSS