Research & Publications
My research interests span Multimodal Deep Learning, Vision-Language Foundation Models, Cross-Modal Attention Fusion, and Automated Misinformation Detection.
-
A Unified Multimodal Framework for Fake News Detection Using BERT and Vision Transformers Accepted & Presented 96.66% SOTA Acc
Pioneered an end-to-end multimodal architecture combining disentangled contextual language representations (DeBERTa-v3 / RoBERTa / BERT) and visual feature hierarchies (Swin Transformer, CLIP-ViT). Introduced an adaptive Cross-Modal Gated Fusion layer that dynamically balances text and visual cues, outperforming competitive baselines on the Fakeddit (91.15%) and SPECTRA (96.66%) benchmarks.
Key Empirical Findings & SOTA Benchmarks:
- SPECTRA Dataset: 96.66% Accuracy | 96.75% Macro F1 (SOTA Benchmark)
- Fakeddit Dataset: 91.15% Accuracy | 89.78% Macro F1
- Text Stream: DeBERTa-v3 disentangled content and position vectors with relative position matrices.
- Vision Stream: Swin-B shifted window hierarchical attention + CLIP-ViT visual projections.
- Novel Gated Fusion: \( g = \sigma(W_g \cdot [h_t; h_v]) \) dynamically filtering incongruent noise.
PyTorch DeBERTa-v3 Swin ViT CLIP-ViT Gated Cross-Attention 5-Fold CV
Current Research Focus Areas
- Cross-Modal Gated Alignment: Designing non-linear gating mechanisms that evaluate modality trustworthiness and mitigate hallucinated or ungrounded correlations in vision-language models.
- Hierarchical Vision Transformers: Exploring multi-scale self-attention mechanisms (Swin Transformer, MaxViT) for high-resolution visual artifact detection.
- Explainable Multimodal AI: Integrating attribution maps and gradient-based cross-attention rollouts to make multimodal decisions interpretable.
