Our paper has been accepted to the Many Lenses, One World: Culturally-Aware Interactive AI in the Metaverse Workshop at the IEEE International Conference on Image Processing (ICIP) 2026 [LINK]

• Title: Revealing Hidden Response Ambiguity in Binary Evaluation of Cultural Gesture Understanding

• Authors: Sunghun Kang, Seungjae Lee, Hongseok Cho, Hyeokjun Kweon, Hak Gu Kim

• Abstract: Gestures are culturally dependent forms of non-verbal communication whose meanings can vary substantially across regions. Recent benchmarks such as MC-SIGNS evaluate whether vision-language models (VLMs) correctly judge the appropriateness of gestures across cultural contexts, often by parsing responses into binary Yes/No outcomes. However, such binary evaluation can obscure important differences in the underlying response forms. In this paper, we investigate the hidden ambiguity underlying parsed “No” responses in culturally sensitive gesture understanding. We show that identical binary outcomes can arise from fundamentally different response forms, including culturally grounded rejection, over-refusal, and self-contradictory reasoning. To analyze this, we introduce a post-hoc response categorization framework that decomposes parsed “No” responses into interpretable categories using lexical cues in raw outputs. Experiments on three open-weight VLMs show that binary evaluation alone can substantially overestimate culturally informed gesture understanding by conflating valid negative judgments with spurious response forms.

Our paper has been accepted to the Interspeech 2026 (Top-tier Conference) [LINK]

• Title: ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion

• Authors: Hyung Kyu Kim, Byungchan Hwang, and Hak Gu Kim

• Abstract: Recent acoustic-to-articulatory inversion (AAI) models rely on electromagnetic articulography (EMA) data, which are costly and limited in scale. To address this limitation, we propose ArtBoost, a novel data augmentation strategy that leverages large-scale speech–mesh datasets originally developed for speech-driven 3D facial animation to improve AAI under limited EMA supervision. ArtBoost extracts pseudo articulatory trajectories from visible facial anchors and uses them for pre-training before fine-tuning on real EMA data. Experiments show consistent improvements in PCC and RMSE. Trajectory analyses confirm that the pseudo articulatory signals reflect physically meaningful visible articulatory dynamics. Additional evaluations across different AAI architectures demonstrate stable performance gains, indicating that ArtBoost can be integrated into diverse AAI models. These results suggest that speech–mesh data provide an effective and scalable source of articulatory supervision for AAI.

Our paper has been accepted to the IEEE Trans. Information Forensics and Security (TIFS) (JCR Top 7.8%, Impact Factor: 8.0) [LINK]

• Title: Deformable 3D Point Cloud Perturbations using Cage-based Deformation for Semantic Consistency

• Authors: Kyo Seok Lee and Hak Gu Kim

• Abstract: Deep neural networks for 3D point cloud analysis are widely used in applications such as autonomous driving and robotics, yet they remain highly vulnerable to adversarial attacks. Existing methods typically minimize point-wise distances to preserve geometry, which constrains perturbations and leads to a trade-off between imperceptibility and attack strength. To address this limitation, we propose a cage-based adversarial deformation framework that generates semantically consistent perturbations aligned with natural intra-class variations. Our method refines a source cage, predicts adversarial cage displace ments by fusing source–target features, and computes smooth point-wise offsets using solid-angle– and distance-aware weights. This enables globally coherent deformations that appear natural to humans while effectively misleading classifiers. Experiments on ModelNet40 and ShapeNet-Part show that our approach achieves state-of-the-art attack success rates while producing the most uniform point distributions and lowest local distortions. Furthermore, the perturbations remain effective against common defenses such as SRS, SOR, and DUP-Net.

Our paper has been accepted to the IEEE Signal Processing Letters (SPL) (Impact Factor: 3.9) [LINK]

• Title: Unsharp-Inspired Adversarial Point Cloud Perturbation via Low-Rank Approximation

• Authors: Kyo Seok Lee, Han-nyoung Lee, and Hak Gu Kim

• Abstract: Deep neural networks (DNNs) for 3D point cloud recognition have achieved remarkable performance, but remain highly vulnerable to adversarial perturbations. Existing adversarial methods often suffer from a trade-off between imperceptibility and attack strength, either producing noticeable outliers or requiring excessive point displacements. In this paper, we propose a novel unsharp-inspired adversarial perturbation that leverages low-rank approximation to balance global structure preservation and fine-detail manipulation. Specifically, the point cloud is decomposed via eigenvalue decomposition (EVD) into global and residual components, where the low-rank reconstruction captures the overall shape and the residual highlights salient details. Perturbations are then selectively applied to these fine-detail regions and smoothed according to local magnitude and orientation, ensuring geometric consistency. Extensive experiments on ModelNet40 and ShapeNet-Part using PointNet and DGCNN demonstrate that our method achieves 100\% attack success rate while significantly reducing geometric distortion compared to state-of-the-art methods. These results highlight the effectiveness of spectral-domain low-rank modeling for generating adversarial point clouds that are both strong and imperceptible.

Our Master’s student Seungjae Lee has won the Best Student Paper Award at IEIE Summer Conference 2025 [LINK]

• Title: 3D Head and Body Pose Estimation 기반 갑상선 환자 재활운동 정량적 평가 및 안내 시스템

• Authors: 이승재, 김호준, 이하린, 김태용, 김학구

• Abstract: With the rising incidence of thyroid cancer in Korea, postoperative neck rehabilitation has become increasingly important. However, many patients struggle to follow physician-recommended exercises due to pain and lack of real-time monitoring, leading to delayed recovery. To address this problem, in this paper, we propose a deep learning-based automatic rehabilitation exercise guide system that quantitatively evaluates rehabilitation performance of a patient using 3D head and body pose estimation with real-time feedback. The proposed system also integrates a metaverse-based environment, enabling patients to perform guided exercises remotely without visiting the hospital.

Our paper has been accepted to the IEEE/CVF International Conference on Computer Vision (ICCV) 2025 (Top-tier Conference) [LINK]

• Title: MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

• Authors: Hyung Kyu Kim, Sangmin Lee (Korea University), and Hak Gu Kim

• Abstract: Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker’s speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: <1-stage> is storing and retrieving general motion (i.e., Memorizing), and <2-stage> is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods. Our source code will be released to facilitate further research.

Our paper has been accepted to the IEEE International Conference on Image Processing (ICIP) 2025 [LINK]

• Title: Enhancing 3D Scene Representation with Structural Dissimilarity-Aware Learning

• Authors: Seungjae Lee, Ho Jun Kim, and Hak Gu Kim

• Abstract: Novel view synthesis aims to generate high-quality unseen views from images at different viewpoints. However, the existing methods often struggle to preserve fine details, leading to structural distortions in complex regions. In this paper, we introduce a simple yet effective structure-aware objective function designed to enhance structural information in novel view synthesis. By leveraging the Structural Similarity Index (SSIM), our method attends to regions exhibiting significant structural distortions. We incorporate structural dissimilarity-based attention to highlight discrepancies in challenging regions between predicted and ground-truth images. It enables recent 3D scene representation models to achieve improved structure preservation, leading to more coherent representations. Experiments on synthetic and real-world datasets demonstrate that our approach enhances structural consistency, particularly in challenging regions, making it a valuable addition to state-of-the-arts.

Our paper has been accepted to the Interspeech 2025 (Top-tier Conference) [LINK]

• Title: Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation

• Authors: Hyung Kyu Kim and Hak Gu Kim

• Abstract: Speech-driven 3D facial animation aims to generate realistic facial movements synchronized with audio. Traditional methods primarily minimize reconstruction loss by aligning each frame with ground-truth. However, this frame-wise approach often fails to capture the continuity of facial motion, leading to jittery and unnatural outputs due to coarticulation. To address this, we propose a novel phonetic context-aware loss, which explicitly models the influence of phonetic context on viseme transitions. By incorporating a viseme coarticulation weight, we assigns adaptive importance to facial movements based on their dynamic changes over time, ensuring smoother and perceptually consistent animations. Extensive experiments demonstrate that replacing the conventional reconstruction loss with ours improves both quantitative metrics and visual quality. It highlights the importance of explicitly modeling phonetic context dependent visemes in synthesizing natural speech-driven 3D facial animation.

Our paper has been accepted to the IEEE Access (Impact Factor: 4.2) [LINK]

• Title: Leveraging Text Signed Distance Function Map for Boundary-Aware Guidance in Scene Text Segmentation

• Authors: Ho Jun Kim and Hak Gu Kim

• Abstract: Scene text segmentation is to predict pixel-wise text regions from an image, enabling in-image text editing or removal. One of the primary challenges is to remove noises including non-text regions and predict intricate text boundaries. To deal with that, traditional approaches utilize a text detection or recognition module explicitly. However, they are likely to highlight noise around the text. Because they did not sufficiently consider the boundaries of text, they fail to accurately predict the fine details of text. In this paper, we introduce leveraging text signed distance function (SDF) map, which encodes distance information from text boundaries, in scene text segmentation to explicitly provide text boundary information. By spatial cross attention mechanism, we encode the text-attended feature from the text SDF map. Then, both visual and text-attended features are utilized to decode the text segmentation map. Our approach not only mitigates confusion between text and complex backgrounds by eliminating false positives such as logos and texture blobs located far from the text, but also effectively captures fine details of complex text patterns by leveraging text boundary information. Extensive experiments demonstrate that leveraging text SDF map in scene text segmentation provides superior performances on various scene text segmentation datasets.

Our workshop proposal has been accepted to the IEEE International Conference on Image Processing (ICIP) 2025 [LINK]

• Title: 2nd Integrating Image Processing with Large-Scale Language/Vision Models for Advanced Visual Understanding

• Authors: Yong Mak Ro (KAIST), Wen-Huang Cheng (National Taiwan Univ.), and Hak Gu Kim (Chung-Ang Univ.)

• Abstract: This workshop aims to bridge the gap between conventional image processing techniques and the latest advancements in large-scale vision and language models. Recent developments in large-scale models have revolutionized image processing tasks, significantly enhancing capabilities in visual object understanding, image classification, and generative image synthesis. Furthermore, the large-scale models have opened new avenues for human-machine multimodal interactive dialogue systems, where the synergy between visual and linguistic processing enables more intuitive and dynamic interactions. This workshop will provide a platform for researchers and practitioners to explore how cutting-edge large-scale models integrate with image processing methods and foster innovation across diverse applications. Discussions will extend beyond conventional tasks to address the role of vision-language models in Generative AI and their use in multimodal systems, such as virtual assistants that interact seamlessly using images, text, and speech.