Our work, “TW-CRL:Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learning”, was accepted and presented at AAAI 2026 for an oral presentation. This work was a collaboration between Yuxuan Li, Yicheng Gao, Ning Yang, and Stephen Xia.

Reinforcement learning systems are powerful, but they often struggle in the real world for a simple reason: failure is silent. In many tasks—robot navigation, manipulation, or control—agents receive a reward only when they succeed, and nothing at all when they make irreversible mistakes. These hidden “trap states” can doom an agent to fail repeatedly without ever being told what went wrong.

We present a new framework called Time-Weighted Contrastive Reward Learning (TW-CRL), which tackles this problem head-on by teaching agents to learn not just from success, but from failure, and to understand when those failures matter most.

What sets this work apart is its contrastive use of both successful and failed demonstrations. Rather than blindly imitating experts, the agent learns to distinguish states that move it closer to a goal from those that quietly push it toward failure. This allows the system to avoid unseen trap states, explore more intelligently, and go beyond simple imitation.

Moreover, instead of treating all steps in a demonstration equally, the method assigns greater importance to states that occur later in a trajectory, when success or failure becomes inevitable. By doing so, TW-CRL learns a dense, informative reward function that highlights critical decision points and implicitly penalizes states that lead to dead ends.

Across eight challenging benchmarks, from maze navigation with hidden traps to continuous-control robotics tasks, TW-CRL consistently outperforms state-of-the-art inverse reinforcement learning methods. It converges faster, achieves higher returns, and exhibits greater robustness, especially in environments where failure is common early on.

By formalizing the notion of trap states and showing how temporal structure and failed experiences can be turned into actionable learning signals, this work pushes reinforcement learning a step closer to real-world reliability, where knowing what not to do is just as important as knowing what works.

Congratulations to the team!

Our work, “MAESTRO: Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series”, was accepted and presented at NeurIPS 2025 as a spotlight. This work was a collaboration between Payal Mohapatra, Yueyuan Sui, Akash Pandey, Stephen Xia, and Qi Zhu from Northwestern University.

In this work, we present a new AI framework designed to make sense of dynamic, multimodal time-series data—even when large chunks of it are missing. Instead of forcing all sensor data into a rigid, one-size-fits-all model, MAESTRO adapts on the fly, deciding which signals matter most and how they should interact for a given task. Novelties, features, and contributions include:

(1) No “primary sensor: bias: Most existing systems depend on a single anchor modality (like video or motion). MAESTRO drops that assumption entirely, letting any modality take the lead when it’s most informative, and stepping back when it’s not.

(2) Built for missing and failing sensors: Real deployments rarely have perfect data. MAESTRO explicitly encodes missing sensors using symbolic tokens and dynamically reallocates its attention budget, allowing it to remain accurate even when up to 40% of modalities are missing.

(3) Efficient learning at scale: Instead of expensive pairwise modeling between every sensor, MAESTRO builds a long multimodal sequence and processes it with sparse attention, reducing computation while still capturing rich cross-modal interactions.

(4) Adaptive expertise via Mixture-of-Experts: At the final stage, MAESTRO routes information through a sparse Mixture-of-Experts (MoE) layer. Different experts naturally specialize for different sensor combinations—without extra losses or hand-crafted rules—making the system robust and flexible in unpredictable settings.

We evaluate across four real-world datasets spanning healthcare, activity recognition, and clinical prediction—with as many as 17 sensing modalities. MAESTRO consistently outperforms strong multivariate and multimodal baselines, including 4–8% gains under full sensor availability, and ~9% average improvement when sensors are partially missing.

MAESTRO moves multimodal learning closer to real-world readiness, offering a general-purpose blueprint for learning from heterogeneous, unreliable sensor data. This is exactly the type of scenarios we see in wearables, mobile health, and IoT systems today.

If multimodal time-series learning is an orchestra, MAESTRO is the conductor that keeps the music playing—even when half the instruments drop out.

Congratulations to the team!

Stephen Xia hosts the IASA 2025 workshop at MobiCom 2025, which is the premier venue for all aspects of acoustic sensing, systems, and applications, including mobile and wearable sensing or communication utilizing sound, vibration, ultrasound, and infrasound, algorithms in acoustic intelligence, security and privacy utilizing acoustics, and data or deployment experiences.

This full day workshop included two key notes: Professor Qian Zhang (HKUST) discussing her work at the intersection between acoustics and health, as well as Professor Nirupam Roy (University of Maryland, College Park), discussing his work at the intersection between acoustics and AI. The workshop also featured a panel of experts and seven full papers covering topics such as in-ear acoustics, smartphone sensing, and fitness/health.

The full program can be found here.

Our work, “IMUSteth: On-Body Stethoscope Localization with Inertial Sensing for Home Self-Screening”, was accepted and presented at the IASA 2025 workshop, during MobiCom 2025. IASA 2025 is the premier workshop for all aspects of acoustic sensing, systems, and applications.

In this work, we extend our prior work on an augmented reality-based stethoscope, originally published to IPSN 2023, that guides users on the steps to take to screen their own health without a professional healthcare provider. This requires the system to understand where the user placed the stethoscope on their body, which leveraged a full-body view from a camera. This is both unwieldy and privacy sensitive.

In this work, we remove the need for the user to position themselves in front of a camera. Instead, we leverage only an inertial measurement unit, several physical characteristics of the user and an initial calibration phase (e.g., move arms in a circle) to track a user’s placement of the stethoscope on their body. This enables more freedom on where and when the user can perform health screening, making accessibility to preventative health screening more available.

Congratulations to Yiting Zhang and the team!

Our work, “FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systems”, was accepted and presented at SenSys 2025 during CPS-IoT Week 2025. This research was developed through a collaborative effort between Northwestern University and Columbia University, with contributions from Minghui (Scott) Zhao, Junxi Xia, Kaiyuan Hou, Yanchen Liu, and Professor Stephen Xia, Professor Xiaofan (Fred) Jiang.

In this work, we addressed a fundamental challenge in enabling foundation models to interact meaningfully with physical environments. While foundation models excel at digital tasks, they struggle when required to understand localized events in large physical spaces or perform real-world actions. Our contributions are summarized as follows:

(1) Novel Foundation Model Orchestration Framework: We introduce a comprehensive framework that integrates multiple large language models, vision-language models, and an open-set object detection model for dynamic, task-driven sensor/actuator selection.

(2) Advanced Image Segmentation Approach: We propose a new segmentation method that reliably pinpoints task-relevant areas even in large, cluttered environments, overcoming limitations of existing approaches.

(3) Custom Reconfigurable Drone Platform: We present a purpose-built drone system capable of autonomously adapting its sensors and actuators in response to commands from the FM orchestration framework, enabling seamless physical interaction and significantly improving task success rates in real-world deployments compared to static sensor approaches.

FlexiFly enables foundation models to “zoom in” on areas of interest using reconfigurable drones that can automatically swap between different sensing and actuation modules. Through extensive real-world evaluations, our system demonstrated up to 85% improvement in task success rates across diverse applications including object identification, environmental monitoring, surveillance, and physical item delivery.

Our paper, in collaboration with Columbia University, “DomAIn: Towards Programless Smart Homes” has been accepted for publication in ACM HumanSys 2025, co-located with CPS-IoT Week 2025, and received the Best Paper Award!

This work tackles a critical barrier to smart home adoption: the complexity of programming automation tasks. While 70% of U.S. homes have smart devices, widespread adoption remains limited because current platforms require users to manually program step-by-step logic, severely limiting accessibility.

DomAIn revolutionizes this by introducing a smart home platform that automatically generates and deploys automation logic from simple voice commands—no programming required. Users can say “Alert me if someone falls” and DomAIn automatically creates appropriate sensor pipelines based on available devices and user preferences across four dimensions: accuracy, privacy, coverage, and power efficiency.

Key innovations include:

• Voice-to-Automation: Simple spoken commands automatically generate complex sensor pipelines
• Preference-Based Optimization: Creates privacy-focused pipelines that minimize camera usage or power-efficient solutions using minimal energy
• Dynamic Adaptability: Automatically reconfigures when devices fail or become unavailable

The system was validated through real-world deployments using various sensors including cameras, microphones, vibration detectors, and programmable drones, demonstrating practical applicability across diverse smart home scenarios.

Congratulations to Yueyuan Sui, Yiting Zhang, Yanchen Liu, Minghui Zhao, Kaiyuan Hou, Jingping Nie, and the rest of the team!

Our paper, “Unsupervised Deep Clustering for Human Behavior Understanding” has been accepted for publication in ACM HumanSys 2025, co-located with CPS-IoT Week 2025!

In this work, we propose Compressed-Pseudo-Temporal Enhanced Representation Learning (C-PTER), a novel unsupervised clustering framework for human-centered behavior analysis (architecture shown above). With the growing prevalence of wearables, smartphones, and IoT devices, vast amounts of human activity data are collected in real-world settings, yet traditional supervised learning approaches require extensive manual labeling, making them impractical for large-scale deployment. Existing deep clustering methods, such as autoencoder-based approaches, often fail to capture temporal dependencies and struggle with noisy sensor readings, leading to suboptimal clustering performance.

C-PTER proposes a pseudo-temporal feature extractor with a parallel CNN-LSTM autoencoder to enable robust spatial-temporal representation learning. We demonstrate up to 30% improvement in normalized mutual information and 21% improvement in downstream task performance on a variety of human activity datasets, compared with existing state-of-art clustering methods.

Congratulations to Weisi Yang and the rest of the team!

Weisi Yang presents our work on a smartphone-based platform for detecting and monitoring intimate partner infiltration at the SIGCHI sponsored Mobile Social Networking Workshop at Discovery Partners Institute.

Our work introduces AID (Automated IPI Detection), the first system designed specifically to detect Intimate Partner Infiltration, a class of abuse that slips past traditional cybersecurity defenses because the perpetrator often already has the phone, the passcode, or the victim’s trust.

Rather than treating these incidents as conventional cyberattacks, we formalize a new threat model that captures the reality of intimate partner abuse: attackers are insiders, not hackers, and harm comes from what they do on the phone as much as who is holding it.

Building on this foundation, AID introduces a dual-branch AI architecture that continuously and silently monitors smartphones. One branch determines whether the current user is the device owner or not, while the other infers whether their actions reflect harmful IPI intent. Only when both conditions are met does the system flag risk—dramatically reducing false alarms that could otherwise retraumatize victims or escalate abuse. Crucially, AID runs entirely on-device, relies only on background-accessible signals, and is designed to remain invisible to abusers.

We demonstrated our work “EmbodiedRDA: Connecting Foundation Models with the Physical World using Reconfigurable Drone Agents” and was awarded the Best Demo Runner-Up award at the 30th Annual International Conference on Mobile Computing and Networking (ACM MobiCom 2024) in Washington, DC.

EmbodiedRDA connects foundation models with the physical world through a custom-designed drone platform that autonomously reconfigures itself with different sensors and actuators depending on the task at hand. The system adapts to various tasks in physical environments – from locating objects to environmental monitoring and physical assistance.

Congratulations to Junxi Xia, Minghui Zhao, Kaiyuan Hou, and the rest of the team!

“TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms” was accepted to Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) and will be presented at UbiComp / ISWC 2025.

In this work, we present a novel architecture for enhancing speech on mobile and wearable platforms. This work bridges the gap speech enhancement quality from computationally heavy techniques (e.g., GANs) and the compute efficiency of lightweight techniques (e.g., U-Net). Congrats to Yueyuan Sui and the rest of the team!