Projects

My research focuses on four interconnected topics spanning from the individual level to the system level. Specifically, I work on: 1) AI-based mental and physical health care in smart home environments; 2) smart wearables for emotion and wellness; 3) indoor intelligent sensor network; and 4) city-scale human-in-the-loop EV-interfaced

AI-based Mental Health and Physical Health Care in Smart Home Environments

Conversational AI Therapist

The growth of smart devices is making typical homes more intelligent. In this work, in collaboration with licensed therapists from the two largest mental health counseling institutions in the area, we introduce a home-based AI therapist that takes advantage of the smart home environment to screen the day-to-day functioning and infer mental wellness of an occupant. Unlike existing “chatbot” works that identify the mental status of users through conversation, our AI therapist additionally leverages smart devices and sensors throughout the home to infer mental well-being and assesses a user’s daily functioning. We propose a series of 37 dimensions of daily functioning, that our system observes through conversing with the user and detecting daily activity events using sensors and smart sensors throughout the home. Our system utilizes these 37 dimensions in conjunction with novel natural language processing architectures to detect abnormalities in mental status (e.g., angry or depressed), well-being, and daily functioning and generate responses to console users when abnormalities are detected. Through a series of user studies, we demonstrate that our system can converse with a user naturally, accurately detect abnormalities in well-being, and provide appropriate responses consoling users. (paper)

AI-based Mental Status Examination

There is a lack of automated tools that utilize artificial intelligence to monitor mental health. The mental status examination (MSE) is an important tool used by mental health providers for assessing mental health. Currently, MSEs are conducted by licensed professionals, which is a barrier for patients in low income and remote areas. We propose an AI-based personal online mental status examination (aiMSE), the first interactive MSE platform. Users can use aiMSE to self-administer MSEs at home through a web browser, using only a camera and microphone. aiMSE uses multimodal image, speech, and natural language processing algorithms to detect signs of abnormalities in mental functioning and recommend them for further examination by a mental health specialist. We conducted a 14-person study, which supports the feasibility of detecting a wide range of signs commonly found in patients with changes in mental or cognitive capacity. (paper)

Smart wearables for Emotion and Wellness

SPIDERS: Wearable Emotion Sensing

SPIDERS (System  for  Processing  In-situ  Bio-signal  Data  for  Emotion  Recognition  and  Sensing) is low-cost, lightweight and compact wearable platform that can monitor human emotions will benefit a wide area of research and applications, such as continuous health monitoring, elderly care, depression treatment, entertainment, so on and so forth. Thus, we present SPIDERS which is a low-cost, wireless, glasses-based platform for continuous in-situ monitoring of user’s facial expressions and real emotions. SPIDERS costs less than $20 to assemble and can continuously run for up to 9 hours before recharging. (Read More) (paper 1, paper 2)

An Intelligent Audio Wearable Platform for Improving Construction Worker Safety in Urban Environments

Vehicle accidents are one of the greatest cause of death and injury in urban areas for pedestrians, workers, and police alike. In this work, we present CSafe, a low power audio-wearable platform that detects, localizes, and provides alerts about oncoming vehicles to improve construction worker safety. Construction worker safety is a much more challenging problem than general urban or pedestrian safety in that the sound of construction tools can be up to orders of magnitude greater than that of vehicles, making vehicle detection and localization exceptionally difficult. To overcome these challenges, we develop a novel sound source separation algorithm, called Probabilistic Template Matching (PTM), as well as a novel noise filtering architecture to remove loud construction noises from our observed signals. We show that our architecture can improve vehicle detection by up to 12% over other state-of-art source separation algorithms. We integrate PTM and our noise filtering architecture into CSafe and show through a series of real-world experiments that CSafe can achieve up to an 82% vehicle detection rate and a 6.90° mean localization error in acoustically noisy construction site scenarios, which is 16% higher and almost 30° lower than the state-of-art audio wearable safety works. (paper)

Indoor Intelligent Sensor Network

Self-Orienting Camera Network for Floor Mapping and Indoor Tracking

Localization and tracking are important in many applications, including in the current COVID-19 pandemic to ensure that social distancing requirements are being met. We present SoFIT, an easily-deployed and privacy-preserving camera network system for occupant tracking. Unlike traditional camera network-based systems, SoFIT does not require a person to calibrate the network or provide real-world references. This enables anyone, including non-professionals, to install SoFIT. Once installed, SoFIT automatically localizes cameras within the network and generates the floor map leveraging movements of people using the space in daily life, before using the floor map and camera locations to track occupants throughout the environment. We demonstrate through a series of deployments that SoFIT can localize cameras with less than 4.8cm error, generate floor maps with 85% similarity to actual floor maps, and track occupants with less than 7.8cm error. (paper)

Plug-and-Play and Mix-and-Match Sensor Platform

A platform that has high configurability, high scalability, and is friendly for both nonprofessionals and professionals is necessary to realize various applications that I envision using IoT systems. We are developing and deploying a Raspberry Pi-based hardware and software platform that enables the fast and easy deployment of sensing systems.

City-Scale Human-in-the-Loop EV-Interfaced Grid Optimizations

Optimal Power Flow Estimation of Microgrid Considering the Grid Services of EV Batteries

Demand for the grid state estimation with partial power network observation is growing rapidly with the increasing amount of distributed energy resources (DER) connected to the grid with incomplete measured information. The grid services that could be provided by these DER, such as electrical vehicles (EVs), have the potential to affect the resilience and efficiency of the grid. We propose a constrained optimization solver based on the AC power flows to recover the incomplete information of the grid. Further reactive power constraints following the specifications in IEEE 1547 are added to the solver to explore the effects of adding grid services to the steady-state microgrid. This work won the Best Student Paper Award in ITEC2021. (paper)

We further evaluate the performance of the proposed solver using the IEEE 9/30/57/118-bus systems and show that it excels the state-of-the-art approaches ( pandapower and MATPOWER) in recovering the grid state from partial observations with up to six orders of magnitude smaller mean absolute error. The proposed solver has better scalability to deal with larger systems and with systems with more unknown measurements. Further reactive power constraints are added to the solver following the specifications in IEEE 1547 to explore the effects of adding grid services to the steady-state microgrid. We demonstrate and discuss novel case studies on these four IEEE bus systems. And the reactions of the IEEE bus systems when following the grid services’ constraints are introduced. We illustrate a sensitivity analysis that reflects how the system’s physical characteristics would impact the implementation of grid services. (paper)

Deep Reinforcement Learning Based Approach for Optimal Power Flow of Microgrid with Grid Services Implementation

Electric vehicles (EVs) have rapidly grown in popularity, and the number of inverter-based EV chargers increases promptly due to their high efficiency and capabilities of providing grid services. EV and other distributed energy resources (DER) would become a crucial part of the resilience and performance of the microgrid. Optimizing the EV-interfaced microgrid is challenging due to the nonlinearity and uncertainty. In this paper, we propose a method based on deep reinforcement learning (DRL) with Twin Delayed Deep Deterministic Policy Gradients (TD3) to optimize the microgrid. The proposed method can be used to optimize different objectives. An example objective of stabilizing the voltage fluctuations in a power system modified from the IEEE 30-bus system is presented. The proposed system can provide grid service policies for reactive power control according to the requirements specified in the IEEE 1547 standard. This model-free DRL approach can be adapted to other microgrid systems. (paper)

Naturalistic Speech Perception in Bilinguals

Under the direction of Nima Mesgarani and in collaboration with Giovanni Di Liberto , we are investigating how modulated semantic surprisal can be observed in the EEG signals of native and non-native speakers (native English: 22 subjects; native Chinese speakers with different proficiency in English: 50 subjects) listening to continuous, natural speech. The main methodology we use is the multivariate Temporal Response Function(mTRF) which allows us to compare an EEG signal to a continuous speech stream, instead of brief discretized events as in ERPs. Specifically, we apply mTRF to quantify the coupling between EEG signals from each participant and the corresponding speech stimulus properties at the level of acoustics, phonemes, and semantics. We are also looking at whether these and other measures can be used to decode proficiency from an EEG signal. (paper)