Summary of Leveraging Foundation Models For Zero-shot Iot Sensing, by Dinghao Xue et al.
Leveraging Foundation Models for Zero-Shot IoT Sensing
by Dinghao Xue, Xiaoran Fan, Tao Chen, Guohao Lan, Qun Song
First submitted to arxiv on: 29 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Human-Computer Interaction (cs.HC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Deep learning models are increasingly deployed on edge Internet of Things (IoT) devices, which typically operate under supervised conditions and fail to recognize unseen classes different from training. To address this, zero-shot learning (ZSL) aims to classify data of unseen classes with the help of semantic information. Foundation models (FMs) trained on web-scale data have shown impressive ZSL capability in natural language processing and visual understanding. However, leveraging FMs’ generalized knowledge for zero-shot IoT sensing using signals such as mmWave, IMU, and Wi-Fi has not been fully investigated. The proposed approach aligns IoT data embeddings with semantic embeddings generated by an FM’s text encoder for zero-shot IoT sensing. To optimize prompts for semantic embedding extraction, cross-attention is used to combine a learnable soft prompt optimized on training data and an auxiliary hard prompt encoding domain knowledge of the IoT sensing task. To address the problem of IoT embeddings biasing to seen classes due to the lack of unseen class data during training, data augmentation is proposed to synthesize unseen class IoT data for fine-tuning the IoT feature extractor and embedding projector. The approach is evaluated on multiple IoT sensing tasks, achieving superior open-set detection and generalized zero-shot learning performance compared with various baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper talks about how to use special kinds of artificial intelligence (AI) models called “foundation models” to recognize new types of data that a computer has never seen before. These models are trained on huge amounts of text data and can understand language, but they’ve never been used for recognizing sensor signals from things like phones or smart home devices. The researchers propose a new way to use these models to recognize unseen types of sensor data using information about the physical world that governs how those sensors work. They also suggest generating fake data to help the model learn to recognize new types of data better. The approach is tested on different tasks and shows it can do better than other methods at recognizing new types of data. |
Keywords
» Artificial intelligence » Cross attention » Data augmentation » Deep learning » Embedding » Encoder » Fine tuning » Natural language processing » Prompt » Supervised » Zero shot
