Penetrative AI: Making LLMs Comprehend the Physical World

Anonymous

Abstract: Recent developments in Large Language Models (LLMs) have demonstrated their remarkable capabilities across a range of tasks. Questions, however, persist about the nature of LLMs and their potential to integrate common-sense human knowledge when performing tasks involving information about the real physical world. This paper delves into these questions by exploring how LLMs can be extended to interact with and reason about the physical world through IoT sensors and actuators, a concept that we term "Penetrative AI". The paper explores such an extension at two levels of LLMs' ability to penetrate into the physical world via the processing of sensory signals. Our preliminary findings indicate that LLMs, with ChatGPT being the representative example in our exploration, have considerable and unique proficiency in employing the embedded world knowledge for interpreting IoT sensor data and reasoning over them about tasks in the physical realm. Not only this opens up new applications for LLMs beyond traditional text-based tasks, but also enables new ways of incorporating human knowledge in cyber-physical systems.

Paper Type: long

Research Area: NLP Applications

Contribution Types: Position papers

Languages Studied: English

Preprint Status: There is a non-anonymous preprint (URL specified in the next question).

A1: yes

A1 Elaboration For Yes Or No: Section 7

A2: yes

A2 Elaboration For Yes Or No: Section 8

A3: yes

A3 Elaboration For Yes Or No: Abstract and Section 1

B: no

B1: n/a

B2: n/a

B3: n/a

B4: n/a

B5: n/a

B6: n/a

C: yes

C1: yes

C1 Elaboration For Yes Or No: Section 2 and Section 3

C2: yes

C2 Elaboration For Yes Or No: Section 2 and Section 3

C3: yes

C3 Elaboration For Yes Or No: Section 2 and Section 3

C4: yes

C4 Elaboration For Yes Or No: Section 3. We adopt ECG Detectors(https://github.com/berndporr/py-ecg-detectors) for baselines.

D: yes

D1: yes

D1 Elaboration For Yes Or No: Section 8

D2: yes

D2 Elaboration For Yes Or No: Section 8

D3: yes

D3 Elaboration For Yes Or No: Section 8

D4: yes

D4 Elaboration For Yes Or No: Section 8

D5: yes

D5 Elaboration For Yes Or No: Section 8

E: yes

E1: yes

E1 Elaboration For Yes Or No: Section 2 and Section 3