Prompting Realities

Extending Prompt Engineering into Physical Realities

The application of LLMs in our everyday life has gained momentum since the ChatGPT breakthrough in November 2022. While much attention has been focused on ways to engineer prompts for specific textual results, there is little emphasis on how such iterative prompt engineering techniques can be extended beyond the limitations of screens and into more tangible, physical realities. This shift can empower users, expanding the familiar concept of “end-user programming” to include any type of everyday artifact. Consequently, Prompting Realities envisions an experiential scenario where AI distributes the authoritative role of engineers and designers as creators to end-users, opening up new opportunities to contest, interrupt, resist, or manipulate everyday products under unfair situations—not through a system, but in the edge of usage.

A Tangible Glimpse at the Future Everyday Interactions

The underlying pipeline operates with a simple yet powerful approach: providing a precise description of the prototype (functionality, appearance, etc.) to the large language model. This enables the model to understand the correspondence between computer variables and the effects they can have on reality, in relation to the prototype's functionality. There is little emphasis on specific hardware or software solutions used in the current prototypes, as the project primarily focuses on offering a new experience for users rather than introducing a new AI-powered system. However, the prototypes utilize a range of technologies, including the OpenAI Assistant API, Telegram Bot Interface, and TU Delft IDE Connected Interaction Kit, all of which can be replaced by similar alternatives.

Redistributing Agency: AI's Role in Democratizing Design

Prompting Realities explores the intersection of AI and physical computing, expanding the application of large language models beyond text-based interactions into tangible, real-world contexts. By empowering users to interact with AI-driven prototypes through conversational interfaces, the project pushes classic notions of end-user programming, redistributing control and agency from engineers and designers to everyday users. This shift has the potential to democratize technology enabling resistance, interruption, and subversion opening new avenues for human-AI collaboration. Through experiential AI prototyping, the project provokes speculative reflection on AI’s role in shaping our physical and digital environments, encouraging deeper engagement with the evolving relationship between humans and machines.