Screen
as Shared Context
HCI Research • Ameer Omidvar
Definition
AI can perceive the same environment the user does. The screen becomes a shared field of attention, allowing AI to understand what’s visible without manual description.
Purpose
To eliminate friction in interaction by grounding AI in shared perception rather than text-based explanation.
Design Guidance
Integrate at the system level so AI can see and reference on-screen elements.
Make context-sharing transparent and user-controlled.
Allow voice or gesture-based invocation within any app.
Maintain continuity across actions—AI should “remember” what’s visible.
Prioritize local awareness over data collection; visibility ≠ surveillance.
Example
A user highlights a UI element and asks, “What does this do?” The AI answers instantly, understanding the context without switching apps or sending screenshots.