# ScreenAI: A visual language model for UI and visually-situated language understanding

DevFeed: [ScreenAI: A visual language model for UI and visually-situated language understanding](<https://devfeed.tech/articles/screenai-a-visual-language-model-for-ui-and-visually-situated-language-understanding-28563.md>)

Original publisher: [Read original article](<http://blog.research.google/2024/03/screenai-visual-language-model-for-ui.html>)

Author: Google AI (noreply@blogger.com)

Published: 2024-03-19T20:15:00Z

Content type: release

Language: en

Sources: [Google Research](<https://devfeed.tech/sources/google-research.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [ui](<https://devfeed.tech/topics/ui.md>), [Google](<https://devfeed.tech/topics/google.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [multimodal](<https://devfeed.tech/topics/multimodal.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [datasets](<https://devfeed.tech/tags/datasets.md>), [google](<https://devfeed.tech/tags/google.md>), [hci](<https://devfeed.tech/tags/hci.md>), [model](<https://devfeed.tech/tags/model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [multimodal-learning](<https://devfeed.tech/tags/multimodal-learning.md>), [self-supervised-learning](<https://devfeed.tech/tags/self-supervised-learning.md>), [ui](<https://devfeed.tech/tags/ui.md>), [vision](<https://devfeed.tech/tags/vision.md>)

## AI overview

Google Research introduces ScreenAI, a vision-language model designed to understand, reason about, and interact with user interfaces and infographics. The article describes its PaLI-based multimodal architecture, pix2struct-inspired flexible patching strategy, training approach, reported benchmark results, and three released datasets for screen understanding and question answering.

## Source excerpt

Posted by Srinivas Sunkara and Gilles Baechler, Software Engineers, Google Research Screen user interfaces (UIs) and infographics, such as charts, diagrams and tables, play important roles in human communication and human-machine interaction as they facilitate rich and interactive user experiences. UIs and infographics share similar design principles and visual language (e.g., icons and layouts), that offer an opportunity to build a single model that can understand, reason, and interact with these interfaces. However, because of their complexity and varied presentation formats, infographics and UIs present a unique modeling challenge. To that end, we introduce "ScreenAI: A Vision-Language Model for UI and Infographics Understanding". ScreenAI improves upon the PaLI architecture with the flexible patching strategy from pix2struct. We train ScreenAI on a unique mixture of datasets and tasks, including a novel Screen Annotation task that requires the model to identify UI element information (i.e., type, location and description) on a screen. These text annotations provide large language models (LLMs) with screen descriptions, enabling them to automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. At only 5B parameters, ScreenAI achieves state-of-the-art results on UI- and infographic-based tasks (WebSRC and MoTIF), and best-in-class performance on Chart QA, DocVQA, and InfographicVQA compared to models of similar size. We are also releasing three new datasets: Screen Annotation to evaluate the layout understanding capability of the model, as well as ScreenQA Short and Complex ScreenQA for a more comprehensive evaluation of its QA capability. ScreenAI ScreenAI's architecture is based on PaLI, composed of a multimodal encoder block and an autoregressive decoder. The PaLI encoder uses a vision transformer (ViT) that creates image embeddings and a multimodal encoder that takes the concatenation of the image and text emb