# Google Research The latest news from Google AI. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Generative AI to quantify uncertainty in weather forecasting DevFeed: [Generative AI to quantify uncertainty in weather forecasting]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-29T18:03:00Z Content type: release Language: en Sources: [Google Research]() Topics: [Generative AI](), [Artificial Intelligence](), [Google](), [Simulation]() Tags: [ai](), [climate](), [diffusion](), [generative-ai](), [machine-learning](), [research](), [science](), [weather]() ### AI overview Google Research introduces SEEDS, a generative AI model designed to efficiently generate large ensembles of weather forecasts for quantifying uncertainty. The article explains that SEEDS aims to reduce the computational cost of traditional physics-based ensemble forecasting and describes its application to weather and climate science. ### Source excerpt Posted by Lizao (Larry) Li, Software Engineer, and Rob Carver, Research Scientist, Google Research Accurate weather forecasts can have a direct impact on people's lives, from helping make routine decisions, like what to pack for a day's activities, to informing urgent actions, for example, protecting people in the face of hazardous weather conditions. The importance of accurate and timely weather forecasts will only increase as the climate changes. Recognizing this, we at Google have been investing in weather and climate research to help ensure that the forecasting technology of tomorrow can meet the demand for reliable weather information. Some of our recent innovations include MetNet-3, Google's high-resolution forecasts up to 24-hours into the future, and GraphCast, a weather model that can predict weather up to 10 days ahead. Weather is inherently stochastic. To quantify the uncertainty, traditional methods rely on physics-based simulation to generate an ensemble of forecasts. However, it is computationally costly to generate a large ensemble so that rare and extreme weather events can be discerned and characterized accurately. With that in mind, we are excited to announce our latest innovation designed to accelerate progress in weather forecasting, Scalable Ensemble Envelope Diffusion Sampler (SEEDS), recently published in Science Advances. SEEDS is a generative AI model that can efficiently generate ensembles of weather forecasts at scale at a small fraction of the cost of traditional physics-based forecasting models. This technology opens up novel opportunities for weather and climate science, and it represents one of the first applications to weather and climate forecasting of probabilistic diffusion models, a generative AI technology behind recent advances in media generation. The need for probabilistic forecasts: the butterfly effect In December 1972, at the American Association for the Advancement of Science meeting in Washington, D.C., MIT meteorology pr ## AutoBNN: Probabilistic time series forecasting with compositional bayesian neural networks DevFeed: [AutoBNN: Probabilistic time series forecasting with compositional bayesian neural networks]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-28T20:53:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Time Series](), [Google](), [Open Source]() Tags: [data](), [forecasting](), [google](), [kernels](), [machine-learning](), [model](), [neural-networks](), [open-source](), [patterns](), [research](), [statistics](), [time-series]() ### AI overview Google Research introduces AutoBNN, an open-source JAX package for automatically discovering interpretable probabilistic time series forecasting models. It combines learned compositional Gaussian process kernels with neural-network scalability and provides uncertainty estimates for large datasets. ### Source excerpt Posted by Urs Köster, Software Engineer, Google Research Time series problems are ubiquitous, from forecasting weather and traffic patterns to understanding economic trends. Bayesian approaches start with an assumption about the data's patterns (prior probability), collecting evidence (e.g., new time series data), and continuously updating that assumption to form a posterior probability distribution. Traditional Bayesian approaches like Gaussian processes (GPs) and Structural Time Series are extensively used for modeling time series data, e.g., the commonly used Mauna Loa CO2 dataset. However, they often rely on domain experts to painstakingly select appropriate model components and may be computationally expensive. Alternatives such as neural networks lack interpretability, making it difficult to understand how they generate forecasts, and don't produce reliable confidence intervals. To that end, we introduce AutoBNN, a new open-source package written in JAX. AutoBNN automates the discovery of interpretable time series forecasting models, provides high-quality uncertainty estimates, and scales effectively for use on large datasets. We describe how AutoBNN combines the interpretability of traditional probabilistic approaches with the scalability and flexibility of neural networks. AutoBNN AutoBNN is based on a line of research that over the past decade has yielded improved predictive accuracy by modeling time series using GPs with learned kernel structures. The kernel function of a GP encodes assumptions about the function being modeled, such as the presence of trends, periodicity or noise. With learned GP kernels, the kernel function is defined compositionally: it is either a base kernel (such as Linear, Quadratic, Periodic, Matérn or ExponentiatedQuadratic) or a composite that combines two or more kernel functions using operators such as Addition, Multiplication, or ChangePoint. This compositional kernel structure serves two related purposes. First, it is simple e ## Computer-aided diagnosis for lung cancer screening DevFeed: [Computer-aided diagnosis for lung cancer screening]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-20T20:54:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Machine learning](), [Google](), [Artificial Intelligence](), [Image]() Tags: [false-positives](), [google](), [health](), [healthcare](), [japan](), [machine-learning](), [reporting](), [user-experience]() ### AI overview Google Research describes a machine-learning system for lung cancer screening that analyzes CT scans, assigns one of four cancer-suspicion ratings, and highlights corresponding regions of interest. The system was evaluated in randomized reader studies in the US and Japan using local scoring systems and realistic image viewers. ### Source excerpt Posted by Atilla Kiraly, Software Engineer, and Rory Pilgrim, Product Manager, Google Research Lung cancer is the leading cause of cancer-related deaths globally with 1.8 million deaths reported in 2020. Late diagnosis dramatically reduces the chances of survival. Lung cancer screening via computed tomography (CT), which provides a detailed 3D image of the lungs, has been shown to reduce mortality in high-risk populations by at least 20% by detecting potential signs of cancers earlier. In the US, screening involves annual scans, with some countries or cases recommending more or less frequent scans. The United States Preventive Services Task Force recently expanded lung cancer screening recommendations by roughly 80%, which is expected to increase screening access for women and racial and ethnic minority groups. However, false positives (i.e., incorrectly reporting a potential cancer in a cancer-free patient) can cause anxiety and lead to unnecessary procedures for patients while increasing costs for the healthcare system. Moreover, efficiency in screening a large number of individuals can be challenging depending on healthcare infrastructure and radiologist availability. At Google we have previously developed machine learning (ML) models for lung cancer detection, and have evaluated their ability to automatically detect and classify regions that show signs of potential cancer. Performance has been shown to be comparable to that of specialists in detecting possible cancer. While they have achieved high performance, effectively communicating findings in realistic environments is necessary to realize their full potential. To that end, in "Assistive AI in Lung Cancer Screening: A Retrospective Multinational Study in the US and Japan", published in Radiology AI, we investigate how ML models can effectively communicate findings to radiologists. We also introduce a generalizable user-centric interface to help radiologists leverage such models for lung cancer screening. The ## Using AI to expand global access to reliable flood forecasts DevFeed: [Using AI to expand global access to reliable flood forecasts]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-20T16:06:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Artificial Intelligence](), [Google](), [real-time](), [data](), [notifications]() Tags: [ai](), [data](), [environment](), [google](), [machine-learning](), [notifications](), [real-time](), [research](), [weather]() ### AI overview Google Research describes how AI and machine learning improved global flood forecasting in regions with scarce flood-related data. The work extended the average reliability of global nowcasts from zero to five days and supports real-time river forecasts up to seven days ahead across more than 80 countries. ### Source excerpt Posted by Yossi Matias, VP Engineering & Research, and Grey Nearing, Research Scientist, Google Research Floods are the most common natural disaster, and are responsible for roughly $50 billion in annual financial damages worldwide. The rate of flood-related disasters has more than doubled since the year 2000 partly due to climate change. Nearly 1.5 billion people, making up 19% of the world's population, are exposed to substantial risks from severe flood events. Upgrading early warning systems to make accurate and timely information accessible to these populations can save thousands of lives per year. Driven by the potential impact of reliable flood forecasting on people's lives globally, we started our flood forecasting effort in 2017. Through this multi-year journey, we advanced research over the years hand-in-hand with building a real-time operational flood forecasting system that provides alerts on Google Search, Maps, Android notifications and through the Flood Hub. However, in order to scale globally, especially in places where accurate local data is not available, more research advances were required. In "Global prediction of extreme floods in ungauged watersheds", published in Nature, we demonstrate how machine learning (ML) technologies can significantly improve global-scale flood forecasting relative to the current state-of-the-art for countries where flood-related data is scarce. With these AI-based technologies we extended the reliability of currently-available global nowcasts, on average, from zero to five days, and improved forecasts across regions in Africa and Asia to be similar to what are currently available in Europe. The evaluation of the models was conducted in collaboration with the European Center for Medium Range Weather Forecasting (ECMWF). These technologies also enable Flood Hub to provide real-time river forecasts up to seven days in advance, covering river reaches across over 80 countries. This information can be used by people, communi ## ScreenAI: A visual language model for UI and visually-situated language understanding DevFeed: [ScreenAI: A visual language model for UI and visually-situated language understanding]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-19T20:15:00Z Content type: release Language: en Sources: [Google Research]() Topics: [vlm](), [ui](), [Google](), [datasets](), [multimodal](), [Transformer]() Tags: [datasets](), [google](), [hci](), [model](), [multimodal](), [multimodal-learning](), [self-supervised-learning](), [ui](), [vision]() ### AI overview Google Research introduces ScreenAI, a vision-language model designed to understand, reason about, and interact with user interfaces and infographics. The article describes its PaLI-based multimodal architecture, pix2struct-inspired flexible patching strategy, training approach, reported benchmark results, and three released datasets for screen understanding and question answering. ### Source excerpt Posted by Srinivas Sunkara and Gilles Baechler, Software Engineers, Google Research Screen user interfaces (UIs) and infographics, such as charts, diagrams and tables, play important roles in human communication and human-machine interaction as they facilitate rich and interactive user experiences. UIs and infographics share similar design principles and visual language (e.g., icons and layouts), that offer an opportunity to build a single model that can understand, reason, and interact with these interfaces. However, because of their complexity and varied presentation formats, infographics and UIs present a unique modeling challenge. To that end, we introduce "ScreenAI: A Vision-Language Model for UI and Infographics Understanding". ScreenAI improves upon the PaLI architecture with the flexible patching strategy from pix2struct. We train ScreenAI on a unique mixture of datasets and tasks, including a novel Screen Annotation task that requires the model to identify UI element information (i.e., type, location and description) on a screen. These text annotations provide large language models (LLMs) with screen descriptions, enabling them to automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. At only 5B parameters, ScreenAI achieves state-of-the-art results on UI- and infographic-based tasks (WebSRC and MoTIF), and best-in-class performance on Chart QA, DocVQA, and InfographicVQA compared to models of similar size. We are also releasing three new datasets: Screen Annotation to evaluate the layout understanding capability of the model, as well as ScreenQA Short and Complex ScreenQA for a more comprehensive evaluation of its QA capability. ScreenAI ScreenAI's architecture is based on PaLI, composed of a multimodal encoder block and an autoregressive decoder. The PaLI encoder uses a vision transformer (ViT) that creates image embeddings and a multimodal encoder that takes the concatenation of the image and text emb ## SCIN: A new resource for representative dermatology images DevFeed: [SCIN: A new resource for representative dermatology images]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-19T15:00:00Z Content type: article Language: en Sources: [Google Research]() Topics: [datasets](), [Artificial Intelligence](), [Google]() Tags: [ai](), [consent](), [crowd-sourcing](), [dataset](), [datasets](), [developers](), [diversity](), [google](), [health](), [open](), [privacy](), [research]() ### AI overview Google Research Health is releasing the Skin Condition Image Network (SCIN), an open-access dermatology image dataset created with Stanford Medicine physicians. It contains over 10,000 voluntarily contributed images covering skin, nail, and hair conditions across various skin tones and body parts, with retrospective dermatologist labels and contributor-provided metadata. ### Source excerpt Posted by Pooja Rao, Research Scientist, Google Research Health datasets play a crucial role in research and medical education, but it can be challenging to create a dataset that represents the real world. For example, dermatology conditions are diverse in their appearance and severity and manifest differently across skin tones. Yet, existing dermatology image datasets often lack representation of everyday conditions (like rashes, allergies and infections) and skew towards lighter skin tones. Furthermore, race and ethnicity information is frequently missing, hindering our ability to assess disparities or create solutions. To address these limitations, we are releasing the Skin Condition Image Network (SCIN) dataset in collaboration with physicians at Stanford Medicine. We designed SCIN to reflect the broad range of concerns that people search for online, supplementing the types of conditions typically found in clinical datasets. It contains images across various skin tones and body parts, helping to ensure that future AI tools work effectively for all. We've made the SCIN dataset freely available as an open-access resource for researchers, educators, and developers, and have taken careful steps to protect contributor privacy. Example set of images and metadata from the SCIN dataset. Dataset composition The SCIN dataset currently contains over 10,000 images of skin, nail, or hair conditions, directly contributed by individuals experiencing them. All contributions were made voluntarily with informed consent by individuals in the US, under an institutional-review board approved study. To provide context for retrospective dermatologist labeling, contributors were asked to take images both close-up and from slightly further away. They were given the option to self-report demographic information and tanning propensity (self-reported Fitzpatrick Skin Type, i.e., sFST), and to describe the texture, duration and symptoms related to their concern. One to three dermatologists ## MELON: Reconstructing 3D objects from images with unknown poses DevFeed: [MELON: Reconstructing 3D objects from images with unknown poses]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-18T18:41:00Z Content type: article Language: en Sources: [Google Research]() Topics: [3D](), [Computer vision](), [Inference](), [Google]() Tags: [3d](), [computer-vision](), [gans](), [gaussian-splatting](), [generative](), [google](), [images](), [inference](), [machine-learning](), [models](), [neural](), [research](), [rgb](), [rotation](), [techniques](), [vision]() ### AI overview This Google Research article explains the challenge of reconstructing 3D objects from a small number of images when the camera poses are unknown. It covers pose inference, pseudo-symmetries, local-minimum failures, and prior approaches including NeRF, 3D Gaussian Splatting, GAN-based methods, BARF, SAMURAI, GNeRF, VMRF, SparsePose, and RUST. ### Source excerpt Posted by Mark Matthews, Senior Software Engineer, and Dmitry Lagun, Research Scientist, Google Research A person's prior experience and understanding of the world generally enables them to easily infer what an object looks like in whole, even if only looking at a few 2D pictures of it. Yet the capacity for a computer to reconstruct the shape of an object in 3D given only a few images has remained a difficult algorithmic problem for years. This fundamental computer vision task has applications ranging from the creation of e-commerce 3D models to autonomous vehicle navigation. A key part of the problem is how to determine the exact positions from which images were taken, known as pose inference. If camera poses are known, a range of successful techniques -- such as neural radiance fields (NeRF) or 3D Gaussian Splatting -- can reconstruct an object in 3D. But if these poses are not available, then we face a difficult "chicken and egg" problem where we could determine the poses if we knew the 3D object, but we can't reconstruct the 3D object until we know the camera poses. The problem is made harder by pseudo-symmetries -- i.e., many objects look similar when viewed from different angles. For example, square objects like a chair tend to look similar every 90° rotation. Pseudo-symmetries of an object can be revealed by rendering it on a turntable from various angles and plotting its photometric self-similarity map. Self-Similarity map of a toy truck model. Left: The model is rendered on a turntable from various azimuthal angles, θ. Right: The average L2 RGB similarity of a rendering from θ with that of θ*. The pseudo-similarities are indicated by the dashed red lines. The diagram above only visualizes one dimension of rotation. It becomes even more complex (and difficult to visualize) when introducing more degrees of freedom. Pseudo-symmetries make the problem ill-posed, with naïve approaches often converging to local minima. In practice, such an approach might mistake the ## HEAL: A framework for health equity assessment of machine learning performance DevFeed: [HEAL: A framework for health equity assessment of machine learning performance]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-15T18:22:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Machine learning](), [Artificial Intelligence](), [Google](), [AI Research]() Tags: [ai](), [fairness](), [google](), [health](), [machine-learning](), [research](), [responsible-ai]() ### AI overview Google Research presents HEAL, a framework for quantitatively assessing whether machine-learning health technologies perform equitably. The framework evaluates model performance in relation to pre-existing health disparities and focuses on populations with the worst health outcomes, illustrated through a dermatology AI model case study. ### Source excerpt Posted by Mike Schaekermann, Research Scientist, Google Research, and Ivor Horn, Chief Health Equity Officer & Director, Google Core Health equity is a major societal concern worldwide with disparities having many causes. These sources include limitations in access to healthcare, differences in clinical treatment, and even fundamental differences in the diagnostic technology. In dermatology for example, skin cancer outcomes are worse for populations such as minorities, those with lower socioeconomic status, or individuals with limited healthcare access. While there is great promise in recent advances in machine learning (ML) and artificial intelligence (AI) to help improve healthcare, this transition from research to bedside must be accompanied by a careful understanding of whether and how they impact health equity. Health equity is defined by public health organizations as fairness of opportunity for everyone to be as healthy as possible. Importantly, equity may be different from equality. For example, people with greater barriers to improving their health may require more or different effort to experience this fair opportunity. Similarly, equity is not fairness as defined in the AI for healthcare literature. Whereas AI fairness often strives for equal performance of the AI technology across different patient populations, this does not center the goal of prioritizing performance with respect to pre-existing health disparities. Health equity considerations. An intervention (e.g., an ML-based tool, indicated in dark blue) promotes health equity if it helps reduce existing disparities in health outcomes (indicated in lighter blue). In "Health Equity Assessment of machine Learning performance (HEAL): a framework and dermatology AI model case study", published in The Lancet eClinicalMedicine, we propose a methodology to quantitatively assess whether ML-based health technologies perform equitably. In other words, does the ML model perform well for those with the worst he ## Cappy: Outperforming and boosting large multi-task language models with a small scorer DevFeed: [Cappy: Outperforming and boosting large multi-task language models with a small scorer]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-14T19:38:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Large Language Model](), [Natural language processing](), [data](), [Google]() Tags: [google](), [language](), [language-models](), [large-language-models](), [machine-learning](), [model](), [models](), [natural-language-processing](), [neurips](), [software](), [training-data]() ### AI overview This Google Research article introduces the motivation for Cappy, a small scorer intended to help adapt and improve large multi-task language models. It describes how instruction-following LLMs generalize across tasks, while noting that their size creates substantial computational, memory, storage, and adaptation challenges. ### Source excerpt Posted by Yun Zhu and Lijuan Liu, Software Engineers, Google Research Large language model (LLM) advancements have led to a new paradigm that unifies various natural language processing (NLP) tasks within an instruction-following framework. This paradigm is exemplified by recent multi-task LLMs, such as T0, FLAN, and OPT-IML. First, multi-task data is gathered with each task following a task-specific template, where each labeled example is converted into an instruction (e.g., "Put the concepts together to form a sentence: ski, mountain, skier") paired with a corresponding response (e.g., "Skier skis down the mountain"). These instruction-response pairs are used to train the LLM, resulting in a conditional generation model that takes an instruction as input and generates a response. Moreover, multi-task LLMs have exhibited remarkable task-wise generalization capabilities as they can address unseen tasks by understanding and solving brand-new instructions. The demonstration of the instruction-following pre-training of multi-task LLMs, e.g., FLAN. Pre-training tasks under this paradigm improves the performance for unseen tasks. Due to the complexity of understanding and solving various tasks solely using instructions, the size of multi-task LLMs typically spans from several billion parameters to hundreds of billions (e.g., FLAN-11B, T0-11B and OPT-IML-175B). As a result, operating such sizable models poses significant challenges because they demand considerable computational power and impose substantial requirements on the memory capacities of GPUs and TPUs, making their training and inference expensive and inefficient. Extensive storage is required to maintain a unique LLM copy for each downstream task. Moreover, the most powerful multi-task LLMs (e.g., FLAN-PaLM-540B) are closed-sourced, making them impossible to be adapted. However, in practical applications, harnessing a single multi-task LLM to manage all conceivable tasks in a zero-shot manner remains difficult, ## Talk like a graph: Encoding graphs for large language models DevFeed: [Talk like a graph: Encoding graphs for large language models]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-12T21:15:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Graphs](), [Large Language Model](), [Benchmark](), [Google](), [Computer science]() Tags: [benchmark](), [generative-ai](), [google](), [graphs](), [iclr](), [language-models](), [large-language-models](), [reasoning](), [research]() ### AI overview Google Research scientists describe a study of how to encode graphs as text so large language models can reason about graph information. The work introduces the GraphQA benchmark and evaluates how encoding methods, task types, and graph structure affect performance. ### Source excerpt Posted by Bahare Fatemi and Bryan Perozzi, Research Scientists, Google Research Imagine all the things around you -- your friends, tools in your kitchen, or even the parts of your bike. They are all connected in different ways. In computer science, the term graph is used to describe connections between objects. Graphs consist of nodes (the objects themselves) and edges (connections between two nodes, indicating a relationship between them). Graphs are everywhere now. The internet itself is a giant graph of websites linked together. Even the knowledge search engines use is organized in a graph-like way. Furthermore, consider the remarkable advancements in artificial intelligence -- such as chatbots that can write stories in seconds, and even software that can interpret medical reports. This exciting progress is largely thanks to large language models (LLMs). New LLM technology is constantly being developed for different uses. Since graphs are everywhere and LLM technology is on the rise, in "Talk like a Graph: Encoding Graphs for Large Language Models", presented at ICLR 2024, we present a way to teach powerful LLMs how to better reason with graph information. Graphs are a useful way to organize information, but LLMs are mostly trained on regular text. The objective is to test different techniques to see what works best and gain practical insights. Translating graphs into text that LLMs can understand is a remarkably complex task. The difficulty stems from the inherent complexity of graph structures with multiple nodes and the intricate web of edges that connect them. Our work studies how to take a graph and translate it into a format that an LLM can understand. We also design a benchmark called GraphQA to study different approaches on different graph reasoning problems and show how to phrase a graph-related problem in a way that enables the LLM to solve the graph problem. We show that LLM performance on graph reasoning tasks varies on three fundamental levels: 1) the ## Chain-of-table: Evolving tables in the reasoning chain for table understanding DevFeed: [Chain-of-table: Evolving tables in the reasoning chain for table understanding]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-11T19:08:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Chain-of-thought](), [Natural language processing](), [Large Language Model](), [data](), [LLM evaluation / benchmarking]() Tags: [actor](), [ai](), [awards](), [benchmarks](), [chain-of-thought](), [data](), [information](), [language-models](), [machine-intelligence](), [models](), [natural-language-processing](), [performance](), [reasoning](), [research](), [table](), [verify]() ### AI overview The article presents Chain-of-Table, a framework for table understanding that trains large language models to reason step by step by iteratively updating a table. The approach uses in-context learning to generate operations dynamically and reports improved results on the WikiTQ, TabFact, and FeTaQA benchmarks. ### Source excerpt Posted by Zilong Wang, Student Researcher, and Chen-Yu Lee, Research Scientist, Cloud AI Team People use tables every day to organize and interpret complex information in a structured, easily accessible format. Due to the ubiquity of such tables, reasoning over tabular data has long been a central topic in natural language processing (NLP). Researchers in this field have aimed to leverage language models to help users answer questions, verify statements, and analyze data based on tables. However, language models are trained over large amounts of plain text, so the inherently structured nature of tabular data can be difficult for language models to fully comprehend and utilize. Recently, large language models (LLMs) have achieved outstanding performance across diverse natural language understanding (NLU) tasks by generating reliable reasoning chains, as shown in works like Chain-of-Thought and Least-to-Most. However, the most suitable way for LLMs to reason over tabular data remains an open question. In "Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding", we propose a framework to tackle table understanding tasks, where we train LLMs to outline their reasoning step by step, updating a given table iteratively to reflect each part of a thought process, akin to how people solve the table-based problems. This enables the LLM to transform the table into simpler and more manageable segments so that it can understand and analyze each part of the table in depth. This approach has yielded significant improvements and achieved new state-of-the-art results on the WikiTQ, TabFact, and FeTaQA benchmarks. The figure below shows the high-level overview of the proposed Chain-of-Table and other methods. Given a complex table where a cyclist's nationality and name are in the same cell, (a) generic, multi-step reasoning is unable to provide the correct answer (b) program-aided reasoning generates and executes programs (e.g., SQL queries) to deliver the answ ## Health-specific embedding tools for dermatology and pathology DevFeed: [Health-specific embedding tools for dermatology and pathology]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-08T19:33:00Z Content type: release Language: en Sources: [Google Research]() Topics: [Medical imaging](), [Embeddings](), [Deep learning](), [Machine learning](), [Google]() Tags: [computer-vision](), [deep-learning](), [embedding](), [google](), [health](), [image-classification](), [medical-imaging](), [product](), [release]() ### AI overview Google Research announces Derm Foundation and Path Foundation, two domain-specific embedding tools for research in dermatology and digital pathology. The tools convert medical images into specialized numerical vectors that researchers can use to develop models for downstream applications. ### Source excerpt Posted by Dave Steiner, Clinical Research Scientist, Google Health, and Rory Pilgrim, Product Manager, Google Research There's a worldwide shortage of access to medical imaging expert interpretation across specialties including radiology, dermatology and pathology. Machine learning (ML) technology can help ease this burden by powering tools that enable doctors to interpret these images more accurately and efficiently. However, the development and implementation of such ML tools are often limited by the availability of high-quality data, ML expertise, and computational resources. One way to catalyze the use of ML for medical imaging is via domain-specific models that utilize deep learning (DL) to capture the information in medical images as compressed numerical vectors (called embeddings). These embeddings represent a type of pre-learned understanding of the important features in an image. Identifying patterns in the embeddings reduces the amount of data, expertise, and compute needed to train performant models as compared to working with high-dimensional data, such as images, directly. Indeed, these embeddings can be used to perform a variety of downstream tasks within the specialized domain (see animated graphic below). This framework of leveraging pre-learned understanding to solve related tasks is similar to that of a seasoned guitar player quickly learning a new song by ear. Because the guitar player has already built up a foundation of skill and understanding, they can quickly pick up the patterns and groove of a new song. Path Foundation is used to convert a small dataset of (image, label) pairs into (embedding, label) pairs. These pairs can then be used to train a task-specific classifier using a linear probe, (i.e., a lightweight linear classifier) as represented in this graphic, or other types of models using the embeddings as input. Once the linear probe is trained, it can be used to make predictions on embeddings from new images. These predictions can be ## Social learning: Collaborative learning with large language models DevFeed: [Social learning: Collaborative learning with large language models]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-07T18:15:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Large Language Model](), [Federated Learning](), [datasets](), [Learning]() Tags: [datasets](), [federated-learning](), [few-shot](), [few-shot-learning](), [large-language-models](), [machine-intelligence](), [natural-language-processing](), [privacy](), [research](), [use-cases]() ### AI overview This Google Research article presents a privacy-aware social learning framework in which large language models teach one another using natural language rather than shared gradients. It describes evaluations involving spam detection, grade-school mathematics, and text-based question answering, including settings where labeled examples cannot be directly shared because of privacy concerns. ### Source excerpt Posted by Amirkeivan Mohtashami, Research Intern, and Florian Hartmann, Software Engineer, Google Research Large language models (LLMs) have significantly improved the state of the art for solving tasks specified using natural language, often reaching performance close to that of people. As these models increasingly enable assistive agents, it could be beneficial for them to learn effectively from each other, much like people do in social settings, which would allow LLM-based agents to improve each other's performance. To discuss the learning processes of humans, Bandura and Walters described the concept of social learning in 1977, outlining different models of observational learning used by people. One common method of learning from others is through a verbal instruction (e.g., from a teacher) that describes how to engage in a particular behavior. Alternatively, learning can happen through a live model by mimicking a live example of the behavior. Given the success of LLMs mimicking human communication, in our paper "Social Learning: Towards Collaborative Learning with Large Language Models", we investigate whether LLMs are able to learn from each other using social learning. To this end, we outline a framework for social learning in which LLMs share knowledge with each other in a privacy-aware manner using natural language. We evaluate the effectiveness of our framework on various datasets, and propose quantitative methods that measure privacy in this setting. In contrast to previous approaches to collaborative learning, such as common federated learning approaches that often rely on gradients, in our framework, agents teach each other purely using natural language. Social learning for LLMs To extend social learning to language models, we consider the scenario where a student LLM should learn to solve a task from multiple teacher entities that already know that task. In our paper, we evaluate the student's performance on a variety of tasks, such as spam detection i ## Croissant: a metadata format for ML-ready datasets DevFeed: [Croissant: a metadata format for ML-ready datasets]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-06T18:26:00Z Content type: release Language: en Sources: [Google Research]() Topics: [datasets](), [Machine Learning & Artificial Intelligence](), [Structured-data](), [Google]() Tags: [collaboration](), [datasets](), [format](), [google](), [ml]() ### AI overview Google Research announces Croissant, a metadata format for ML-ready datasets. It standardizes how dataset resources, organization, and machine-learning metadata are described without changing the underlying data formats, and is intended to support responsible use and training, test, and validation set definitions. ### Source excerpt Posted by Omar Benjelloun, Software Engineer, Google Research, and Peter Mattson, Software Engineer, Google Core ML and President, MLCommons Association Machine learning (ML) practitioners looking to reuse existing datasets to train an ML model often spend a lot of time understanding the data, making sense of its organization, or figuring out what subset to use as features. So much time, in fact, that progress in the field of ML is hampered by a fundamental obstacle: the wide variety of data representations. ML datasets cover a broad range of content types, from text and structured data to images, audio, and video. Even within datasets that cover the same types of content, every dataset has a unique ad hoc arrangement of files and data formats. This challenge reduces productivity throughout the entire ML development process, from finding the data to training the model. It also impedes development of badly needed tooling for working with datasets. There are general purpose metadata formats for datasets such as schema.org and DCAT. However, these formats were designed for data discovery rather than for the specific needs of ML data, such as the ability to extract and combine data from structured and unstructured sources, to include metadata that would enable responsible use of the data, or to describe ML usage characteristics such as defining training, test and validation sets. Today, we're introducing Croissant, a new metadata format for ML-ready datasets. Croissant was developed collaboratively by a community from industry and academia, as part of the MLCommons effort. The Croissant format doesn't change how the actual data is represented (e.g., image or text file formats) -- it provides a standard way to describe and organize it. Croissant builds upon schema.org, the de facto standard for publishing structured data on the Web, which is already used by over 40M datasets. Croissant augments it with comprehensive layers for ML relevant metadata, data resources, data or ## Google at APS 2024 DevFeed: [Google at APS 2024]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-03-04T15:06:00Z Content type: news Language: en Sources: [Google Research]() Topics: [Google](), [Simulation](), [Open Source](), [Algorithms, Complexity]() Tags: [competition](), [conference](), [conferences](), [google](), [library](), [open-source](), [physics](), [prototype](), [quantum-ai](), [quantum-computing](), [simulation](), [talks]() ### AI overview Google Research's Quantum AI team outlines its presence at the 2024 March Meeting of the American Physical Society, including a booth, more than 50 talks, organizing activities, and sessions on quantum simulation, machine learning, circuit errors, and quantum applications. ### Source excerpt Posted by Kate Weber and Shannon Leon, Google Research, Quantum AI Team Today the 2024 March Meeting of the American Physical Society (APS) kicks off in Minneapolis, MN. A premier conference on topics ranging across physics and related fields, APS 2024 brings together researchers, students, and industry professionals to share their discoveries and build partnerships with the goal of realizing fundamental advances in physics-related sciences and technology. This year, Google has a strong presence at APS with a booth hosted by the Google Quantum AI team, 50+ talks throughout the conference, and participation in conference organizing activities, special sessions and events. Attending APS 2024 in person? Come visit Google's Quantum AI booth to learn more about the exciting work we're doing to solve some of the field's most interesting challenges. You can learn more about the latest cutting edge work we are presenting at the conference along with our schedule of booth events below (Googlers listed in bold). Organizing Committee Session Chairs include: Aaron Szasz Booth Activities This schedule is subject to change. Please visit the Google Quantum AI booth for more information. Crumble: A prototype interactive tool for visualizing QEC circuits Presenter: Matt McEwen Tue, Mar 5 | 11:00 AM CST Qualtran: An open-source library for effective resource estimation of fault tolerant algorithms Presenter: Tanuj Khattar Tue, Mar 5 | 2:30 PM CST Qualtran: An open-source library for effective resource estimation of fault tolerant algorithms Presenter: Tanuj Khattar Thu, Mar 7 | 11:00 AM CST $5M XPRIZE / Google Quantum AI competition to accelerate quantum applications Q&A Presenter: Ryan Babbush Thu, Mar 7 | 11:00 AM CST Talks Monday Certifying highly-entangled states from few single-qubit measurements Presenter: Hsin-Yuan Huang Author: Hsin-Yuan Huang Session A45: New Frontiers in Machine Learning Quantum Physics Toward high-fidelity analog quantum simulation with superconducting qub ## VideoPrism: A foundational visual encoder for video understanding DevFeed: [VideoPrism: A foundational visual encoder for video understanding]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-22T20:05:00Z Content type: article Language: en Sources: [Google Research]() Topics: [foundation-models](), [dataset](), [datasets](), [Google](), [data](), [Software]() Tags: [classification](), [dataset](), [foundation-models](), [google](), [machine-intelligence](), [machine-perception](), [model](), [performance](), [qa](), [retrieval](), [training](), [video]() ### AI overview Google Research introduces VideoPrism, a video foundation model designed for general-purpose video understanding across tasks including classification, localization, retrieval, captioning, and question answering. The model is pretrained on video-text pairs and video clips with noisy or machine-generated text, and the article reports state-of-the-art performance using a single frozen model. ### Source excerpt Posted by Long Zhao, Senior Research Scientist, and Ting Liu, Senior Staff Software Engineer, Google Research An astounding number of videos are available on the Web, covering a variety of content from everyday moments people share to historical moments to scientific observations, each of which contains a unique record of the world. The right tools could help researchers analyze these videos, transforming how we understand the world around us. Videos offer dynamic visual content far more rich than static images, capturing movement, changes, and dynamic relationships between entities. Analyzing this complexity, along with the immense diversity of publicly available video data, demands models that go beyond traditional image understanding. Consequently, many of the approaches that best perform on video understanding still rely on specialized models tailor-made for particular tasks. Recently, there has been exciting progress in this area using video foundation models (ViFMs), such as VideoCLIP, InternVideo, VideoCoCa, and UMT. However, building a ViFM that handles the sheer diversity of video data remains a challenge. With the goal of building a single model for general-purpose video understanding, we introduce "VideoPrism: A Foundational Visual Encoder for Video Understanding". VideoPrism is a ViFM designed to handle a wide spectrum of video understanding tasks, including classification, localization, retrieval, captioning, and question answering (QA). We propose innovations in both the pre-training data as well as the modeling strategy. We pre-train VideoPrism on a massive and diverse dataset: 36 million high-quality video-text pairs and 582 million video clips with noisy or machine-generated parallel text. Our pre-training approach is designed for this hybrid data, to learn both from video-text pairs and the videos themselves. VideoPrism is incredibly easy to adapt to new video understanding challenges, and achieves state-of-the-art performance using a single frozen ## Advances in private training for production on-device language models DevFeed: [Advances in private training for production on-device language models]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-21T20:15:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Language models](), [Federated Learning](), [Google](), [Mobile]() Tags: [differential-privacy](), [federated-learning](), [gboard](), [google](), [language-models](), [on-device](), [on-device-learning](), [phones](), [privacy](), [production](), [research](), [responsible-ai](), [training]() ### AI overview This Google Research article describes advances in privately training Gboard's on-device language models. It explains how federated learning keeps training data on users' devices while differential privacy provides formal privacy guarantees. The article reports that all Gboard next-word-prediction neural language models use federated learning with formal differential privacy guarantees, across more than 30 models, 7 or more languages, and 15 or more countries. ### Source excerpt Posted by Zheng Xu, Research Scientist, and Yanxiang Zhang, Software Engineer, Google Language models (LMs) trained to predict the next word given input text are the key technology for many applications [1, 2]. In Gboard, LMs are used to improve users' typing experience by supporting features like next word prediction (NWP), Smart Compose, smart completion and suggestion, slide to type, and proofread. Deploying models on users' devices rather than enterprise servers has advantages like lower latency and better privacy for model usage. While training on-device models directly from user data effectively improves the utility performance for applications such as NWP and smart text selection, protecting the privacy of user data for model training is important. Gboard features powered by on-device language models. In this blog we discuss how years of research advances now power the private training of Gboard LMs, since the proof-of-concept development of federated learning (FL) in 2017 and formal differential privacy (DP) guarantees in 2022. FL enables mobile phones to collaboratively learn a model while keeping all the training data on device, and DP provides a quantifiable measure of data anonymization. Formally, DP is often characterized by (ε, δ) with smaller values representing stronger guarantees. Machine learning (ML) models are considered to have reasonable DP guarantees for ε=10 and strong DP guarantees for ε=1 when δ is small. As of today, all NWP neural network LMs in Gboard are trained with FL with formal DP guarantees, and all future launches of Gboard LMs trained on user data require DP. These 30+ Gboard on-device LMs are launched in 7+ languages and 15+ countries, and satisfy (ɛ, δ)-DP guarantees of small δ of 10-10 and ɛ between 0.994 and 13.69. To the best of our knowledge, this is the largest known deployment of user-level DP in production at Google or anywhere, and the first time a strong DP guarantee of ɛ < 1 is announced for models trained directly on ## Learning the importance of training data under concept drift DevFeed: [Learning the importance of training data under concept drift]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-14T18:32:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Artificial Intelligence](), [Training AI Models](), [Benchmark](), [datasets](), [data](), [Google]() Tags: [ai](), [ai-models](), [benchmark](), [continual-learning](), [datasets](), [deep-learning](), [performance](), [research](), [supervised-learning](), [training](), [training-data]() ### AI overview Google Research describes a method for handling slow concept drift in nonstationary learning by assigning training instances importance scores based on their content and age. An auxiliary model jointly learned with the primary model produces the scores, and the authors report up to 15% relative accuracy gains on a large benchmark dataset. ### Source excerpt Posted by Nishant Jain, Pre-doctoral Researcher, and Pradeep Shenoy, Research Scientist, Google Research The constantly changing nature of the world around us poses a significant challenge for the development of AI models. Often, models are trained on longitudinal data with the hope that the training data used will accurately represent inputs the model may receive in the future. More generally, the default assumption that all training data are equally relevant often breaks in practice. For example, the figure below shows images from the CLEAR nonstationary learning benchmark, and it illustrates how visual features of objects evolve significantly over a 10 year span (a phenomenon we refer to as slow concept drift), posing a challenge for object categorization models. Sample images from the CLEAR benchmark. (Adapted from Lin et al.) Alternative approaches, such as online and continual learning, repeatedly update a model with small amounts of recent data in order to keep it current. This implicitly prioritizes recent data, as the learnings from past data are gradually erased by subsequent updates. However in the real world, different kinds of information lose relevance at different rates, so there are two key issues: 1) By design they focus exclusively on the most recent data and lose any signal from older data that is erased. 2) Contributions from data instances decay uniformly over time irrespective of the contents of the data. In our recent work, "Instance-Conditional Timescales of Decay for Non-Stationary Learning", we propose to assign each instance an importance score during training in order to maximize model performance on future data. To accomplish this, we employ an auxiliary model that produces these scores using the training instance as well as its age. This model is jointly learned with the primary model. We address both the above challenges and achieve significant gains over other robust learning methods on a range of benchmark datasets for nonstationary ## DP-Auditorium: A flexible library for auditing differential privacy DevFeed: [DP-Auditorium: A flexible library for auditing differential privacy]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-13T22:11:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Library](), [Testing](), [Open Source](), [Python](), [Tool]() Tags: [algorithms](), [data-protection](), [differential-privacy](), [library](), [open-source](), [privacy](), [python](), [research](), [responsible-ai](), [security-and-privacy](), [testing]() ### AI overview Google Research introduces DP-Auditorium, an open source Python library that audits differential privacy guarantees using black-box access to mechanisms. It includes testing algorithms for Rényi, pure, and approximate differential privacy and can identify guarantee violations and suggest suitable tests for particular bugs. ### Source excerpt Posted by Mónica Ribero Díaz, Research Scientist, Google Research Differential privacy (DP) is a property of randomized mechanisms that limit the influence of any individual user's information while processing and analyzing data. DP offers a robust solution to address growing concerns about data protection, enabling technologies across industries and government applications (e.g., the US census) without compromising individual user identities. As its adoption increases, it's important to identify the potential risks of developing mechanisms with faulty implementations. Researchers have recently found errors in the mathematical proofs of private mechanisms, and their implementations. For example, researchers compared six sparse vector technique (SVT) variations and found that only two of the six actually met the asserted privacy guarantee. Even when mathematical proofs are correct, the code implementing the mechanism is vulnerable to human error. However, practical and efficient DP auditing is challenging primarily due to the inherent randomness of the mechanisms and the probabilistic nature of the tested guarantees. In addition, a range of guarantee types exist, (e.g., pure DP, approximate DP, Rényi DP, and concentrated DP), and this diversity contributes to the complexity of formulating the auditing problem. Further, debugging mathematical proofs and code bases is an intractable task given the volume of proposed mechanisms. While ad hoc testing techniques exist under specific assumptions of mechanisms, few efforts have been made to develop an extensible tool for testing DP mechanisms. To that end, in "DP-Auditorium: A Large Scale Library for Auditing Differential Privacy", we introduce an open source library for auditing DP guarantees with only black-box access to a mechanism (i.e., without any knowledge of the mechanism's internal properties). DP-Auditorium is implemented in Python and provides a flexible interface that allows contributions to continuously improve ## Graph neural networks in TensorFlow DevFeed: [Graph neural networks in TensorFlow]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-06T19:17:00Z Content type: release Language: en Sources: [Google Research]() Topics: [Graphs](), [Tensorflow](), [Library](), [Google](), [Neural Network]() Tags: [google](), [graph](), [graph-mining](), [graphs](), [library](), [machine-learning](), [neural-networks](), [release](), [tensorflow]() ### AI overview Google Research announces TensorFlow GNN 1.0, a production-tested library for building and training graph neural networks at large scales. The library supports heterogeneous graphs, TensorFlow modeling and training, and extracting input graphs from large data stores. ### Source excerpt Posted by Dustin Zelle, Software Engineer, Google Research, and Arno Eigenwillig, Software Engineer, CoreML Objects and their relationships are ubiquitous in the world around us, and relationships can be as important to understanding an object as its own attributes viewed in isolation -- take for example transportation networks, production networks, knowledge graphs, or social networks. Discrete mathematics and computer science have a long history of formalizing such networks as graphs, consisting of nodes connected by edges in various irregular ways. Yet most machine learning (ML) algorithms allow only for regular and uniform relations between input objects, such as a grid of pixels, a sequence of words, or no relation at all. Graph neural networks, or GNNs for short, have emerged as a powerful technique to leverage both the graph's connectivity (as in the older algorithms DeepWalk and Node2Vec) and the input features on the various nodes and edges. GNNs can make predictions for graphs as a whole (Does this molecule react in a certain way?), for individual nodes (What's the topic of this document, given its citations?) or for potential edges (Is this product likely to be purchased together with that product?). Apart from making predictions about graphs, GNNs are a powerful tool used to bridge the chasm to more typical neural network use cases. They encode a graph's discrete, relational information in a continuous way so that it can be included naturally in another deep learning system. We are excited to announce the release of TensorFlow GNN 1.0 (TF-GNN), a production-tested library for building GNNs at large scales. It supports both modeling and training in TensorFlow as well as the extraction of input graphs from huge data stores. TF-GNN is built from the ground up for heterogeneous graphs, where types of objects and relations are represented by distinct sets of nodes and edges. Real-world objects and their relations occur in distinct types, and TF-GNN's heterogen ## A decoder-only foundation model for time-series forecasting DevFeed: [A decoder-only foundation model for time-series forecasting]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-02T19:07:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Time Series](), [foundation-models](), [Google](), [Deep learning]() Tags: [deep-learning](), [forecasting](), [google](), [google-cloud-platform](), [machine-learning](), [model](), [time-series]() ### AI overview Google Research introduces TimesFM, a decoder-only foundation model for time-series forecasting. The model is pretrained on 100 billion real-world time-points and is reported to provide zero-shot forecasts on unseen datasets across domains and temporal granularities, with 200 million parameters. ### Source excerpt Posted by Rajat Sen and Yichen Zhou, Google Research Time-series forecasting is ubiquitous in various domains, such as retail, finance, manufacturing, healthcare and natural sciences. In retail use cases, for example, it has been observed that improving demand forecasting accuracy can meaningfully reduce inventory costs and increase revenue. Deep learning (DL) models have emerged as a popular approach for forecasting rich, multivariate, time-series data because they have proven to perform well in a variety of settings (e.g., DL models performed well in the M5 competition). At the same time, there has been rapid progress in large foundation language models used for natural language processing (NLP) tasks, such as translation, retrieval-augmented generation, and code completion. These models are trained on massive amounts of textual data derived from a variety of sources like common crawl and open-source code that allows them to identify patterns in languages. This makes them very powerful zero-shot tools; for instance, when paired with retrieval, they can answer questions about and summarize current events. Despite DL-based forecasters largely outperforming traditional methods and progress being made in reducing training and inference costs, they face challenges: most DL architectures require long and involved training and validation cycles before a customer can test the model on a new time-series. A foundation model for time-series forecasting, in contrast, can provide decent out-of-the-box forecasts on unseen time-series data with no additional training, enabling users to focus on refining forecasts for the actual downstream task like retail demand planning. To that end, in "A decoder-only foundation model for time-series forecasting", we introduce TimesFM, a single forecasting model pre-trained on a large time-series corpus of 100 billion real world time-points. Compared to the latest large language models (LLMs), TimesFM is much smaller (200M parameters), yet we ## Intervening on early readouts for mitigating spurious features and simplicity bias DevFeed: [Intervening on early readouts for mitigating spurious features and simplicity bias]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-02-02T17:49:00Z Content type: article Language: en Sources: [Google Research]() Topics: [AI Research](), [Deep learning](), [responsible-ai](), [generalization in machine learning](), [Model Development](), [Training AI Models]() Tags: [bias](), [dataset](), [deep-learning](), [icml](), [machine-learning](), [ml-fairness](), [research](), [responsible-ai](), [supervised-learning](), [training]() ### AI overview Google Research describes methods for detecting and reducing spurious features and simplicity bias in deep learning models. Early readouts expose confidently wrong predictions associated with spurious features, while feature forgetting helps models identify more predictive features and generalize to unseen domains. ### Source excerpt Posted by Rishabh Tiwari, Pre-doctoral Researcher, and Pradeep Shenoy, Research Scientist, Google Research Machine learning models in the real world are often trained on limited data that may contain unintended statistical biases. For example, in the CELEBA celebrity image dataset, a disproportionate number of female celebrities have blond hair, leading to classifiers incorrectly predicting "blond" as the hair color for most female faces -- here, gender is a spurious feature for predicting hair color. Such unfair biases could have significant consequences in critical applications such as medical diagnosis. Surprisingly, recent work has also discovered an inherent tendency of deep networks to amplify such statistical biases, through the so-called simplicity bias of deep learning. This bias is the tendency of deep networks to identify weakly predictive features early in the training, and continue to anchor on these features, failing to identify more complex and potentially more accurate features. With the above in mind, we propose simple and effective fixes to this dual challenge of spurious features and simplicity bias by applying early readouts and feature forgetting. First, in "Using Early Readouts to Mediate Featural Bias in Distillation", we show that making predictions from early layers of a deep network (referred to as "early readouts") can automatically signal issues with the quality of the learned representations. In particular, these predictions are more often wrong, and more confidently wrong, when the network is relying on spurious features. We use this erroneous confidence to improve outcomes in model distillation, a setting where a larger "teacher" model guides the training of a smaller "student" model. Then in "Overcoming Simplicity Bias in Deep Networks using a Feature Sieve", we intervene directly on these indicator signals by making the network "forget" the problematic features and consequently look for better, more predictive features. This substanti ## MobileDiffusion: Rapid text-to-image generation on-device DevFeed: [MobileDiffusion: Rapid text-to-image generation on-device]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-01-31T21:59:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Machine learning](), [Mobile](), [text-to-image](), [Inference](), [model-deployment](), [Optimization](), [Android](), [iOS](), [MediaPipe]() Tags: [android](), [computer-vision](), [deployment](), [diffusion](), [generation](), [image](), [inference](), [ios](), [machine-learning](), [mediapipe](), [mobile](), [on-device](), [on-device-learning](), [optimization](), [text-to-image]() ### AI overview Google researchers introduce MobileDiffusion, a latent diffusion model designed for on-device text-to-image generation. Using DiffusionGAN for one-step sampling, it was tested on premium iOS and Android devices and generated 512x512 images in about half a second with a 520-million-parameter model. ### Source excerpt Posted by Yang Zhao, Senior Software Engineer, and Tingbo Hou, Senior Staff Software Engineer, Core ML Text-to-image diffusion models have shown exceptional capabilities in generating high-quality images from text prompts. However, leading models feature billions of parameters and are consequently expensive to run, requiring powerful desktops or servers (e.g., Stable Diffusion, DALL-E, and Imagen). While recent advancements in inference solutions on Android via MediaPipe and iOS via Core ML have been made in the past year, rapid (sub-second) text-to-image generation on mobile devices has remained out of reach. To that end, in "MobileDiffusion: Subsecond Text-to-Image Generation on Mobile Devices", we introduce a novel approach with the potential for rapid text-to-image generation on-device. MobileDiffusion is an efficient latent diffusion model specifically designed for mobile devices. We also adopt DiffusionGAN to achieve one-step sampling during inference, which fine-tunes a pre-trained diffusion model while leveraging a GAN to model the denoising step. We have tested MobileDiffusion on iOS and Android premium devices, and it can run in half a second to generate a 512x512 high-quality image. Its comparably small model size of just 520M parameters makes it uniquely suited for mobile deployment. Rapid text-to-image generation on-device. Background The relative inefficiency of text-to-image diffusion models arises from two primary challenges. First, the inherent design of diffusion models requires iterative denoising to generate images, necessitating multiple evaluations of the model. Second, the complexity of the network architecture in text-to-image diffusion models involves a substantial number of parameters, regularly reaching into the billions and resulting in computationally expensive evaluations. As a result, despite the potential benefits of deploying generative models on mobile devices, such as enhancing user experience and addressing emerging privacy concer ## Mixed-input matrix multiplication performance optimizations DevFeed: [Mixed-input matrix multiplication performance optimizations]() Original publisher: [Read original article]() Author: Google AI (noreply@blogger.com) Published: 2024-01-26T19:56:00Z Content type: article Language: en Sources: [Google Research]() Topics: [Large language models (LLMs)](), [quantization](), [Hardware](), [GPU](), [Nvidia](), [Tensor Cores](), [Open Source](), [Software](), [data type]() Tags: [accelerators](), [ai](), [algorithms](), [ampere](), [compute](), [conversion](), [data](), [deep-learning](), [effective](), [efficiency](), [hardware](), [implementation](), [language-models](), [large-language-models-llms](), [models](), [nvidia](), [optimization](), [performance](), [precision](), [quantization](), [research](), [software](), [tensor-cores]() ### AI overview This Google Research article explains software techniques for mapping mixed-input matrix multiplication onto NVIDIA Ampere hardware. It describes using lower-precision weights with higher-precision inputs, data-type conversion, and layout transformations to support weight-only quantization. The authors report minimal software overhead and performance close to peak hardware capabilities, and state that the techniques were released in the open-source NVIDIA/CUTLASS repository. ### Source excerpt Posted by Manish Gupta, Staff Software Engineer, Google Research AI-driven technologies are weaving themselves into the fabric of our daily routines, with the potential to enhance our access to knowledge and boost our overall productivity. The backbone of these applications lies in large language models (LLMs). LLMs are memory-intensive and typically require specialized hardware accelerators to efficiently deliver tens of exaflops of computing power. This blog post shows how we can start addressing the computational challenges by utilizing memory more effectively. The bulk of an LLM's memory and compute are consumed by weights in matrix multiplication operations. Using narrower data types reduces memory consumption. For example, storing weights in the 8-bit integer (i.e., U8 or S8) data type reduces the memory footprint by 4x relative to single-precision (F32) and 2x relative to half-precision (F16) or bfloat16 (BF16). Furthermore, previous work has shown that LLM models running matrix multiplications with weights in S8 and input in F16 (preserving higher precision of the user-input) is an effective method for increasing the efficiency with acceptable trade-offs in accuracy. This technique is known as weight-only quantization and requires efficient implementation of matrix multiplication with mixed-inputs, e.g., half-precision input multiplied with 8-bits integer. Hardware accelerators, including GPUs, support a fixed set of data types, and thus, mixed-input matrix multiplication requires software transformations to map to the hardware operations. To that end, in this blog we focus on mapping mixed-input matrix multiplication onto the NVIDIA Ampere architecture. We present software techniques addressing data type conversion and layout conformance to map mixed-input matrix multiplication efficiently onto hardware-supported data types and layouts. Our results show that the overhead of additional work in software is minimal and enables performance close to the peak har [Next page]()