# Mixed-input matrix multiplication performance optimizations

DevFeed: [Mixed-input matrix multiplication performance optimizations](<https://devfeed.tech/articles/mixed-input-matrix-multiplication-performance-optimizations-28543.md>)

Original publisher: [Read original article](<http://blog.research.google/2024/01/mixed-input-matrix-multiplication.html>)

Author: Google AI (noreply@blogger.com)

Published: 2024-01-26T19:56:00Z

Content type: article

Language: en

Sources: [Google Research](<https://devfeed.tech/sources/google-research.md>)

Topics: [Large language models (LLMs)](<https://devfeed.tech/topics/large-language-models-llms.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Software](<https://devfeed.tech/topics/software.md>), [data type](<https://devfeed.tech/topics/data-type.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [ai](<https://devfeed.tech/tags/ai.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [ampere](<https://devfeed.tech/tags/ampere.md>), [compute](<https://devfeed.tech/tags/compute.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [data](<https://devfeed.tech/tags/data.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [effective](<https://devfeed.tech/tags/effective.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [models](<https://devfeed.tech/tags/models.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [research](<https://devfeed.tech/tags/research.md>), [software](<https://devfeed.tech/tags/software.md>), [tensor-cores](<https://devfeed.tech/tags/tensor-cores.md>)

## AI overview

This Google Research article explains software techniques for mapping mixed-input matrix multiplication onto NVIDIA Ampere hardware. It describes using lower-precision weights with higher-precision inputs, data-type conversion, and layout transformations to support weight-only quantization. The authors report minimal software overhead and performance close to peak hardware capabilities, and state that the techniques were released in the open-source NVIDIA/CUTLASS repository.

## Source excerpt

Posted by Manish Gupta, Staff Software Engineer, Google Research AI-driven technologies are weaving themselves into the fabric of our daily routines, with the potential to enhance our access to knowledge and boost our overall productivity. The backbone of these applications lies in large language models (LLMs). LLMs are memory-intensive and typically require specialized hardware accelerators to efficiently deliver tens of exaflops of computing power. This blog post shows how we can start addressing the computational challenges by utilizing memory more effectively. The bulk of an LLM's memory and compute are consumed by weights in matrix multiplication operations. Using narrower data types reduces memory consumption. For example, storing weights in the 8-bit integer (i.e., U8 or S8) data type reduces the memory footprint by 4x relative to single-precision (F32) and 2x relative to half-precision (F16) or bfloat16 (BF16). Furthermore, previous work has shown that LLM models running matrix multiplications with weights in S8 and input in F16 (preserving higher precision of the user-input) is an effective method for increasing the efficiency with acceptable trade-offs in accuracy. This technique is known as weight-only quantization and requires efficient implementation of matrix multiplication with mixed-inputs, e.g., half-precision input multiplied with 8-bits integer. Hardware accelerators, including GPUs, support a fixed set of data types, and thus, mixed-input matrix multiplication requires software transformations to map to the hardware operations. To that end, in this blog we focus on mapping mixed-input matrix multiplication onto the NVIDIA Ampere architecture. We present software techniques addressing data type conversion and layout conformance to map mixed-input matrix multiplication efficiently onto hardware-supported data types and layouts. Our results show that the overhead of additional work in software is minimal and enables performance close to the peak har