# Transformer architecture Published articles for Transformer architecture. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Pathway's brain-inspired architecture development on Amazon SageMaker HyperPod DevFeed: [Pathway's brain-inspired architecture development on Amazon SageMaker HyperPod]() Original publisher: [Read original article]() Author: Paulo Aragão Published: 2026-09-08T19:12:51Z Content type: article Language: en Sources: [Artificial Intelligence]() Topics: [Language models](), [Training AI Models](), [compression and generalization](), [LLM evaluation / benchmarking]() Tags: [amazon](), [amazon-sagemaker](), [amazon-sagemaker-hyperpod](), [announcements](), [architecture](), [benchmark](), [chain-of-thought](), [distributed-training](), [intermediate-200](), [pytorch](), [transformer-architecture]() ### AI overview Pathway describes BDH, a brain-inspired architecture that performs reasoning in latent space rather than producing chain-of-thought token traces. The article covers its recurrent internal memory, its contrast with transformer limitations, and scaling training with Amazon SageMaker HyperPod. ### Source excerpt Pathway's Baby Dragon Hatchling (BDH) is a brain-inspired, post-transformer architecture that reasons in latent space instead of emitting chain-of-thought tokens. See how Pathway develops and scales BDH on Amazon SageMaker HyperPod, and how BDH-CQ set a new cost-efficiency mark on the ARC-AGI-1 benchmark. ## TimesFM-3: A zero-shot foundation model for multivariate forecasting DevFeed: [TimesFM-3: A zero-shot foundation model for multivariate forecasting]() Original publisher: [Read original article]() Published: 2026-08-31T17:19:40Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [foundation-models](), [Time Series](), [Google](), [datasets](), [Transformer architecture](), [Fine-tuning](), [generalization in machine learning]() Tags: [data-management](), [features](), [fine-tuning](), [foundation-models](), [google](), [machine-intelligence](), [model](), [product](), [time-series](), [transformer-architecture](), [zero-shot]() ### AI overview Google Research introduces TimesFM-3, a 330-million-parameter time-series foundation model designed for accurate multivariate forecasting in a single forward pass. Pre-trained on more than one trillion real-world and synthetic time points, it jointly models coevolving series and external covariates in zero-shot settings without task-specific fine-tuning. ### Source excerpt Data Management ## Granite 4.2 LLMs: How They're Built DevFeed: [Granite 4.2 LLMs: How They're Built]() Original publisher: [Read original article]() Author: Yousaf Shah; Swanand Kadhe; Riddhiman Moulick; Ashish Sunil Agrawal; Santosh Borse Published: 2026-08-25T15:14:14Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Language models](), [AI Chat](), [AI-assisted coding](), [releases]() Tags: [agentic](), [agents](), [apache](), [chain-of-thought](), [code](), [fine-tuning](), [grouped-query-attention](), [ibm](), [llms](), [models](), [post-training](), [reasoning](), [reinforcement-learning](), [sglang](), [tool](), [training](), [transformer-architecture](), [vllm]() ### AI overview Granite 4.2 is a family of 3B, 8B, and 30B dense decoder-only reasoning language models. The article covers their training pipeline, thinking modes, native tool calling, and agentic reinforcement learning for the 8B and 30B models. ### Source excerpt Authors: Granite Team, IBM TL;DR: Granite 4.2 is our first family of dense, decoder-only reasoning LLMs, released in three sizes: 3B, 8B, and 30B. These models are post-trained from Granite-4.1 base models. Granite-4.1 base models were pre-trained from scratch on roughly 15T tokens with a five-phase strategy that extends the context window to 512K tokens, supervised fine-tuned on chain-of-thought, reasoning, and agentic-trajectory data, then post-trained with a multi-stage reinforcement... ## Making LLMs faster without sacrificing accuracy DevFeed: [Making LLMs faster without sacrificing accuracy]() Original publisher: [Read original article]() Author: Tao Yu; Youngsuk Park Published: 2026-05-15T13:00:00Z Content type: article Language: en Sources: [Amazon Science homepage]() Topics: [Language models](), [Training AI Models](), [web applications]() Tags: [architecture](), [chinchilla-scaling-law](), [conversational-ai](), [generative-ai](), [grouped-query-attention](), [hyperparameter-optimization](), [iclr-2026](), [inference](), [inference-efficiency](), [large-language-models](), [large-language-models-llms](), [llm-optimization](), [llms](), [machine-learning](), [model-architecture](), [network-architectures](), [scaling-laws](), [training](), [transformer-architecture]() ### AI overview The article presents scaling laws that connect LLM architectural choices to the tradeoff between accuracy and efficiency. It describes how these choices can improve inference throughput without reducing accuracy. ### Source excerpt A new scaling law that relates particular architectural choices to loss helps identify models that improve throughput by up to 47% with no loss of accuracy. ## A guide to Transformer architecture in modern language models DevFeed: [A guide to Transformer architecture in modern language models]() Original publisher: [Read original article]() Author: Ashish Pratap Singh Published: 2026-05-14T04:15:11Z Content type: tutorial Language: en Sources: [AlgoMaster Newsletter]() Topics: [Transformer architecture](), [LLMs](), [Transformer]() Tags: [architecture-pattern](), [better](), [deep-dive](), [layer](), [llms](), [model](), [parallel](), [performance](), [semantics](), [sequence](), [syntax](), [tokens](), [transformer-architecture](), [transformers]() ### AI overview This tutorial explains the Transformer architecture, including its original encoder-decoder design for translation and the decoder-only variant used for modern language generation. It describes decoder components such as masked multi-head self-attention, feed-forward networks, layer normalization, and residual connections, and introduces the Pre-LayerNorm pattern. ### Source excerpt A single 2017 research paper changed the future of AI forever and gave rise to multiple unicorn companies. ## Catalyzing scientific impact through global partnerships and open resources DevFeed: [Catalyzing scientific impact through global partnerships and open resources]() Original publisher: [Read original article]() Published: 2026-05-01T16:37:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Open Source](), [datasets](), [Google](), [Transformer architecture]() Tags: [apis](), [architecture](), [australia](), [collaboration](), [community](), [data-mining-modeling](), [developers](), [general-science](), [global](), [google](), [health-bioscience](), [india](), [japan](), [open](), [open-source](), [open-source-models-datasets](), [partnerships](), [publications](), [research](), [science](), [software](), [technology](), [transformer-architecture]() ### AI overview Google Research describes an open-science approach centered on responsible research, open-source software, open-access datasets, and partnerships with scientific organizations and consortia worldwide. It highlights shared technologies and datasets, including Transformer architecture and specialized models used across fields such as medicine, genomics, neuroscience, climate, and energy. ### Source excerpt Data Mining & Modeling ## Granite 4.1 LLMs: How They're Built DevFeed: [Granite 4.1 LLMs: How They're Built]() Original publisher: [Read original article]() Author: Yousaf Shah Published: 2026-04-29T15:01:48Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [LLMs](), [ibm](), [long-context](), [Fine-tuning](), [grpo](), [Training AI Models](), [Transformer architecture](), [Data Quality]() Tags: [apache](), [fine-tuning](), [grpo](), [ibm](), [learning](), [llms](), [long-context](), [tokens](), [training](), [transformer-architecture]() ### AI overview The article introduces Granite 4.1, IBM's family of dense decoder-only LLMs in 3B, 8B, and 30B sizes. It describes their five-stage training process, which uses about 15 trillion tokens, data-quality refinement, long-context extension up to 512K tokens, supervised fine-tuning, and reinforcement learning with on-policy GRPO and DAPO loss. The models use a dense transformer architecture and are released under the Apache 2.0 license. ### Source excerpt Authors: Granite Team, IBM TL;DR -- Granite 4.1 is a family of dense, decoder-only LLMs (3B, 8B, and 30B) trained on ~15T tokens using a multi-stage pre-training pipeline, including long-context extension of up to 512K tokens. The models are further refined with supervised fine-tuning on ~4.1M high-quality curated samples and reinforcement learning via on-policy GRPO with DAPO loss (Yu et al., 2025). ## D4RT: Teaching AI to see the world in four dimensions DevFeed: [D4RT: Teaching AI to see the world in four dimensions]() Original publisher: [Read original article]() Author: Guillaume Le Moing; Mehdi S. M. Sajjadi Published: 2026-01-16T10:39:00Z Content type: article Language: en Sources: [Google DeepMind News]() Topics: [AI, ML & Data Engineering]() Tags: [3d](), [ai](), [ai-models](), [architecture](), [framework](), [real-time](), [research](), [robotics](), [transformer-architecture](), [video]() ### AI overview D4RT is a unified AI model for reconstructing and tracking dynamic 3D scenes over time from video. It uses an encoder-decoder Transformer and a query-based mechanism, with claimed efficiency gains for real-time robotics and augmented-reality uses. ### Source excerpt D4RT: Unified, efficient 4D reconstruction and tracking up to 300x faster than prior methods. ## Introducing Gemma 3n: The developer guide DevFeed: [Introducing Gemma 3n: The developer guide]() Original publisher: [Read original article]() Author: Omar Sanseviero; Ian Ballantyne Published: 2025-10-25T17:54:47Z Content type: release Language: en Sources: [Google DeepMind News]() Topics: [Local AI]() Tags: [ai](), [developer](), [gemma](), [hugging-face](), [inference](), [llama-cpp](), [on-device](), [on-device-ai](), [release](), [transformer-architecture]() ### AI overview Gemma 3n is released as a mobile-first, multimodal model for on-device AI. The guide introduces its MatFormer architecture, pre-extracted E4B and E2B models, and tooling for fine-tuning and deployment. ### Source excerpt Gemma 3n is designed for the developer community that helped shape Gemma. ## How I learn about generative AI DevFeed: [How I learn about generative AI]() Original publisher: [Read original article]() Author: Pamela Fox (noreply@blogger.com) Published: 2025-08-19T05:59:00Z Content type: opinion Language: en Sources: [Pamela Fox]() Topics: [Generative AI](), [AI Engineering](), [LLM evaluation / benchmarking](), [web applications](), [LLMs](), [Retrieval Augmented Generation (RAG)](), [Python](), [PyTorch](), [AI-assisted coding](), [Transformer](), [Deep learning](), [Language models]() Tags: [ai](), [ai-assisted-coding](), [ai-engineering](), [deep-learning](), [evaluation](), [generative-ai](), [llm](), [openai](), [python](), [pytorch](), [transformer-architecture]() ### AI overview The author shares the books, videos, newsletters, communities, and blogs they used to learn generative AI. The resources cover AI engineering, building large language models with Python and PyTorch, neural networks, model evaluation, retrieval-augmented generation, and AI-assisted coding. ### Source excerpt I do not consider myself an expert in generative AI, but I now know enough to build full-stack web applications on top of generative AI models, evaluate the quality of those applications, and decide whether new models or frameworks will be useful. These are the resources that I personally used for getting up to speed with generative AI. AI foundation Let's start first with the long-form content: books and videos that gave me a more solid foundation. AI Engineering By Chip Huyen This book is a fantastic high-level overview of the AI Engineering industry from an experienced ML researcher. I recommend that everybody read this book at some point in your learning journey. Despite Chip's background in ML, the book is very accessible - no ML background is needed, though a bit of programming with LLMs would be a good warm-up for the book. I loved how Chip included both research and industry insights, and her focus on the need for evaluation in the later chapters. Please, read this book! Build a Large Language Model By Sebastian Raschka This book is a deep dive into building LLMs from scratch using Python and Pytorch, and includes a GitHub repository with runnable code. I found it helpful to see that LLMs are all about matrix manipulation, and to wrap my head around how the different layers in the LLM architecture map to matrices. I recommend it to Python developers who want to understand concepts like the transformer architecture, or even just common LLM parameters like temperature and top p. If you're new to Pytorch, this book thankfully includes an intro in the appendix, but I also liked the Deep Learning with PyTorch book. Zero to Hero By Andrej Karpathy This video series builds neural networks from scratch, entirely in Jupyter notebooks. Andrej is a fantastic teacher, and has a great way of explaining complex topics. Admittedly, I have not watched every video from start to finish, but every time I do watch a video from Andrej, I learn so much. Andrej also gives great ta ## AI's capabilities, uncertain trajectory, and potential economic consequences DevFeed: [AI's capabilities, uncertain trajectory, and potential economic consequences]() Original publisher: [Read original article]() Published: 2025-08-13T15:59:56Z Content type: opinion Language: en Sources: [Antirez]() Topics: [Artificial Intelligence](), [Large Language Model](), [Transformer architecture](), [systems]() Tags: [ai](), [future](), [investors](), [llms](), [models](), [science](), [transformer-architecture]() ### AI overview This opinion article examines AI's progress in language understanding, programming, and bug finding while emphasizing uncertainty about future advances. It also considers how greater AI independence could challenge labor markets, service businesses, and the concentration of intelligence providers. ### Source excerpt Regardless of their flaws, AI systems continue to impress with their ability to replicate certain human skills. Even if imperfect, such systems were a few years ago science fiction. It was not even clear that we were so near to create machines that could understand the human language, write programs, and find bugs in a complex code base: bugs that escaped the code review of a competent programmer. Since LLMs and in general deep models are poorly understood, and even the most prominent experts in the field failed miserably again and again to modulate the expectations (with incredible errors on both sides: of reducing or magnifying what was near to come), it is hard to tell what will come next. But even before the Transformer architecture, we were seeing incredible progress for many years, and so far there is no clear sign that the future will not hold more. After all, a plateau of the current systems is possible and very credible, but it would likely stimulate, at this point, massive research efforts in the next step of architectures. However, if AI avoids plateauing long enough to become significantly more useful and independent of humans, this revolution is going to be very unlike the past ones. Yet the economic markets are reacting as if they were governed by stochastic parrots. Their pattern matching wants that previous technologies booms created more business opportunities, so investors are polarized to think the same will happen with AI. But this is not the only possible outcome. We are not there, yet, but if AI could replace a sizable amount of workers, the economic system will be put to a very hard test. Moreover, companies could be less willing to pay for services that their internal AIs can handle or build from scratch. Nor is it possible to imagine a system where a few mega companies are the only providers of intelligence: either AI will be eventually a commodity, or the governments would do something, in such an odd economic setup (a setup where a single ## Diffusers welcomes Stable Diffusion 3.5 Large DevFeed: [Diffusers welcomes Stable Diffusion 3.5 Large]() Original publisher: [Read original article]() Author: YiYi Xu; Aryan V S; Dhruv Nair; Sayak Paul; Linoy Tsaban; Apolinário from multimodal AI art; Alvaro Somoza; Aritra Roy Gosthipaty Published: 2024-10-22T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [diffusers](), [stable-diffusion](), [quantization](), [Transformer architecture](), [Optimization](), [Artificial Intelligence](), [Open Source]() Tags: [artificial-intelligence](), [diffusers](), [diffusion](), [guide](), [inference](), [memory-optimization](), [open-source](), [quantization](), [sd3-5](), [stable-diffusion](), [transformer-architecture]() ### AI overview Hugging Face introduces Stable Diffusion 3.5 Large, providing an 8B base model and a timestep-distilled variant for few-step image generation. The article explains how to use the models with Diffusers for inference and training, including gated access, memory optimization, and quantized execution for lower-memory systems. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Fine-tuning Florence-2 - Microsoft's Cutting-edge Vision Language Models DevFeed: [Fine-tuning Florence-2 - Microsoft's Cutting-edge Vision Language Models]() Original publisher: [Read original article]() Author: Andres Marafioti; merve; Piotr Skalski Published: 2024-06-24T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Fine-tuning](), [vlm](), [Computer vision](), [datasets](), [Embeddings]() Tags: [collaboration](), [community](), [computer-vision](), [dataset](), [datasets](), [fine-tuning](), [object-detection](), [ocr](), [open-source](), [research](), [transformer-architecture](), [vision](), [vqa]() ### AI overview The article explains how to fine-tune Microsoft's Florence-2 vision-language model for DocVQA. It describes the model's sequence-to-sequence architecture, its large FLD-5B pre-training dataset, prompting experiments, and evaluation using Levenshtein similarity. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Unlocking Longer Generation with Key-Value Cache Quantization DevFeed: [Unlocking Longer Generation with Key-Value Cache Quantization]() Original publisher: [Read original article]() Author: Raushan Turganbay Published: 2024-05-16T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [quantization](), [LLMs](), [long-context](), [text-generation](), [Transformer architecture](), [Language models]() Tags: [efficiency](), [generation](), [llm](), [llms](), [long-context](), [memory](), [quantization](), [text-generation](), [transformer-architecture]() ### AI overview The article explains how key-value cache quantization reduces memory usage during long-context text generation in large language models, while offering trade-offs between memory efficiency, generation speed, and output quality. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Transformers for software engineers DevFeed: [Transformers for software engineers]() Original publisher: [Read original article]() Author: Nelson Elhage Published: 2022-04-01T20:00:00Z Content type: tutorial Language: en Sources: [Nelson Elhage]() Topics: [Transformer architecture](), [Programming](), [Software Engineering](), [Reverse Engineering](), [Transformers]() Tags: [anthropic](), [architecture](), [codex](), [deep-learning](), [gpt-3](), [model-architecture](), [programming](), [reverse-engineering](), [software-engineering](), [storm](), [transformer-architecture](), [transformers]() ### AI overview This tutorial explains Transformer architecture for software engineers, using software engineering and programming perspectives to discuss how GPT-style Transformer models work. It also connects the architecture to interpretability and reverse-engineering efforts. ### Source excerpt Ever since its introduction in the 2017 paper, Attention is All You Need, the Transformer model architecture has taken the deep-learning world by storm. Initially introduced for machine translation, it has become the tool of choice for a wide range of domains, including text, audio, video, and others. Transformers have also driven most of the massive increases in model scale and capability in the last few years. OpenAI's GPT-3 and Codex models are Transformers, as are DeepMind's Gopher models and many others.