# Transformers for software engineers

DevFeed: [Transformers for software engineers](<https://devfeed.tech/articles/transformers-for-software-engineers-21973.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/transformers-for-software-engineers/>)

Author: Nelson Elhage

Published: 2022-04-01T20:00:00Z

Content type: tutorial

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Transformer architecture](<https://devfeed.tech/topics/transformer-architecture.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Reverse Engineering](<https://devfeed.tech/topics/reverse-engineering.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [anthropic](<https://devfeed.tech/tags/anthropic.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [codex](<https://devfeed.tech/tags/codex.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [gpt-3](<https://devfeed.tech/tags/gpt-3.md>), [model-architecture](<https://devfeed.tech/tags/model-architecture.md>), [programming](<https://devfeed.tech/tags/programming.md>), [reverse-engineering](<https://devfeed.tech/tags/reverse-engineering.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [storm](<https://devfeed.tech/tags/storm.md>), [transformer-architecture](<https://devfeed.tech/tags/transformer-architecture.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

## AI overview

This tutorial explains Transformer architecture for software engineers, using software engineering and programming perspectives to discuss how GPT-style Transformer models work. It also connects the architecture to interpretability and reverse-engineering efforts.

## Source excerpt

Ever since its introduction in the 2017 paper, Attention is All You Need, the Transformer model architecture has taken the deep-learning world by storm. Initially introduced for machine translation, it has become the tool of choice for a wide range of domains, including text, audio, video, and others. Transformers have also driven most of the massive increases in model scale and capability in the last few years. OpenAI's GPT-3 and Codex models are Transformers, as are DeepMind's Gopher models and many others.