# Announcing pg\_tiktoken: A Postgres Extension for Fast BPE Tokenization

DevFeed: [Announcing pg\_tiktoken: A Postgres Extension for Fast BPE Tokenization](<https://devfeed.tech/articles/announcing-pg-tiktoken-a-postgres-extension-for-fast-bpe-tokenization-4970.md>)

Original publisher: [Read original article](<https://neon.com/blog/announcing-pg-tiktoken-a-postgres-extension-for-fast-bpe-tokenization>)

Author: Stas Kelvich

Published: 2023-03-14T17:04:34Z

Content type: release

Language: en

Sources: [Blog -- Neon Docs](<https://devfeed.tech/sources/blog-neon-docs.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [extension](<https://devfeed.tech/tags/extension.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [natural-language-processing](<https://devfeed.tech/tags/natural-language-processing.md>), [openai](<https://devfeed.tech/tags/openai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [release](<https://devfeed.tech/tags/release.md>), [speed](<https://devfeed.tech/tags/speed.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

## AI overview

Neon announces pg_tiktoken, a Postgres extension that performs BPE tokenization and token counting inside database queries. It wraps OpenAI's tokenizer and can help fit stored message histories within model token limits.

## Source excerpt

We're excited to announce the release of the pg_tiktoken extension on Neon. This new Postgres extension provides fast and efficient tokenization using the BPE (Byte Pair Encoding) algorithm. pg_tiktoken is a wrapper around OpenAI's tokenizer, known for its speed and performance i...