# Open Sourcing Transformer Embeddings

DevFeed: [Open Sourcing Transformer Embeddings](<https://devfeed.tech/articles/open-sourcing-transformer-embeddings-24578.md>)

Original publisher: [Read original article](<https://medium.com/headspace-engineering/open-sourcing-transformer-embeddings-3c8cf6f4e540?source=rss-3da90e297190------2>)

Author: Headspace

Published: 2022-11-02T20:56:38Z

Content type: release

Language: en

Sources: [Stories by Headspace on Medium](<https://devfeed.tech/sources/stories-by-headspace-on-medium.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Natural language processing](<https://devfeed.tech/topics/nlp.md>), [Python](<https://devfeed.tech/topics/python.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [embeddings](<https://devfeed.tech/tags/embeddings.md>), [inference](<https://devfeed.tech/tags/inference.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [naturallanguageprocessing](<https://devfeed.tech/tags/naturallanguageprocessing.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pypi](<https://devfeed.tech/tags/pypi.md>), [python](<https://devfeed.tech/tags/python.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [s3](<https://devfeed.tech/tags/s3.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

Headspace Health open-sources transformer-embeddings, an internal Python package, and publishes it to PyPI. The package supports pooled embeddings for phrases, sentences, paragraphs, and documents, with configurable pooling and batch sizes, CPU or GPU execution, and S3 export.

## Source excerpt

Photo by Jr Korpa on UnsplashBy Setu Shah, Staff Machine Learning Engineer The Data Science team at Headspace Health is a heavy user of building, training and production-izing transformer-based NLP models. Depending on the application, our NLP models operate at varying levels of atomicity: token, word, phrase, sentence, paragraph and document. A lot of our models rely on embeddings from off-the-shelf Transformer-based models, while some are fine-tuned for the application we are building for. Similarly, as a part of our work, we often try out different models during exploration, while comparing various pooling methods. About 18 months ago, we discovered the need for a streamlined way to use embeddings across our work for our exploration and production use cases. That is when we started building transformer-embeddings, an internal Python package that made it easy to interact with transformer-based models that are built using PyTorch. Today, we are open-sourcing this package and publishing it to PyPI to make it easy for anyone in the transformer and NLP communities to be able to use aggregated and pooled embeddings for phrases, sentences and paragraphs and documents, at varying levels of aggregation, in any of their applications. You should use this if you want to... Automatically apply tokenization (with the model defaults) before your model's forward pass. Stack outputs from the model into a single, iterable array that map 1:1 with your input. Simplify interactions with any transformer model available on the HuggingFace Model Hub for exploration and inference. Easily apply and compare the impact of different pooling strategies (mean, max, min, pooler) on your downstream tasks. Use your model on CPUs or GPUs, without worrying about if you asked PyTorch to use the right device. Export the model and additional artifacts (custom scikit-learn / tree-based models, model cards, etc.) to S3. Customize batch sizes for different models as you play with them. You should not use t