# Fine-tuning a DistilBERT classifier with numerical and text inputs

DevFeed: [Fine-tuning a DistilBERT classifier with numerical and text inputs](<https://devfeed.tech/articles/fine-tuning-a-distilbert-classifier-with-numerical-and-text-inputs-30004.md>)

Original publisher: [Read original article](<https://engineering.freeagent.com/2026/04/10/fine-tuning-a-distilbert-classifier-with-numerical-and-text-inputs/>)

Author: Paloma Jol

Published: 2026-04-10T10:04:33Z

Content type: tutorial

Language: en

Sources: [FreeAgent](<https://devfeed.tech/sources/freeagent.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [foundation-models](<https://devfeed.tech/topics/foundation-models.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Kaggle](<https://devfeed.tech/topics/kaggle.md>), [scikit-learn](<https://devfeed.tech/topics/scikit-learn.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data](<https://devfeed.tech/tags/data.md>), [data-ml](<https://devfeed.tech/tags/data-ml.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [kaggle](<https://devfeed.tech/tags/kaggle.md>), [llms](<https://devfeed.tech/tags/llms.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [performance](<https://devfeed.tech/tags/performance.md>), [text-classification](<https://devfeed.tech/tags/text-classification.md>), [training-data](<https://devfeed.tech/tags/training-data.md>), [validation](<https://devfeed.tech/tags/validation.md>)

## AI overview

This tutorial explains how to fine-tune a DistilBERT text classifier while incorporating a numerical feature into the same network. It uses the Kaggle wine reviews dataset, combining review descriptions with price and preprocessing the data for training, validation, and testing.

## Source excerpt

Text classification is often done through fine-tuning of a pretrained foundation model with domain-specific data. In FreeAgent we use transformer based models to automatically classify incoming bank transactions. Specifically we use a DistilBERT model that is fine-tuned on hundreds of millions of bank transactions with customer-labelled accounting categories. The model inputs are currently text-based, built from a combination of bank transaction descriptions and amounts. In this post we describe an approach to fine-tuning the DistilBERT model and training the classifier including the numerical amount feature as a single network.