# Implementing Multi-GPU Distributed Training for Stitch Fix's Personalized Recommendations

DevFeed: [Implementing Multi-GPU Distributed Training for Stitch Fix's Personalized Recommendations](<https://devfeed.tech/articles/accelerating-ai-implementing-multi-gpu-distributed-training-for-personalized-recommendations-29344.md>)

Original publisher: [Read original article](<https://multithreaded.stitchfix.com/blog/2023/06/08/distributed-model-training/>)

Published: 2023-06-08T09:00:00Z

Content type: article

Language: en

Sources: [Stitch Fix](<https://devfeed.tech/sources/stitch-fix.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [recommendations](<https://devfeed.tech/topics/recommendations.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [sharding](<https://devfeed.tech/topics/sharding.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [sharding](<https://devfeed.tech/tags/sharding.md>)

## AI overview

This Stitch Fix engineering article explains how the company implemented multi-GPU distributed training for its Client Time Series Model (CTSM), a PyTorch-based model used in personalized recommendations. It describes sharding training data across GPUs and training mini-batches in parallel to reduce training time, along with the surrounding retraining and deployment workflow.

## Source excerpt

Stitch Fix uses a cutting-edge multi-tiered recommender system stack to personalize styling recommendations at scale. This stack comprises several critical components, including feature generation, scoring, ranking, and inventory optimization techniques. Our scoring module is based on the Client Time Series Model (CTSM) which is an award winning novel sequence based model that uses temporally masked encoders. CTSM is built using PyTorch, and was initially trained on a single Graphics Processing Unit (GPU) instance. Since we first put this model into production last year, we have launched several updates to the model that improved its performance. Many of these improvements involved adding new features or increasing the time window of our training data. As a result, the model training time increased significantly, making it harder for us to iterate quickly and get feedback on new ideas we want to try for improving the model. We needed a way to reduce the model training time. This blog delves into the steps we followed to overcome this challenge and our journey to implement multi-GPU distributed model training for CTSM. By sharding the training data across multiple GPUs and training multiple mini-batches in parallel, we aimed to achieve significant reductions in training time. We present empirical results showcasing the observed reduction in training time when we scaled up resources from 1 to N GPUs, and share some future directions we are considering in our continued effort to speed up model training. Model Training Workflow The scores generated by CTSM are leveraged by multiple downstream services to get insight into what items a client is likely to purchase. The model is retrained at a regular cadence to ensure that it is using the most updated information about each client when making predictions and does not degrade in its performance. We leverage configuration driven machine learning pipelines to set up a Directed Acyclic Graph (DAG) that automatically retrains