# Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines

DevFeed: [Leveraging Spark 3 and NVIDIA's GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines](<https://devfeed.tech/articles/leveraging-spark-3-and-nvidia-s-gpus-to-reduce-cloud-cost-by-up-to-70-for-big-data-pipelines-31935.md>)

Original publisher: [Read original article](<https://medium.com/paypal-tech/leveraging-spark-3-and-nvidias-gpus-to-reduce-cloud-cost-by-up-to-70-for-big-data-pipelines-e0bc02ec4f88?source=rss----6423323524ba---4>)

Author: Ilay Chen

Published: 2024-02-21T16:42:14Z

Content type: tutorial

Language: en

Sources: [PayPal Technology](<https://devfeed.tech/sources/paypal-technology.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [RAPIDS](<https://devfeed.tech/topics/rapids.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [migration](<https://devfeed.tech/topics/migration.md>), [upgrade](<https://devfeed.tech/topics/upgrade.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [migration](<https://devfeed.tech/tags/migration.md>), [rapids](<https://devfeed.tech/tags/rapids.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

## AI overview

A PayPal engineering blog explains how upgrading from Apache Spark 2 to Spark 3 and migrating workloads to GPU clusters with NVIDIA Spark RAPIDS can accelerate selected big-data processing tasks and potentially reduce cloud costs by up to 70%. It covers the migration, parameter tuning, challenges, and reported benefits.

## Source excerpt

By Ilay Chen and Tomer Akirav At PayPal, hundreds of thousands of Apache Spark jobs run on an hourly basis, processing petabytes of data and requiring a high volume of resources. To handle the growth of machine learning solutions, PayPal requires scalable environments, cost awareness and constant innovation. This blog explains how Apache Spark 3 and GPUs can help enterprises potentially reduce Apache Spark's jobs cloud costs by up to 70% for big data processing and AI applications. Our journey will begin with a brief introduction of Spark RAPIDS -- Apache Spark's accelerator that leverages GPUs to accelerate processing via the RAPIDS libraries. We will then review PayPal's CPU-based Spark 2 application, our upgrade to Spark 3 and its new capabilities, explore the migration of our Apache Spark application to a GPU cluster, and how we tuned Spark RAPIDS parameters. We will then discuss some challenges we encountered and the benefits of the updates. Libra scales in the cloud, generated by AIBackground GPUs are everywhere, and their parallelism characteristics are perfect for processing AI and graphics applications, among other things. For those unfamiliar: what makes GPUs different from CPUs, computation-wise, is that CPUs have a limited amount of very strong cores, whereas GPUs have thousands, or even tens of thousands or more, relatively weak cores that work together very well. PayPal has been leveraging GPUs to train models for some time now, and so we decided to evaluate if the parallelism of the GPU can be helpful with processing big data applications based on Apache Spark. In our research, we encountered NVIDIA's Spark RAPIDS open-source project. It has many purposes, however we focused on Spark RAPIDS's cost reduction potential, because enterprises like PayPal spend lots of money on running Spark jobs in the cloud. Using Spark with GPUs isn't common in the industry yet, but according to our findings as described in this blog, the potential benefits could be enorm