# How Target Performance Tunes Machine Learning Applications

DevFeed: [How Target Performance Tunes Machine Learning Applications](<https://devfeed.tech/articles/how-target-performance-tunes-machine-learning-applications-20400.md>)

Original publisher: [Read original article](<https://target.github.io/data%20science%20and%20engineering/numspark>)

Author: Target Brands, Inc

Published: 2016-09-29T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Filesystems](<https://devfeed.tech/topics/filesystems.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science-and-engineering](<https://devfeed.tech/tags/data-science-and-engineering.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [science](<https://devfeed.tech/tags/science.md>)

## AI overview

The article discusses improving the performance and relevance of machine-learning applications by evaluating matrix multiplication at scale. It describes distributed computing requirements, distributed file storage, scalability, fault tolerance, and the use of Apache Hadoop, Apache Spark, and Apache Mahout.

## Source excerpt

At Target we aim to make shopping more fun and relevant for our guests through extensive use of data - and believe me, we have lots of data! Tens of millions of guests and hundreds of thousands of items lead to billions of transactions and interactions. We regularly employ a number of different machine learning techniques on such large datasets for dozens of algorithms. We are constantly looking for ways to improve speed and relevance of our algorithms and one such quest brought us to carefully evaluate matrix multiplications at scale - since that forms the bedrock for most algorithms. If we make matrix multiplication more efficient, we can speed up most of our algorithms! Before we dig in, let me describe some properties of the landscape we will be working in. First, what do I mean by large scale? A large scale application, at a minimum, will require its computation to be spread over multiple nodes of a distributed computing environment to finish in a reasonable amount of time. These calculations will use existing data that are stored on a distributed file system that provides high-throughput access from the computing environment. Scalability, in terms of storage and compute, should grow as we add to these resources. As the system grows larger and more complex, failures will become more commonplace. Thus, software should be fault-tolerant. Fortunately, there is a lot of existing open-source software that we can leverage to work in such an environment, particularly Apache Hadoop for storing and interacting with our data, Apache Spark as the compute engine, and both Apache Spark and Apache Mahout for applying and building distributed machine learning algorithms. There are many other tools that we can add to the mix as well, but for the purposes of this post we will limit our discussion to these three. With that out of the way, lets dig in! Don't Forget the Basics Begin with good old paper and pencil. Yeah, I know this is about large scale matrix operations that you c