# headspace-engineering

Published articles for headspace-engineering.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Real-Time Machine Learning at Headspace

DevFeed: [Real-Time Machine Learning at Headspace](<https://devfeed.tech/articles/real-time-machine-learning-at-headspace-24571.md>)

Original publisher: [Read original article](<https://headspace.medium.com/infrastructure-design-for-real-time-machine-learning-inference-e140793d6741?source=rss-3da90e297190------2>)

Author: Headspace

Published: 2021-08-16T16:53:19Z

Content type: article

Language: en

Sources: [Stories by Headspace on Medium](<https://devfeed.tech/sources/stories-by-headspace-on-medium.md>)

Topics: [Machine Learning & Artificial Intelligence](<https://devfeed.tech/topics/machine-learning-artificial-intelligence.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [recommendations](<https://devfeed.tech/topics/recommendations.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon SageMaker AI](<https://devfeed.tech/topics/amazon-sagemaker-ai.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>)

Tags: [android](<https://devfeed.tech/tags/android.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data](<https://devfeed.tech/tags/data.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [headspace-engineering](<https://devfeed.tech/tags/headspace-engineering.md>), [inference](<https://devfeed.tech/tags/inference.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [sqs](<https://devfeed.tech/tags/sqs.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

This article describes Headspace's infrastructure for real-time machine learning inference and personalized content recommendations. It explains how the team uses modular Publishing, Receiver, Orchestration, and Serving layers with Apache Spark Structured Streaming on Databricks, AWS SQS, Lambda, and SageMaker.

### Source excerpt

Author: Yu Chen, Senior Software Engineer / Co-author: Koyuki Nakamori, Senior Engineering Manager Introduction Data is often most valuable when it can be immediately leveraged to make decisions in the moment, but traditionally consumer data is ingested, transformed, persisted, and sits dormant for lengthy periods of time before being used by machine learning and analytics teams. Headspace's core products are iOS, Android, and web-based apps that focus on improving the health and happiness of its users through mindfulness, meditation, sleep, exercise, and focus content. Machine learning models are core to our user experiences by offering recommendations that engage our users with new relevant, personalized content that builds consistent habits in their lifelong journey. Finding a way to leverage user data to generate real-time insights and decisions means that consumer-facing products like the Headspace app can dramatically shorten the end-to-end user feedback loop: actions that users perform a few seconds, or minutes ago can be incorporated in generating a more relevant, personalized, and context-specific content recommendation for the user. This means our ML models could incorporate dynamic features that update throughout the course of a user's day, or even an individual session: Current session bounce rates for sleep content Semantic embeddings for recent user search terms (if a user recently searched for "preparing for big exam", the ML model can assign more weight to Focus-themed meditations) Users' biometric data (i.e., if step counts and heart rate are increasing over the last 10 minutes, we can recommend Move or Exercise content) The Headspace Machine Learning team architected a solution by decomposing the infrastructure systems into modular Publishing, Receiver, Orchestration, and Serving layers, leveraging Apache Spark Structured Streaming on Databricks, AWS SQS, Lambda, and Sagemaker to deliver real-time inference capabilities for our machine learning mod