# feature-engineering Published articles for feature-engineering. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## 🍔🧠 Pinterest's Fix for the Hardest Problem in ML Infra DevFeed: [🍔🧠 Pinterest's Fix for the Hardest Problem in ML Infra]() Original publisher: [Read original article]() Author: Alexandre Zajac Published: 2026-09-14T15:31:30Z Content type: article Language: en Sources: [Hungry Minds]() Topics: [recommendation systems](), [Feature Engineering](), [Inference](), [Streaming](), [configuration](), [Framework](), [data](), [Low Latency]() Tags: [batch](), [configuration](), [data](), [feature-engineering](), [low-latency](), [ml](), [pinterest](), [ranking](), [recommendation-systems](), [streaming]() ### AI overview Pinterest redesigned its user-sequence platform for ranking, retrieval, and recommendation systems by defining signals once and instantiating them consistently across streaming, batch, and serving workloads. The approach uses Python configuration with validated schemas, a shared execution engine, cooperating streaming and batch paths, and columnar time-partitioned storage to improve freshness, completeness, consistency, and operational efficiency. ### Source excerpt PLUS: OpenAI agents beat math 🧮, Test techniques for agents ⚡, Postgres survival guide 📖 ## Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability... DevFeed: [Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability...]() Original publisher: [Read original article]() Author: Manjunath Shettar Published: 2026-09-10T16:12:28Z Content type: article Language: en Sources: [Lyft Engineering - Medium]() Topics: [dataset](), [airflow]() Tags: [airflow](), [data-processing](), [feature-engineering](), [latency](), [lyft](), [offline](), [tea]() ### AI overview Lyft describes rebuilding its Neighborhood Reachability Signals: offline, regional travel-time matrices between geohash-6 cells and their associated neighborhood-center lists. The refresh replaces older static data and is intended to support marketplace pricing, driver guidance, and demand heatmaps, with future work aimed at time-aware travel times. ### Source excerpt Refreshing the Travel-Time Map Behind Lyft's Marketplace: Rebuilding Neighborhood Reachability Signals Every time Lyft calculates pricing to balance a market, nudges a driver toward an under-served pocket of a city, or paints a heatmap of where demand is building, there is a quiet lookup table doing work in the background. It answers a deceptively simple question: how long does it take to get from here to there?, for millions of pairs of places, across hundreds of regions. That lookup table is the Neighborhood Reachability Signal, and for years large parts of it were frozen in a snapshot of the world from 2018-2019. This is the story of how we rebuilt it, why a refresh substantial enough to be worth adopting was what finally moved Pricing to switch, the cleanly positive results that came out of that switch, and where we're taking it next, from one static file per region to time-aware travel times that change with the rhythm of the day. What is a Neighborhood Reachability Signal? A geohash is a compact way of carving the world into a grid of cells. At geohash-6 resolution, each cell is roughly the size of a few city blocks. Slice a region into geohash-6 cells and you get a clean, discrete coordinate system for "neighborhoods" that downstream systems can reason about. The Forecasting & Real-Time Optimization (FORTOP) team produces the Neighborhood Reachability Signals dataset, which consists of two companion files for each region: Neighborhood Reachability Matrix: the estimated travel time, in minutes, between the centers of pairs of geohash-6 cells. Think of it as a sparse origin-to-destination travel-time matrix for a region. Neighborhood Centers: the list of all geohashes that appear in the ETA files for that region, i.e. the "vocabulary" of cells that the marketplace is allowed to talk about. Both files are generated offline on a schedule by an Airflow DAG. They are static in the sense that they are precomputed and shipped, rather than queried live (which is exact ## Planetary prediction engine: Automating global models via Earth AI DevFeed: [Planetary prediction engine: Automating global models via Earth AI]() Original publisher: [Read original article]() Published: 2026-08-27T17:37:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Earth AI](), [Artificial Intelligence](), [data-engineering](), [Google](), [Machine learning](), [AI Development](), [Feature Engineering](), [Model Development]() Tags: [agents](), [ai](), [analytics](), [data-engineering](), [earth-ai](), [evaluation](), [feature-engineering](), [generative-ai](), [google](), [insights](), [machine-intelligence](), [mapping](), [research](), [training](), [validation](), [workflow](), [workflows]() ### AI overview Google Research introduces the Planetary Prediction Engine, an experimental Earth AI capability that autonomously performs geospatial data discovery, cleanup, feature engineering, model training, evaluation, and report generation from natural-language queries. The system targets applications including public health, food security, environmental risk, and socioeconomic analysis, reducing the stated workflow from weeks of manual data engineering to minutes. ### Source excerpt Earth AI ## Data Engineering Weekly #283 DevFeed: [Data Engineering Weekly #283]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-17T02:59:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [Microservice](), [Apache Spark](), [Kafka](), [Feature Engineering](), [audit trail](), [Protocol (disambiguation)]() Tags: [ai](), [anthropic](), [article](), [data-engineering](), [data-quality](), [engineering](), [feature-engineering](), [kafka](), [newsletter](), [pipelines](), [services](), [testing]() ### AI overview Data Engineering Weekly #283 is a newsletter covering data platform fundamentals, multiagent system coordination, payments platform data contracts, financial data quality, declarative data engineering, and cost-efficient export workloads. ### Source excerpt The Weekly Data Engineering Newsletter ## Data Engineering Weekly #281 DevFeed: [Data Engineering Weekly #281]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-03T12:34:40Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Loop Engineering](), [Artificial Intelligence](), [Large Language Model](), [post-training]() Tags: [agent-harness](), [ai](), [data-engineering](), [feature-engineering](), [genai](), [netflix](), [newsletter](), [post-training](), [weekly]() ### AI overview Data Engineering Weekly #281 covers building data platforms, emerging approaches to AI workflow architecture, data modernization, Netflix's GenRec recommendation system, AI infrastructure modernization, and evaluation practices for generative AI at scale. ### Source excerpt The Weekly Data Engineering Newsletter ## GenRec: Towards LLM-Native Recommendation at Netflix DevFeed: [GenRec: Towards LLM-Native Recommendation at Netflix]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-07-30T20:10:15Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [recommendation systems](), [Large Language Model](), [LLM Techniques](), [personalization](), [vllm]() Tags: [data](), [feature-engineering](), [genai](), [large-language-models](), [large-language-models-llms](), [llm](), [llms](), [netflix](), [personalization](), [ranking](), [recommendation-system](), [vllm]() ### AI overview Netflix presents GenRec, an LLM-backed recommendation ranker trained on Netflix-specific data and objectives. It converts user histories, item metadata, and context into text, uses a catalog-aware scoring head, aligns recommendations with long-term member value and business goals, and runs in prefill-only mode on Netflix's LLM serving stack. In a large-scale A/B test, GenRec improved short- and long-term online metrics while using fewer labeled examples and input signals than a mature production ranker. ### Source excerpt Authors: Ying Li, Arjun Rao, Shradha Sehgal Introduction Recommendations sit at the heart of the Netflix experience. Our current production models rely on thousands of hand-crafted features over users, items, and interactions, along with specialized architectures for sequence modeling, feature interactions, and multi-task objectives. This stack has evolved over many years to support diverse content types (movies, series, games, live, podcasts) and product surfaces, but its complexity makes it costly to onboard new use cases: adding a content type or surface can require significant feature engineering, architecture change, infrastructure work, and experimentation. At the same time, large language models (LLMs) are changing how we think about recommendation, as shown by recent work such as PLUM, GLIDE, and OneRec-Think. Their broad world knowledge and strong language understanding make it possible to represent user histories and item metadata directly as text, capture rich relationships in a shared semantic space, and steer recommendations via natural-language prompts. However, off-the-shelf LLMs are still far from production-ready recommenders: they often over-recommend globally popular content, hallucinate out-of-catalog items, ignore business constraints, and provide only limited personalization. To address this, we built GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec shows that an LLM-based ranker can match or exceed a mature production system while relying on far fewer labeled examples and input signals. Figure 1: GenRec pipeline. Raw logs of user history, item metadata, and context are transformed via context engineering into natural-language prompts and fed into the GenRec, which runs on vLLM in prefill-only mode and outputs scores for each catalog item, yielding a recommendation ranking. At a high level, GenRec: Verbalizes user histories, item metadata, and context as text ## Introducing TabFM: A zero-shot foundation model for tabular data DevFeed: [Introducing TabFM: A zero-shot foundation model for tabular data]() Original publisher: [Read original article]() Published: 2026-06-30T10:26:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [foundation-models](), [datasets](), [Machine learning](), [Google](), [Feature Engineering](), [Hyperparameter optimization](), [Language models](), [BigQuery](), [GitHub](), [hugging face](), [AI Inference]() Tags: [bigquery](), [classification](), [data](), [data-management](), [feature-engineering](), [github](), [google](), [hugging-face](), [hyperparameter-optimization](), [machine-intelligence](), [model](), [product](), [zero-shot]() ### AI overview Google Research introduces TabFM, a zero-shot foundation model for tabular-data classification and regression. It frames prediction as in-context learning, reducing the need for dataset-specific training, hyperparameter optimization, and feature engineering, with availability through Hugging Face, GitHub, and BigQuery. ### Source excerpt Data Management ## GenPage: Towards End-to-End Generative Homepage Construction at Netflix DevFeed: [GenPage: Towards End-to-End Generative Homepage Construction at Netflix]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-06-29T13:01:02Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [recommendation systems](), [Large Language Model](), [Machine learning](), [Reinforcement learning](), [Transformer]() Tags: [ai](), [app](), [diversity](), [engineering](), [feature-engineering](), [generation](), [generative](), [large-language-models](), [machine-learning](), [model](), [netflix](), [optimization](), [ranking](), [recommendation-system](), [reinforcement-learning](), [rl]() ### AI overview This Netflix developer article introduces GenPage, a generative approach that uses a single autoregressive model to construct a personalized homepage by generating recommendation rows, entities, and layout together. It describes replacing a multi-stage recommendation pipeline with end-to-end modeling and using reinforcement learning to optimize whole-page rewards, including interactions such as diversity and the balance between rows. ### Source excerpt Authors: Lequn Wang, Jiangwei Pan, and Linas Baltrunas Figure 1. Autoregressive homepage generation. GenPage builds a Netflix homepage one row or entity at a time, each one conditioned on what's already on the page and the user's context.Introduction The Netflix homepage is the first thing users see when they open the app and the primary way they discover content to enjoy. Almost every part of it is personalized, including which rows appear, which entities show up within those rows, and how everything is arranged on the page. Constructing that homepage is a genuinely hard problem. It is not simply producing one ranked list. The homepage is a structured, two-dimensional layout, made up of recommendation rows and the entities within them. Here, an entity can be a movie, show, game, live event, or other recommendable item. Each choice can affect the value of the others. Traditionally, it is built through a complex, multi-stage pipeline, with separate components for candidate generation and ranking at both the row and entity levels. We saw an opportunity to rethink this design. Large language models have shown that a single generative model can perform diverse tasks just by generating a response to a prompt. Inspired by this prompt-response paradigm, we trained a single generative model to build the homepage by directly answering one question: Given everything we know about this user and this request, what homepage should we generate to maximize user satisfaction? We call this approach GenPage. It treats the user history and request context as the prompt, and autoregressively generates the entire homepage as the response (Figure 1). Unlike most generative recommenders, such as TIGER, HSTU, and OneRec, which generate flat ranked lists, GenPage generates the rows, entities, and layout together. This shift is motivated by several goals: End-to-end modeling. A single transformer model that constructs the page from raw input signals can replace a complex multi-stage recommen ## Teaching Sidekick to say no: automated data curation with LLM judge consensus DevFeed: [Teaching Sidekick to say no: automated data curation with LLM judge consensus]() Original publisher: [Read original article]() Author: Shuang Xie Published: 2026-06-15T19:30:00Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [data](), [Large Language Model](), [Fine-tuning](), [AI Development](), [foundation-models](), [Training AI Models](), [Artificial Intelligence](), [Open Source Models & Datasets](), [Open Source]() Tags: [agentic](), [ai](), [analysis](), [customer](), [data](), [feature-engineering](), [fine-tuning](), [llm](), [open-source](), [shopify](), [training]() ### AI overview Shopify Engineering describes how Sidekick's production training data failed to teach refusal behavior because it contained only successful merchant queries. The article presents automated data curation using consensus among LLM judges to identify blind spots and improve an AI assistant built from an outer planner and specialized skill models. ### Source excerpt Production training data only captures successful queries; it can't teach a model when to say no. We built an automated curation pipeline using LLM judge consensus to close that gap. ## Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use DevFeed: [Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-05-21T16:01:00Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [data](), [recommendation systems](), [datasets](), [Machine learning](), [Embeddings](), [Feature Engineering](), [Inference](), [Latency](), [systems]() Tags: [data](), [data-infrastructure](), [datasets](), [embeddings](), [engineering](), [feature-engineering](), [inference](), [latency](), [machine-learning](), [pinterest](), [production](), [recommendation-system](), [recommendation-systems](), [retrieval](), [systems](), [train](), [transformers]() ### AI overview Pinterest describes a redesign of its user-sequence platform for ranking, retrieval, and recommendation workloads. The article explains how enriched event sequences support training datasets, offline analysis, online inference, and latency-sensitive production use cases, with goals of reducing cost, improving extensibility, and simplifying debugging. ### Source excerpt Authors (listed alphabetically) Ads Feature Engineering Infra team: Ajay Venkatakrishnan, Le Zhang Core ML Infra team: Eric Shang, Pihui Wei ML Data team: Connor Votroubek, Yi He User Understanding team: Camilo Munoz, Simin Li If you work on ranking, retrieval, or recommendation systems, you've probably asked for some version of the same thing: "Give me the last N meaningful actions this user took, with the right enrichments, in a format that's easy to train and serve ML models." On paper, that sounds simple. In practice, "user sequences" often become one of the most expensive and fragile parts of the ML data stack. They end up powering everything from training datasets to offline analysis and online inference, so they need to be fresh and complete at the same time. They must remain consistent as you add new events and enrichments. And they have to do all of this while serving latency-sensitive production workloads. This article walks through how we redesigned our user-sequence platform to make these sequences cheaper to run, faster to extend, and easier to debug, while still supporting demanding production use cases. What We Mean by "User Sequence" In this context, a user sequence is an ordered list of recent, relevant events for a user, along with the enrichments (signals) attached to each event. Here, enrichments mean all the extra signals we attach to raw events, so they're useful for models: embeddings (for example, Pin or query representations), contextual features (such as surface, device, or country), and derived attributes or counters that describe how the user interacted with a piece of content over time. A concrete example helps. Imagine a sequence made up of the last 500 engagements a user had with Pinterest Pins. Each event in that sequence might carry a timestamp, an action type, the surface where the action occurred, and a handful of embedding features or categorical attributes. As a data primitive, user sequences are powerful. They capture temporal b ## How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update DevFeed: [How Whatnot's Hourly ML Feature Pipeline Safely Handled a Failed Update]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-05-07T16:01:01Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [Machine learning](), [recommendation systems](), [datasets](), [data](), [real-time](), [Processes]() Tags: [ai](), [data](), [data-engineering](), [feature](), [feature-engineering](), [feature-store](), [machine-learning](), [ml](), [pipeline](), [ranking](), [real-time](), [recommendation-system](), [recommendations](), [run]() ### AI overview Whatnot describes how its hourly ML feature pipeline handled a bad pull request that caused a feature group to produce zero rows. The online feature store served last-known-good features for up to two days, anomaly monitors detected the row-count drop, and the team reverted the change before the freshness gap became significant. The post shares lessons about safe degradation and operating frequently updated recommendation features. ### Source excerpt David Kawashima, Miguel Fernandez-Montes Cuberta, Ford Bohrmann | Discovery A bad PR went out just as our hourly feature pipeline was about to run. The pipeline was responsible for recomputing fresh user behavior signals used in livestream ranking on the home feed. These features capture recent user intent: watches, bids, orders aggregated across users, sellers, and categories. A missed feature update means that the system stops responding to user signals immediately. This PR release caused one of our feature groups to suddenly produce zero rows ... but nothing broke! Even though a critical feature group for the model stopped updating, the online feature store continued serving the last-known-good features, buffered by a 2-day TTL and nothing broke explicitly. Our anomaly monitors went off immediately, flagging the drop in row counts. The team identified the bad change and reverted it before the freshness gap grew large enough to matter. No corrupted data reached production. The system absorbed the failure and degraded safely. This graceful degradation didn't happen by accident; it was the result of hardened processes from years of lived experience, seeing the way these systems can degrade in obvious or subtle ways. In this post, we share some of those lessons learned. The system: hourly feature pipeline Whatnot is a highly dynamic marketplace where buyers' preferences change at a moment's notice. A buyer might research a laptop, join a football-card break a few hours later, and buy chocolate before logging off for the night. Daily feature updates capture long-term preferences but miss intra-day shifts. To keep up with these rapidly changing buyer interests (in 2025) we moved from daily to hourly updates for features that power our ML feed ranking models. Fresher features are a critical component for capturing the live nature of the Whatnot marketplace, where the inventory available to purchase changes in real time. Moving from daily to hourly updates was a meaningful ## The generative recommender behind Shopify's commerce engine DevFeed: [The generative recommender behind Shopify's commerce engine]() Original publisher: [Read original article]() Author: Yang Liu Published: 2026-02-25T16:04:54Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [Temporal data](), [model-deployment](), [real-time](), [Feature Engineering]() Tags: [architecture](), [engineering](), [feature-engineering](), [generative](), [latency](), [model](), [production](), [real-time](), [scale](), [shopify](), [train](), [training]() ### AI overview Shopify describes a generative recommender that treats buyer journeys as raw event sequences. An autoregressive, causally masked model predicts next products or ads while meeting real-time production constraints at Shopify's scale. ### Source excerpt Treating buyer journeys as sequences instead of simplified signals, and building a model fast enough to serve at scale. ## Engineering Features at Scale: Inside the Darwin Feature Store DevFeed: [Engineering Features at Scale: Inside the Darwin Feature Store]() Original publisher: [Read original article]() Author: Dream Blog Published: 2026-01-27T15:38:25Z Content type: article Language: en Sources: [Dream11 Engineering]() Topics: [Feature Engineering](), [Low Latency](), [Apache Cassandra](), [Machine learning](), [Open Source]() Tags: [cassandra](), [data](), [engineering](), [feature-engineering](), [feature-store](), [inference](), [kafka](), [latency](), [machine-learning](), [offline](), [open-source](), [pipelines](), [software-engineering](), [tech]() ### AI overview Dream Horizon's Darwin Feature Store is an open-source, low-latency feature platform designed to support real-time machine learning and offline training at Dream11. The article describes the challenges that led to its development, including batch-only pipelines, manual schema changes, and scaling issues. It reports serving more than 200 million feature requests per minute with p99 read latency below 5 milliseconds and no production incidents over the preceding year. ### Source excerpt By Mohit Jain and Ujjwal Bagrania Dream Horizon, our open-source effort to make Dream11's battle-tested tech available to every developer, brings you the Darwin Feature Store -- a unified, low-latency feature platform built to power real-time ML at scale, and shaped to help teams build, manage, and trust features in production. Explore the Darwin Feature Store here. At Dream11, data and ML have always powered how millions of users experience sports in real time. From personalisation and relevance to ensuring efficiency under peak match traffic, ML sits deep in the critical path of the product. But models are only half the story. The real challenge is managing features -- how they're defined, versioned, and accessed consistently across training and live traffic, especially under peak sports pressure. And at the scale we operate at, feature access doesn't mean the occasional lookup; it means hundreds of millions of requests per minute, spanning both real-time inference and offline training pipelines. As Dream11 grew, feature engineering stopped being something we could manage with pipelines and tables, and became infrastructure that everything else depended on. That shift forced us to build the Darwin Feature Store: a system designed not just to tackle sports-scale traffic, but to make feature engineering predictable, reliable, and developer-friendly in production. Today, that translates into serving 200M+ feature requests per minute, delivering p99 read latencies under 5 ms, and doing so reliably through the most demanding live sports moments, without a single production incident over the last year. In this post, we'll walk through how we built the Feature Store, and what it takes to serve features reliably. The Early Days: Batch-Only, Fragile, and Manual Before the Darwin Feature Store existed, our feature engineering was mostly a few pipelines, a lot of Cassandra tables, and an increasing number of engineers building ML models. Here's what the first version looked li ## Platform Thinking: Three Views from Two Sigma Leaders DevFeed: [Platform Thinking: Three Views from Two Sigma Leaders]() Original publisher: [Read original article]() Author: Emily Majewski Published: 2025-10-23T14:49:45Z Content type: article Language: en Sources: [Two Sigma Engineering]() Topics: [data-engineering](), [data](), [Large Language Model](), [Development](), [Feature Engineering](), [Artificial Intelligence](), [BigQuery](), [CI/CD](), [AI research agents](), [Open Source Models & Datasets]() Tags: [ai](), [ai-agents](), [bigquery](), [ci-cd](), [data](), [data-engineering](), [datasets](), [feature-engineering](), [llms](), [open-source](), [platform](), [post]() ### AI overview Two Sigma leaders describe the company's platform-oriented approach to data, technology, and management. The article highlights parallel delivery of raw and curated datasets, BigQuery and CI/CD practices, reusable data contracts, LLM-assisted feature engineering, controls for temporal leakage, and possible uses of multiple AI agents in quantitative research. ### Source excerpt The post Platform Thinking: Three Views from Two Sigma Leaders appeared first on Two Sigma. ## Optimizing AI Code for Performance DevFeed: [Optimizing AI Code for Performance]() Original publisher: [Read original article]() Author: admin Published: 2024-05-27T13:47:23Z Content type: tutorial Language: en Sources: [Posts on Chris Banes]() Topics: [AI Development](), [Optimization](), [Code](), [Feature Engineering](), [data]() Tags: [ai](), [batching](), [code](), [feature-engineering](), [format](), [machine-learning](), [optimization](), [pandas](), [parquet](), [performance](), [uncategorized]() ### AI overview This guide explains techniques for optimizing AI and machine learning code for faster execution and lower computing costs. It covers data loading and preprocessing, algorithm selection, hardware acceleration, distributed training, and batching. ### Source excerpt AI code optimization is about getting almost everything out of your algorithms and models. It's about making sure that your AI projects run smoothly, quickly, and cost-effectively. In this guide, we'll dive into the world of AI code optimization, exploring different techniques and best practices that can help you speed up your machine learning projects. [...] ## Declarative Feature Engineering at PayPal DevFeed: [Declarative Feature Engineering at PayPal]() Original publisher: [Read original article]() Author: Marina Lyan Published: 2023-12-11T15:56:51Z Content type: tutorial Language: en Sources: [PayPal Technology]() Topics: [Feature Engineering](), [Machine learning](), [data](), [Requirements]() Tags: [declarative-programming](), [engineering](), [feature](), [feature-engineering](), [fraud](), [machine-learning](), [paypal](), [production](), [real-time](), [requirements](), [scale]() ### AI overview PayPal describes its declarative, or config-based, feature engineering approach for machine learning. The approach lets data scientists declare feature definitions while execution details are handled separately, helping engineers address scale, predictable time to market, and total cost of ownership. The article introduces feature-cost metrics and strategies for reusing existing features. ### Source excerpt Photo by fabio on Unsplash PayPal supports over 400 million active consumers and merchants worldwide. Every minute there are several thousand payment transactions. To prevent fraud in real-time at such a scale, we need to streamline our ML workflow and feature engineering processes to build strong predictors of behaviors and risk indicators. On top of that, it must be done with consistently predictable Time to Market (TTM) and sustainable Total Cost of Ownership (TCO). What is declarative feature engineering? While the declarative feature engineering term was first introduced by Zipline AirBnB in 2019, we have successfully used this paradigm in PayPal for the last decade, though we know it as config-based feature engineering. The idea is to allow data scientists to write a declaration of what their features look like rather than explicitly specify how to construct them on top of different execution platforms. In this way, the concerns of feature construction and execution are abstracted away from scientists so that engineers can worry about those complexities. This post is the first in a series of two posts that will outline how the declarative feature engineering approach helps our engineers to address scale, TTM, and TCO requirements. Let's start with the definitions of metrics that measure our success. TTM of the ML Feature TTM of the Machine Learning (ML) feature is the length of time from the feature concept until it is released to production. Predictable TTM is a paramount need for overall AI maturation and business-first AI strategies. Cost of the ML Feature To qualify and estimate TCO, we break down the cost of the feature as follows: Layers of feature cost The main insight from the feature cost analysis is that we have to reuse existing features across the teams whenever possible. Otherwise, we just pay twice for every aspect. To approach the above challenges, we separate features into three levels of complexity so we can define tailored strategies for scal ## Pre-processing temporal data made easier with TensorFlow Decision Forests and Temporian DevFeed: [Pre-processing temporal data made easier with TensorFlow Decision Forests and Temporian]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2023-09-11T20:14:00Z Content type: article Language: en Sources: [The TensorFlow Blog]() Topics: [Temporal data](), [Temporian](), [TensorFlow Decision Forests](), [Feature Engineering](), [Python](), [datasets]() Tags: [blog](), [data](), [dataset](), [feature-engineering](), [logs](), [machine-learning](), [python](), [sales](), [temporal-data](), [temporian](), [tensorflow-decision-forests](), [time-sequences](), [time-series](), [transactions](), [user-event]() ### AI overview This article demonstrates preprocessing temporal sales data with Temporian and forecasting weekly sales with TensorFlow Decision Forests. It explains event sets, aggregation, moving sums, indexing, and the trade-off between transactional detail and uniformly sampled time series. ### Source excerpt Posted by Google: Mathieu Guillame-Bert, Richard Stotz, Robert Crowe, Luiz GUStavo Martins (Gus), Ashley Oldacre, Kris Tonthat, Glenn Cameron, and Tryolabs: Ian Spektor, Braulio Rios, Guillermo Etchebarne, Diego Marvid, Lucas Micol, Gonzalo Marín, Alan Descoins, Agustina Pizarro, Lucía Aguilar, Martin Alcala Rubi Temporal data is omnipresent in applied machine learning applications. Data often changes over time or is only available or valuable at a certain point in time. For example, market prices and weather conditions change constantly. Temporal data is also often highly discriminative in decision-making tasks. For example, the rate of change and interval between two consecutive heartbeats provides valuable insights into a person's physical health, and temporal patterns of network logs are used to detect configuration issues and intrusions. Hence, it is essential to incorporate temporal data and temporal information in ML applications. INFO: Temporian is a new open-source Python library for preprocessing and feature engineering temporal data for machine learning applications. It is developed in collaboration between Google and Tryolabs. Check the sister blog post for more details. This blog post demonstrates how to train a forecasting model on transactional data. Specifically, we will show how to forecast the total weekly sales from individual sales records. For the modeling part, we will use TensorFlow Decision Forests as they are well suited to handle temporal data. To feed the transaction data to our model, and to compute temporal specific features, we will use Temporian, a newly released library designed for ingesting and aggregating transactional data from multiple non-synchronized sources. Time series are the most commonly used representation for temporal data. They consist of uniformly sampled values, which can be useful for representing aggregate signals. However, time series are sometimes not sufficient to represent the richness of available data. Instead ## Configurable Features using Kotlin Multiplatform DevFeed: [Configurable Features using Kotlin Multiplatform]() Original publisher: [Read original article]() Author: Artem Zasypalov Published: 2023-09-01T06:55:38Z Content type: tutorial Language: en Sources: [YAZIO Engineering - Medium]() Topics: [Kotlin Multiplatform](), [multiplatform](), [Kotlin](), [configuration](), [Compose](), [business logic](), [Mobile](), [client library]() Tags: [business-logic](), [compose](), [configuration](), [development](), [feature-engineering](), [kotlin](), [kotlin-multiplatform](), [library](), [mobile](), [mobile-app-development](), [multiplatform](), [product-development](), [software-development]() ### AI overview YAZIO describes using Kotlin Multiplatform to make its onboarding feature configurable. A shared module provides configuration structures and validation for both the mobile app and a Compose Desktop Flow Configurator, allowing product teams to construct and experiment with screen flows without code changes for existing screen types. ### Source excerpt At YAZIO we love working efficiently. All business logic of our app's new features is written using Kotlin Multiplatform, and we share its code up to the ViewModels. This approach reduces engineering effort, increases stability, and conserves resources. We are always on the lookout for the next step. Some screens and even entire features require swift reactions in changing situations to provide the highest level of experience to our users. The first candidate was Onboarding. Our creative Product Team strives for excellence in first user impressions, meaning that changes in this part of the app occur more frequently than in others. That's why our Development department decided to make this feature not just flexible for content changes, but fully configurable. This means that engineering effort will be reduced to zero for changes that involve only existing Screen Types. Here comes our first term. A Screen Type is a combination of layout and business logic used for the screen in a Flow. And a Flow is a sequence of screens presented to a user in the feature. Our goal was to provide the Product team with a way to construct Flows and use them in the app without code changes. We took advantage of Kotlin Multiplatform again. First, we decided to have a single source of truth for both the app and the Flow Configurator. Configuration data structures and their validations are in a separate multiplatform module. This module is shared between our KMM client library with business logic and the Compose Desktop Flow Configurator app. In the Configurator, we can use our Screen Types to create screen instances and connect them to construct a Flow, create branching to cover all user's questions and concerns, and experiment with content and screen sequences. The saved configuration is then exported and used in the YAZIO mobile app. By delegating responsibility for screen sequences and content, we didn't neglect the safety of our solution. Each flow undergoes a series of validations to ## How to Create a Malware Detection System With Machine Learning DevFeed: [How to Create a Malware Detection System With Machine Learning]() Original publisher: [Read original article]() Author: Simone Margaritelli Published: 2019-05-22T21:59:13Z Content type: tutorial Language: en Sources: [evilsocket]() Topics: [Malware](), [Machine Learning & Artificial Intelligence](), [Neural Network](), [Windows](), [Code]() Tags: [ai](), [antivirus](), [binary-analysis](), [classification](), [computer-virus](), [cuda](), [cybersecurity](), [deep-learning](), [deep-neural-networks](), [deployment](), [dnn](), [ergo](), [feature-engineering](), [features](), [gpu](), [how-to](), [keras](), [machine-learning](), [malware](), [malware-detection](), [neural-network](), [neural-networks](), [nvidia](), [portable-executable](), [security-research](), [tensorflow](), [tf](), [windows-pe]() ### AI overview A practical tutorial on using machine learning and artificial neural networks to detect Windows malware without relying on an explicit signatures database. It uses malware detection as an example for the ergo project, which automates parts of model creation, data encoding, GPU training, benchmarking, and deployment. ### Source excerpt In this post we'll talk about two topics I love and that have been central elements of my (private) research for the last ~7 years: machi ## Thoughts from AAAI 2019 DevFeed: [Thoughts from AAAI 2019]() Original publisher: [Read original article]() Author: Loren Puchalla Fiore Published: 2019-05-13T00:00:00Z Content type: opinion Language: en Sources: [Jane Street]() Topics: [Machine Learning & Artificial Intelligence](), [Deep neural networks](), [Feature Engineering]() Tags: [ai](), [conference](), [fairness](), [feature-engineering](), [machine-learning](), [neural-networks]() ### AI overview Jane Street colleagues reflect on themes and projects presented at AAAI 2019, including neural-network interpretability, adversarial weaknesses in interpretation techniques, classical AI, AI for social good, social AI, and fairness in machine learning. ### Source excerpt At Jane Street, for the last several years, we have been increasingly interested in machine learning and its many use cases. This is why it was exciting when earlier this year myself and a few of my colleagues had the opportunity to attend the AAAI 2019 conference. We'd like to take this space to share with you some of the interesting projects and themes we saw at the conference. ## A Peek into Machine Learning at Square DevFeed: [A Peek into Machine Learning at Square]() Original publisher: [Read original article]() Author: Sara Vera Published: 2017-08-04T17:34:16Z Content type: article Language: en Sources: [Square Corner Blog RSS Feed]() Topics: [Machine Learning & Artificial Intelligence](), [datasets](), [Development](), [Feature Engineering](), [Instrumentation]() Tags: [business](), [data](), [data-science](), [dataset](), [development](), [feature-engineering](), [instrumentation](), [machine-learning]() ### AI overview Square describes how it uses machine learning across its business, including internal tools and customer-facing experiences. The article outlines a workflow covering problem definition, data collection and validation, model selection and evaluation, and iterative improvements such as refactoring and feature engineering. ### Source excerpt We'll be posting more about specific projects, methods, and how we use machine learning in our day-to-day work -- stay tuned! For now... ## Tricks and tips for feature engineering DevFeed: [Tricks and tips for feature engineering]() Original publisher: [Read original article]() Author: Mrinal Parekh Published: 2016-07-20T18:32:44Z Content type: tutorial Language: en Sources: [CleverTap]() Topics: [Feature Engineering](), [data]() Tags: [ai-and-ml](), [ai-and-ml-data-and-analytics-data-insights-engineering-data-science](), [data](), [data-and-analytics](), [data-insights](), [data-science](), [engineering](), [feature-engineering](), [tips](), [tricks]() ### AI overview The article explains feature engineering as the creation of meaningful new input variables from existing fields for predictive modeling. It describes how domain expertise, reasoning, intuition, or automation can improve data insights and predictive model performance. ### Source excerpt Predictive modeling is a formula that transforms a list of input fields or variables into some output of interest. Feature The post Tricks and tips for feature engineering first appeared on CleverTap.