# data-analysis Published articles for data-analysis. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Automating EDA With fg-data-profiling DevFeed: [Automating EDA With fg-data-profiling]() Original publisher: [Read original article]() Author: Real Python Published: 2026-09-15T14:00:00Z Content type: tutorial Language: en Sources: [Real Python]() Topics: [Data analysis](), [pandas](), [Python](), [Data Quality](), [Statistics](), [HTML](), [JSON]() Tags: [data-analysis](), [data-quality](), [pandas](), [python](), [statistics](), [tutorial]() ### AI overview A video course on automating exploratory data analysis with fg-data-profiling. It covers generating interactive reports from DataFrames, exporting them to HTML or JSON, analyzing time series, and comparing datasets. ### Source excerpt Automate exploratory data analysis by transforming DataFrames into interactive reports with one command from fg-data-profiling. ## Beyond the data: what a Business Analyst does at Nubank DevFeed: [Beyond the data: what a Business Analyst does at Nubank]() Original publisher: [Read original article]() Author: Nubank Editorial Published: 2026-09-15T12:14:41Z Content type: opinion Language: en Sources: [Nubank]() Topics: [Data analysis](), [decision-making](), [context](), [Tech Lead]() Tags: [adoption](), [business-analyst](), [data](), [data-analysis](), [data-analytics](), [data-science-machine-learning](), [decision-making](), [manager](), [tech-lead](), [user-experience]() ### AI overview A first-person account of the Business Analyst role at Nubank, describing how BAs connect data analysis, context, and experimentation to product and business decisions. The article emphasizes clarifying trade-offs, investigating metrics, and collaborating with product and technical roles. ### Source excerpt Understand how Business Analysts at Nubank use data, context, and experimentation to guide product decisions. The post Beyond the data: what a Business Analyst does at Nubank appeared first on Building Nubank. ## DataAgents: How we turned 9 months of analysis into 10 days DevFeed: [DataAgents: How we turned 9 months of analysis into 10 days]() Original publisher: [Read original article]() Author: Capital One Tech Published: 2026-06-09T22:42:18Z Content type: tutorial Language: en Sources: [Capital One Tech]() Topics: [Cloud](), [AI Agent](), [Amazon Web Services](), [Azure](), [Google Cloud Platform (GCP)](), [data]() Tags: [agentic-ai](), [ai](), [aws](), [azure](), [cloud](), [cloud-platform](), [data-analysis](), [false-positives](), [gcp](), [least-privilege](), [security]() ### AI overview This engineering deep dive describes the DataAgents pattern for analyzing heterogeneous cloud resources at scale. It focuses on cloud resource dormancy detection across AWS, Azure, and Google Cloud Platform, using entity-specific criteria, confidence-based prioritization, and documented reasoning. The article reports reducing the analysis effort from an estimated 6-9 months to 10 days. ### Source excerpt An engineering deep dive into the pattern that changed how we approach large-scale classification problems. Every engineering team has that project sitting in the backlog. The one where someone says, "We really should analyze all of these," and the room goes quiet. Everyone knows what "all of these" means -- hundreds of entities, complex rules, no clear starting point. For us, it was cloud resource dormancy detection. We had around 350 distinct cloud resource types spread across AWS, Azure and Google Cloud Platform (GCP). Each type has different behavior patterns. An EC2 instance sitting idle looks nothing like a dormant Amazon S3 (S3) bucket or an unattached Elastic IP. Detecting dormancy required understanding what "active" means for each specific resource, then writing detection logic that wouldn't flood operations teams with false positives. Traditional estimate: 6-9 months of expert analysis. Actual time: 10 days. Here's how we did it, and more importantly, here's the reusable pattern behind it. The problem with large-scale analysis: Before we get to the solution, it's worth naming the pattern that makes these projects so painful. It shows up everywhere: Cloud resources - Which of our 350 resource types are dormant? Data governance - Which of our 800 tables have quality issues we should monitor? Security - Which of our access entitlements violate least-privilege principles? Compliance - Which of our 500 policy controls need remediation? In every case, the structure is similar -- a large catalog of heterogeneous entities, entity-specific rules that don't generalize, unknown priorities and a high cost for getting it wrong. The traditional approach is not just slow. It's structurally limited. You get coverage of the "obvious" cases, inconsistent logic across analysts, and tribal knowledge that evaporates when people leave. What you need is something that can assess each entity, apply consistent criteria, prioritize by confidence and document its reasoning. The DataA ## Building Reliable AI Products in the Agentic Era DevFeed: [Building Reliable AI Products in the Agentic Era]() Original publisher: [Read original article]() Author: Dinesh Murthy Published: 2026-05-29T21:30:04Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Artificial Intelligence](), [Generative AI](), [Large Language Model](), [Data analysis]() Tags: [2026](), [agentic](), [ai](), [ai-agents](), [community](), [data-analysis](), [generative-ai](), [llm]() ### AI overview A DigitalOcean Deploy 2026 panel explores how founders build dependable AI products when teams can access the same frontier models through APIs. The discussion focuses on reliability, human oversight, production-scale agent behavior, model selection, and creating differentiation beyond the underlying model. ### Source excerpt Getting your hands on a capable AI model is the easy part now. Every team can reach the same frontier models through an API, so a strong model is not what sets a product apart. What separates a working product from a demo is everything around the model. You have to measure whether the agent is actually doing its job, then keep grinding on reliability until it stops making expensive mistakes in front of real users. I moderated a panel on exactly that at DigitalOcean's Deploy 2026 conference in San Francisco, a forty-minute conversation with four founders on what they've learned shipping AI products that people depend on: Angela Hoover, co-founder and CEO of Andi AI, an ad-free consumer search engine that pairs generative AI with live web data to give people direct answers instead of a page of ad-heavy links. Alex Mashrabov, co-founder and CEO of Higgsfield AI, a platform that lets creators and agencies produce cinematic video without any physical production. Hovsep Seraydarian, co-founder and CTO of LawVo, a Canadian legal platform that pairs hundreds of AI agents trained in specific legal areas with human lawyers who verify their accuracy. Peter Elias, founder of Probably, a data analysis agent that lets non-technical people query their data in plain English and runs calculations on a local engine instead of an LLM so it can decline to answer when the data does not support a clear result. The discussion got into what each founder underestimated once their agents had to run at scale, how they choose models from a field that keeps growing, what "agentic" actually means in production, and where a real moat comes from when everyone builds on the same foundation. Watch the full session from Deploy 2026: View YouTube video Making agents work in production When the founders were asked what they underestimated once their agents had to run in production, none of them pointed to the model. You need creative DNA Higgsfield spent a year on R&D without traction. What finally mov ## Leading design through the AI shift DevFeed: [Leading design through the AI shift]() Original publisher: [Read original article]() Author: andyacevedo Published: 2026-05-21T16:47:34Z Content type: opinion Language: en Sources: [Slack Design]() Topics: [Artificial Intelligence](), [AI-assisted coding](), [Code](), [Data analysis](), [Low-Code / Internal Tools](), [Tooling](), [ui]() Tags: [agents](), [ai](), [ai-tools](), [coding-agents](), [data-analysis](), [figma](), [internal-tools](), [ui](), [uncategorized](), [workflow]() ### AI overview A Slack Design leader describes how AI is changing the team's design workflow, including the use of coding agents for data analysis and rapid prototyping, code-based demos, and designer-built internal tools. The article emphasizes combining AI tools with human judgment and taste while acknowledging increased stress, uncertainty, and ethical questions. ### Source excerpt AI has eaten the technology industry. What started as a slow, hallucination-prone chatbot has become something much harder to ignore. As a designer, it can feel like the ground is shifting under your feet, quietly at first, then all at once. Every week, something new disrupts your workflow, your toolset, your expectations about what's possible. [...] The post Leading design through the AI shift appeared first on Slack Design. ## Pharmacy late-night opening hours analysis featured in the Financial Times DevFeed: [Pharmacy late-night opening hours analysis featured in the Financial Times]() Original publisher: [Read original article]() Author: Robin Wilson Published: 2026-02-04T14:10:37Z Content type: article Language: en Sources: [Robin Wilson]() Topics: [Data analysis](), [geospatial](), [Geographic Information System](), [Python](), [pandas]() Tags: [academic](), [analysis](), [article](), [data](), [data-analysis](), [geospatial](), [gis](), [pandas](), [python]() ### AI overview The article describes an analysis of NHS data on community pharmacy opening hours in England. It reports that the number of pharmacies open past 9pm on weekdays fell by approximately 95% between 2022 and 2025, leaving large areas without late-night pharmacy access. The analysis used Python, pandas, geospatial tools and mapping libraries, and was featured in the Financial Times. ### Source excerpt Some data analysis I've done has been featured in the Financial Times today - see this article (the link may not work any more unless you have a FT subscription - sorry). The brief story is that I had terrible back pain over Christmas, and spoke to an out-of-hours GP on the phone who prescribed [...] ## One in a million: celebrating the customers shaping AI's future DevFeed: [One in a million: celebrating the customers shaping AI's future]() Original publisher: [Read original article]() Published: 2025-12-22T00:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence](), [OpenAI](), [ChatGPT](), [codex](), [API](), [coding](), [Data analysis]() Tags: [agents](), [ai](), [api](), [chatgpt](), [codex](), [coding](), [customers](), [data-analysis](), [design](), [developers](), [images](), [openai](), [platform](), [products](), [video](), [voice](), [workflows]() ### AI overview OpenAI celebrates more than one million customers using its products to support teams and create new opportunities. The article describes organizations deploying ChatGPT for writing, coding, research, data analysis, and design; building agents to automate workflows; using Codex to accelerate development; and creating new products with the OpenAI API across voice, video, images, and other modalities. ### Source excerpt More than one million customers around the world now use OpenAI to empower their teams and unlock new opportunities. This post highlights how companies like PayPal, Virgin Atlantic, BBVA, Cisco, Moderna, and Canva are transforming the way work gets done with AI. ## Mob Programming: Smells Like Team Spirit DevFeed: [Mob Programming: Smells Like Team Spirit]() Original publisher: [Read original article]() Author: Dmytro Kurets Published: 2025-12-04T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Programming](), [Code](), [pull-requests](), [Back end](), [Front end](), [Data analysis](), [Refactoring]() Tags: [back-end](), [code](), [collaboration](), [data-analysis](), [engineering](), [engineering-culture](), [front-end](), [keyboard](), [pairing](), [programming](), [review](), [team]() ### AI overview The article explains mob programming as structured, shared-focus collaboration with one driver, rotating typing roles, and navigators guiding the work. It describes a trivago team experimenting with collaborative review and coding changes to address pull requests piling up and slowing delivery. ### Source excerpt Mob programming often sparks hot takes. "It's counter-productive." "One person could do it faster." "Why waste five people on one keyboard?" If that's what it looks like in your head--one senior ... ## How Data Powers Agent Productivity DevFeed: [How Data Powers Agent Productivity]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2025-10-06T04:22:31Z Content type: article Language: en Sources: [Helpshift]() Topics: [data](), [dashboards]() Tags: [analytics](), [apache-spark](), [big-data](), [customers](), [data](), [data-analysis](), [metrics](), [support]() ### AI overview This article describes how a customer support team developed custom metrics to measure agent productivity more accurately. The metrics aim to distinguish productive work from idle or merely available time and support staffing, scheduling, and performance decisions. ### Source excerpt As a data engineer, I used to see metrics as just numbers on a dashboard -- until I realized they're the lens through which customers view and run their operations. In customer support, for example, agent productivity metrics aren't just figures, they're actionable insights that drive efficiency, shape staffing decisions, and directly impact customer satisfaction. These aren't just charts -- they help customers understand the value we provide, how well things are working, and what decisions to make next. Realizing this changed how I think about building analytics. ➡💡The Question That Shifted Our Perspective In customer support, how well the team works really matters. It affects how much the company spends, how happy the customers are, and how the team feels about their work. Support managers often ask: Are we staffed correctly for the volume we're handling? Are agents spending their time productively? How can we optimize scheduling and performance? When we began our Agent Workforce Management project, we already had a few standard metrics in place like online time, login time, and available time. These told us when agents were present -- but not what they were actually doing. Customers weren't asking "Are our agents online?" They were asking "How productive are our agents?" And truthfully, we didn't have a good answer. There was no visibility into how much time was being spent on real work versus idle time. No way to differentiate between being "available" and being "productive". This made it hard for teams to identify gaps, support high performers, or spot patterns that needed attention. This project was all about answering that question in the right way. 🔍📊 Custom Metrics We Built We designed a set of new metrics that give a clearer picture of how agents spend their time. These metrics give us a deeper understanding of how time is actually being spent, helping us move beyond assumptions and focus on what really drives productivity. Engagement Metrics These show how ## An Ethos for Using AI Coding Tools: Ownership and Exploiting High-Value Opportunities DevFeed: [An Ethos for Using AI Coding Tools: Ownership and Exploiting High-Value Opportunities]() Original publisher: [Read original article]() Published: 2025-09-15T00:00:00Z Content type: opinion Language: en Sources: [Tim Kellogg]() Topics: [Artificial Intelligence](), [ai-coding](), [Software Engineering](), [Data analysis](), [future of software]() Tags: [ai](), [ai-coding](), [coding](), [data-analysis](), [future-of-software](), [software-engineering]() ### AI overview The author describes an ethos for using AI coding tools, centered on owning and understanding the generated code. They also argue that effective AI coding involves finding opportunities where limited AI effort can produce substantial value, such as proof-of-concept work and rapid data analysis. ### Source excerpt A few people have asked me how I use AI coding tools. I don't think it's a straightforward answer. For me it's not really a procedure or recipe, it's more of an ethos. ## Mastering Entropy: How Mercado Libre captures Millions of Users' Experiences DevFeed: [Mastering Entropy: How Mercado Libre captures Millions of Users' Experiences]() Original publisher: [Read original article]() Author: Franco Bortolazzi Conte Published: 2025-08-14T14:45:10Z Content type: article Language: en Sources: [Mercado Libre Tech]() Topics: [data](), [JSON](), [Artificial Intelligence]() Tags: [analysis](), [architecture](), [big-data](), [data](), [data-analysis](), [data-science](), [governance](), [json](), [mercadolibre](), [technology]() ### AI overview Mercado Libre describes its tracking infrastructure and governance for organizing a large volume of user interaction data. The article explains that its ecosystem generates nearly 50,000 event types and more than 17 billion daily interactions, with tracking records represented in JSON. ### Source excerpt Building a Tracking Infrastructure and Governance for Flexible Information and AI-Driven Data AnalysisGrab a coffee! Let's find order for this chaos together ☕ In the world of e-commerce, every click tells a story. We want to make sure we don't miss a single one. Let's explore the product diversity of Mercado Libre's ecosystem. It is a dynamic and expansive network designed to facilitate and enhance e-commerce & payments across Latin America. At its core is the Mercado Libre Marketplace, a platform that connects millions of buyers and sellers, offering a diverse range of products. Mercado Pago offers an innovative digital payments solution, complementing the ecosystem. To support logistics, Mercado Envíos provides efficient shipping and fulfillment services. Additionally, Mercado Crédito offers financial inclusion by providing loans to businesses and empowering sellers with access to working capital. We can also mention Mercado Play as a content platform, Mercado Ads to connect buyers to sellers or some newer players like the crypto world. Together, these interconnected services create a robust and user-friendly ecosystem, generating a huge amount of data that we collect to enhance our users' experience. The scale is truly remarkable: our platform processes nearly 50,000 distinct event types, culminating in over 17 billion daily interactions -- representing an extraordinary 50% year-over-year growth in the last years. This data deluge presents both unprecedented challenges and opportunities for creating meaningful insights. Then... how do we efficiently organize and manage this diversity and quantity of events?What is an "event" in our context? 🔍 In our ecosystem, we call it a track. We deliberately place the user at the heart of the decision-making process to enrich their experience. This approach directly embodies one of our core cultural principles: "We create value for our users". Technically, the essential form of tracks is a representation in JSON (JavaScript Obj ## How often do LLMs visit llms.txt? DevFeed: [How often do LLMs visit llms.txt?]() Original publisher: [Read original article]() Author: Tiffany Chen Published: 2025-06-27T00:00:00Z Content type: article Language: en Sources: [Mintlify Blog]() Topics: [Large Language Model](), [Markdown](), [Data analysis](), [Crawler](), [Retrieval-Augmented Generation]() Tags: [ai-trends](), [claude](), [data-analysis](), [google](), [llms](), [markdown](), [pages](), [retrieval](), [retrieval-augmented-generation]() ### AI overview The article presents a Profound data analysis of how language models access llms.txt and llms-full.txt. The supplied text says both files receive AI traffic, with a strong preference for llms-full.txt and ChatGPT accounting for most visits. It attributes this preference to models embedding full content instead of relying on retrieval-augmented generation, while noting that the analysis covers 25 companies. ### Source excerpt Last month, we explored signals for the emerging standard of llms.txt, which is a Markdown file that makes websites easier for LLMs to index. ## Why streaming is the backbone for AI-native data platforms DevFeed: [Why streaming is the backbone for AI-native data platforms]() Original publisher: [Read original article]() Author: Tyler Rockwood Published: 2025-06-24T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Streaming](), [data-platforms](), [Artificial Intelligence](), [AI Development](), [Data analysis](), [Embeddings](), [SQL](), [dashboards](), [personalization](), [Development]() Tags: [ai](), [ai-agents](), [ai-and-data-warehouse-evolution](), [ai-automation-in-products](), [ai-data-analysis-automation](), [ai-enhanced-product-development](), [ai-native-data-platforms](), [ai-product-integration](), [ai-technology-integration](), [dashboards](), [data-analysis](), [data-platforms](), [data-platforms-for-ai-scalability](), [development](), [embeddings](), [modern-data-warehouse-ai](), [personalization](), [real-time-ai-personalization](), [sql](), [streaming](), [streaming-for-ai-data-platforms](), [structuring-data-lakes-with-ai](), [thought-leadership]() ### AI overview The article argues that streaming is the backbone of AI-native data platforms. It presents the modern data warehouse as a central source of context for AI and describes how AI can support analysis, dashboards, SQL queries, trend detection, embeddings, personalization, recommendations, automation, and faster product development. It also emphasizes guardrails and performance measurement for AI-driven automation. ### Source excerpt To power AI at scale, organizations must adapt to the evolving role of the modern data warehouse. Here's what you need to keep up in a rapidly evolving industry. ## Here and Now: Reusing Code at Feedzai with JupyterLab Snippets DevFeed: [Here and Now: Reusing Code at Feedzai with JupyterLab Snippets]() Original publisher: [Read original article]() Author: João Palmeiro Published: 2025-03-17T13:11:26Z Content type: article Language: en Sources: [Feedzai]() Topics: [jupyterlab](), [Data Science](), [Code](), [Data analysis]() Tags: [code](), [data-analysis](), [data-science](), [debugging](), [jupyter](), [jupyterlab](), [python](), [repositories](), [snippet]() ### AI overview Feedzai developed JupyterLab Snippets to help data scientists reuse code from the many notebooks created for daily work. The article describes research using notebook repository statistics and user interviews, and reports that duplicating notebooks, maintaining reference collections of snippets, and sharing notebooks or snippets are common practices. ### Source excerpt Data scientists use different Jupyter notebooks every day -- ranging from disposable ones for quick tasks to those shareable with clients. Over time, more and more notebooks accumulate, making it increasingly difficult to reuse them in whole or in part. To mitigate this problem and make the most relevant pieces of code quickly accessible to every data scientist, we developed JupyterLab Snippets at Feedzai -- our take on leveraging code snippets directly on JupyterLab. JupyterLab is a computational notebook platform that enables us to carry on data science work (and beyond) via notebooks. These notebooks, where code, text, and images come together, allow us to complete all kinds of tasks, keeping each input close to each output. At Feedzai, data scientists have access to different JupyterLab environments packed with custom notebooks and Python packages. Here they go from analyzing data to training models, from preparing reports to debugging the system -- a "lab of all trades", we could say. Given the importance of JupyterLab and notebooks in the daily work of our data scientists, we started a research project that culminated in JupyterLab Snippets. First, we collected statistics from internal notebook repositories across teams and conducted user interviews with junior and senior data scientists. We needed to know more about how data scientists actually use JupyterLab and notebooks, and what their ideas are for a better platform. A sneak peek of JupyterLab Snippets. From the very start, every data scientist can import snippets immediately or start their collection by creating one from scratch. After compiling the insights, several aspects became clear: JupyterLab and notebooks are heavily used for all sorts of data science and ad-hoc tasks (typically one per notebook). There is no single type of task (e.g., exploratory data analysis) that is more commonly addressed with notebooks than others. While file structures may look similar at first glance (each environment has at ## Data Agent Benchmark for Multi-step Reasoning (DABstep) DevFeed: [Data Agent Benchmark for Multi-step Reasoning (DABstep)]() Original publisher: [Read original article]() Author: Adyen Published: 2025-02-05T09:19:27Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Benchmark](), [Data analysis](), [LLMs](), [AI Bots]() Tags: [adyen](), [agents](), [ai-models](), [benchmark](), [data](), [data-agents](), [data-analysis](), [data-science](), [genaimodels](), [hugging-face]() ### AI overview Adyen and Hugging Face introduce DABstep, a benchmark with more than 450 data analysis tasks for evaluating language models and AI agents on multi-step reasoning. The article reports that the strongest reasoning-based agents achieved 16% accuracy, indicating that complex real-world data analysis remains difficult for current models. ### Source excerpt Alex Egg (Adyen), Martin Iglesias (Adyen), Friso Kingma (Adyen), Andreu Mora (Adyen), Leandro Von Werra (HuggingFace), Thomas Wolf (HuggingFace) https://huggingface.co/spaces/adyen/DABstep 👉🏽 You can access DABStep with this link here: https://huggingface.co/spaces/adyen/DABstep Language models are becoming increasingly capable and can solve tasks autonomously as agents. There are many exciting use cases, especially at the intersection of reasoning, code, and data. However, proper evaluation benchmarks on real-world problems are lacking and hinder progress in the field. To tackle this challenge, Adyen and Hugging Face built the Data Agent Benchmark for Multi-step Reasoning (DABstep) together. DABstep consists of over 450 data analysis tasks designed to evaluate the capabilities of state-of-the-art LLMs and AI agents. Our findings reveal that DABstep presents a significant challenge for current AI models, with the most capable Reasoning-based agents achieving only 16% accuracy, highlighting significant progress to be made in the field. DABStep requires AI models to: dive in details of data and be rigorous (no hallucinations) reason over free form text and databases connect with real life use-cases (not just math or code) In this blog post, we'll cover the design and construction of the benchmark, explore evaluation results, and discuss the significant gap between current models and the ability to solve complex data analysis tasks effectively.. Motivation Data analysis is both an art and a science that requires technical skill, domain knowledge and creativity, and thus, it's rarely straightforward. Even seasoned data analysts face challenges like: Simple but time-consuming tasks: The sheer volume of even simple tasks often turns straightforward analysis into hours of repetitive work. Complex context and high cognitive load: Some tasks require analysis to juggle intricate domain-specific knowledge, making them both time-intensive and mentally draining. For example, (1) ## DABStep: Data Agent Benchmark for Multi-step Reasoning DevFeed: [DABStep: Data Agent Benchmark for Multi-step Reasoning]() Original publisher: [Read original article]() Author: Alex Egg; Martin Iglesias Goyanes; Friso Kingma; Andreu Mora; Leandro von Werra; Thomas Wolf; Aymeric Roucher Published: 2025-02-04T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Benchmark](), [Data analysis](), [Large Language Model](), [Human-AI evaluation]() Tags: [agentic](), [ai-agents](), [artificial-intelligence](), [benchmark](), [data](), [data-analysis](), [evaluation](), [llms](), [reasoning](), [research]() ### AI overview Adyen and Hugging Face introduce DABstep, a benchmark of more than 450 data analysis tasks for evaluating state-of-the-art LLMs and AI agents. The article reports that the strongest reasoning-based agents reached only 16% accuracy, revealing a substantial gap in current systems' ability to handle rigorous, context-rich, real-world data analysis. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## "Show Me What's Wrong!": Enhancing Fraud Detection Analysis by Combining Charts and Text DevFeed: ["Show Me What's Wrong!": Enhancing Fraud Detection Analysis by Combining Charts and Text]() Original publisher: [Read original article]() Author: Beatriz Feliciano Published: 2024-11-22T18:35:14Z Content type: article Language: en Sources: [Feedzai]() Topics: [Transactions](), [data](), [synthetic-data](), [Tool]() Tags: [data](), [data-analysis](), [data-visualization](), [financial-fraud](), [fraud](), [fraud-detection](), [fraud-investigation](), [image](), [interface](), [research](), [synthetic-data](), [transactions]() ### AI overview The article describes a fraud-analysis tool that combines charts and text to help analysts review suspicious financial transactions. It explains that tabular review can make it difficult to identify patterns and anomalies within a 1-to-5-minute review window, and presents an interface using synthetic data to help analysts prioritize investigation areas. ### Source excerpt Every year, millions of people fall victim to financial fraud. In 2023, the losses tied to this type of crime were estimated at US$159 billion just in the US, with some people losing all of their retirement savings to scammers. However, the impacts of this issue stretch beyond someone's finances. It can also impact a victim's life in many dimensions. Detecting and quickly acting upon suspicious transactions is essential to tackle this problem. Finding Fraud Through Data Tables To review the data of alerted transactions, analysts look at information in tabular format (similar to what is presented in Figure 1), scrolling through it to assess past activity patterns of the alerted person and comparing those with the alerted event. "How much money was spent on average on past transactions?" or "Is that significantly different from the amount on the current alert?" are some questions they might try to answer during their review. Figure 1: Image of a table that analysts typically use to review the data of alerted transactions. The issue with this approach is that finding groups of patterns and anomalies in tabular data can be overwhelming since it requires an increased cognitive load from analysts to interpret the data effectively. This becomes even more complex since these professionals must review and classify the alerted transaction in a short time -- between 1 and 5 minutes. Revamping the analysis To solve this problem, we present a tool that combines charts and text to guide the analysis of financial transactions. As presented in Figure 2, the tool (populated with synthetic data) is divided into three regions that provide different levels of information detail -- from the most high-level to the most detailed. The goal is that the analyst can scan the charts and prioritize their review towards specific areas of the alert. Figure 2: Proposed interface composed of multiple regions: the Knowledge Area Console (A) to detect suspicious areas of the analysis; the Knowledge Are ## Improving Parquet Dedupe on Hugging Face Hub DevFeed: [Improving Parquet Dedupe on Hugging Face Hub]() Original publisher: [Read original article]() Author: yuchenglow; Di Xiao Published: 2024-10-05T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [parquet](), [content defined chunking](), [datasets](), [hugging face](), [Data analysis](), [Databases]() Tags: [analysis](), [content-defined-chunking](), [data](), [data-analysis](), [datasets](), [dedupe](), [format](), [hugging-face](), [parquet](), [storage](), [xet]() ### AI overview This article examines how Parquet file layout affects byte-level Content-Defined Chunking deduplication on Hugging Face Hub. Experiments show that appending rows deduplicates 99.1% of a 2GB file, while modifying one row achieves 89% deduplication because rewritten column headers contain absolute file offsets. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## postgres.new: In-browser Postgres with an AI interface DevFeed: [postgres.new: In-browser Postgres with an AI interface]() Original publisher: [Read original article]() Author: Greg Richardson Published: 2024-08-12T07:00:00Z Content type: article Language: en Sources: [Supabase Blog]() Topics: [Databases](), [Artificial Intelligence](), [browser](), [WebAssembly](), [Large Language Model](), [SQL](), [Data analysis](), [CSV](), [Amazon S3]() Tags: [ai](), [analysis](), [browser](), [data-analysis](), [database](), [large-language-model](), [s3](), [sql](), [wasm](), [web]() ### AI overview The article introduces database.build, an in-browser Postgres sandbox with AI assistance. It explains how PGlite, a WebAssembly build of Postgres, enables local database queries, CSV imports, report and chart generation, database diagrams, and AI-assisted SQL and migration workflows without a remote Postgres container or WebSocket proxy. ### Source excerpt Introducing postgres.new, the in-browser Postgres sandbox with AI assistance. ## SQL vs NoSQL Explained DevFeed: [SQL vs NoSQL Explained]() Original publisher: [Read original article]() Author: Nelson Djalo Published: 2024-07-02T16:00:57Z Content type: comparison Language: en Sources: [Amigoscode Newsletter]() Topics: [NoSQL](), [SQL](), [Databases](), [clickhouse]() Tags: [acid](), [business-intelligence](), [consistency](), [data-analysis](), [database](), [database-scalability](), [durability](), [microsoft](), [mysql](), [nosql](), [olap](), [oracle](), [postgresql](), [rdbms](), [relational-database-management-systems-rdbms](), [relational-databases](), [schema](), [sql]() ### AI overview This comparison explains the differences between SQL and NoSQL databases, including their characteristics, scalability models, and common use cases. It covers SQL database features such as structured schemas and ACID compliance, along with relational and OLAP database categories. ### Source excerpt Choosing the Right Database for Your Needs ## A behind-the-scenes look at building interactive analysis capabilities in Benchling DevFeed: [A behind-the-scenes look at building interactive analysis capabilities in Benchling]() Original publisher: [Read original article]() Author: Wonja Fairbrother Published: 2024-06-11T13:01:25Z Content type: article Language: en Sources: [Benchling]() Topics: [data-processing](), [web applications](), [Amazon EKS](), [Amazon S3](), [API](), [Back end](), [Front end](), [JSON]() Tags: [apache-arrow](), [apache-parquet](), [api](), [architecture](), [backend](), [data](), [data-analysis](), [data-transformation](), [data-visualization](), [duckdb](), [frontend](), [json](), [real-time](), [s3](), [storage](), [systems]() ### AI overview This article explains the architecture and design decisions behind Benchling's Interactive Analysis capabilities. The system lets scientists select data from multiple sources, transform and analyze it in real time, and visualize results within Benchling. Its architecture uses the Benchling web application, a stateless service running on EKS, temporary S3 storage, and a JSON-based transformation API. ### Source excerpt Authors: Wonja Fairbrother and Eli Levine Science is iterative. To design the next experiment, scientists need to analyze the results of previous ones. Interactive Analysis in Benchling allows scientists to perform real-time data transformation, visualization, and analysis without having to transfer it into other systems. In this post we will describe the architecture behind interactive analysis capabilities in Benchling and give a peek into the decision journey we took along the way¹. Interactive Analysis allows scientists to: 1. Select data from many sources: Benchling entity and results data Instrument data Notebook tables Data upload via both API and UI 2. Transform, visualize, and analyze data in real time, without leaving Benchling: Data transformations: filtering, aggregations, window functions, etc. Visualizations: line chart, bar chart, scatter plot, etc. Scientific analysis methods: IC50 and various curve fitting functions Overall architecture The architecture backing Interactive Analysis consists of: The Benchling web application An auto-scaling stateless internal service running on EKS that performs the transformations Temporary S3 storage locations for input and output data, shared between the web app and the service The frontend of the application is responsible for taking in input datasets and transformation configurations from users. The backend of the web application collects all the input data from the appropriate sources, serializes and uploads the data to S3, and sends a synchronous transformation request to the service. The service's API consists of one main endpoint that takes in a JSON payload of transformation parameters. The service can accept a single transformation, or a list of many transformations to perform. In this endpoint, the service downloads and deserializes the input data, performs the transformation with an analysis engine, and serializes and uploads the resulting data to S3. Each request spins up its own self-contained in-memor ## Insights from Workflow History Export on Temporal Cloud DevFeed: [Insights from Workflow History Export on Temporal Cloud]() Original publisher: [Read original article]() Author: Alice Yin Published: 2024-04-24T06:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [export](), [Amazon S3](), [Data pipelines](), [Data analysis](), [parquet](), [airflow](), [AWS Glue]() Tags: [airflow](), [amazon-s3](), [api](), [aws-glue](), [case-study](), [data](), [data-analysis](), [data-pipelines](), [export](), [parquet](), [product-news]() ### AI overview This tutorial explains how to export closed workflow histories from Temporal Cloud to Amazon S3, convert Protocol Buffer files to Parquet, and build data pipelines for analyzing execution metadata, operational efficiency, and bottlenecks. ### Source excerpt Explore how exporting workflow histories in Temporal Cloud gives you deep insights into execution metadata, events, and performance over time. ## A Guide to Logical Replication and CDC in PostgreSQL with Airbyte DevFeed: [A Guide to Logical Replication and CDC in PostgreSQL with Airbyte]() Original publisher: [Read original article]() Author: Jacob Prall Published: 2024-01-11T16:04:46Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [PostgreSQL](), [Replication](), [Database](), [Data analysis](), [Concurrency](), [Open Source]() Tags: [community](), [concurrency](), [data-analysis](), [database](), [guide](), [high-availability](), [load-balancing](), [open-source](), [postgresql](), [rdbms](), [recovery](), [replication](), [transactions]() ### AI overview This guide explains logical replication and change data capture in PostgreSQL, including why data may need to be replicated between PostgreSQL servers or moved to systems suited for analysis and activation. It introduces database replication concepts, compares replication approaches, and describes PostgreSQL's Write-Ahead Log and its roles in crash recovery, point-in-time recovery, and concurrent transactions. ### Source excerpt PostgreSQL is a database that needs no introduction. Started as an open-source project out of UC Berkeley, it has evolved through decades of careful stewardship to become one of the world's most relied on and beloved relational database management systems (RDBMS). There will inev... ## Improving product reliability by imposing constraints as a part of CI/CD process DevFeed: [Improving product reliability by imposing constraints as a part of CI/CD process]() Original publisher: [Read original article]() Author: Pedro Moura Published: 2023-11-03T14:26:59Z Content type: article Language: en Sources: [Revolut Engineering]() Topics: [CI/CD](), [Application Security](), [Vulnerabilities](), [Data analysis]() Tags: [application-security](), [blocking](), [ci-cd](), [data-analysis](), [devsecops](), [engineering](), [sast](), [sca](), [scanners](), [security](), [security-vulnerabilities](), [software-development]() ### AI overview Revolut describes a CI/CD approach that uses automated risk calculation, data analysis, security scanning, and deployment constraints to improve product reliability. The approach aims to identify vulnerabilities and bugs early, mitigate them, and potentially block high-risk applications from reaching production. ### Source excerpt At Revolut, we always aim to consistently provide efficient, high-quality, and secure services. Our primary goals are to enhance product quality, increase customer satisfaction, and reduce business risk. To achieve those goals, we should aim to be as proactive -- rather than reactive -- as possible. This means fixing any point of failure before it reaches production by identifying, reporting, and potentially blocking deployments that can impact the reliability of our products. In this article, we'll explain how we achieve those goals using automated risk calculation, data analysis, and imposing constraints to push for the mitigation of open vulnerabilities and the fix of reported bugs identified in our products. Challenges faced by Revolut Each application/service has its own specifications, with different technology stacks and architectures, bringing a diversity of security challenges and different vulnerabilities. That being said, application risk cannot be based only on the reported security vulnerabilities but also in the context of that specific application. Continuous scanning, reporting, visibility, and risk evaluation are essential to providing the best security advice and automated security controls. Nowadays there are several types of security scanners (SAST, SCA, DAST, IaC, etc.) that help security professionals to identify and report vulnerabilities, where each of them can belong to a different third-party provider. For further information on these, read our article on continuous security. Data scraped from different sources creates friction when it's needed to group findings and provide mitigations. This can also decrease mean time to identify vulnerabilities and mean time to provide mitigations. Also, to avoid such negative workflow impact, security must be shifted left. Without a centralised source of truth for security-related application data, most of the time is wasted on searching for information/findings in segmented platforms and finding the corre [Next page]()