# Scribd Tech Scribd technology builds and delivers one of the world's largest libraries, bringing the best books, audiobooks, and journalism to millions of people around the world. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## A Repeatable Human-in-the-Loop Process for Large-Scale LLM Classification DevFeed: [A Repeatable Human-in-the-Loop Process for Large-Scale LLM Classification]() Original publisher: [Read original article]() Author: Anish Kumar Published: 2026-07-11T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Large Language Model](), [Prompt Engineering](), [Inference](), [datasets](), [data]() Tags: [asynchronous](), [batch](), [content-trust-series](), [cost](), [data](), [databricks](), [datasets](), [evaluation](), [featured](), [inference](), [llm](), [machinelearning](), [models](), [production](), [prompt](), [scribd](), [workflow]() ### AI overview The article presents a repeatable human-in-the-loop process for large-scale LLM classification. It combines fast-model labeling, judge-model disagreement detection, targeted SME review, a golden dataset built from corrections, and selective prompt iteration. ### Source excerpt LLM models change. Prompt quality changes. Cost changes. We assumed that from day one. ## Dual-Embedding Trust Scoring DevFeed: [Dual-Embedding Trust Scoring]() Original publisher: [Read original article]() Author: Eric Chang Published: 2026-02-25T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [trust and safety](), [Embeddings](), [Artificial Intelligence](), [Machine learning](), [data](), [datasets](), [genai]() Tags: [ai](), [content-trust-series](), [data](), [dataset](), [embeddings](), [featured](), [generative-ai](), [machine-learning](), [machinelearning](), [pii](), [privacy](), [research](), [scribd](), [trust](), [trust-and-safety]() ### AI overview Scribd describes a Content Trust Score that combines Generative AI signals, proprietary multilingual embeddings, and classical machine learning to assess the severity of documents violating defined trust and safety pillars. The research covers illegal, explicit, privacy/PII, and low-quality content, using annotated data from roughly 100,000 documents. ### Source excerpt Scribd is a digital library serving academics and lifelong learners, offering hundreds of millions of documents. This very nature presents a significant concern: content trust and safety. Protecting our library from undesirable and unsafe content is a top priority, but the multilingual and multimodal (text and images) nature of our platform makes this mission very challenging. Also, while third-party tools exist, they often fall short, lacking the nuance to handle our specific trust and safety categories. ## Engineering Around Extreme S3 Scale at Scribd DevFeed: [Engineering Around Extreme S3 Scale at Scribd]() Original publisher: [Read original article]() Author: R Tyler Croy Published: 2026-02-10T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Amazon S3](), [Cloud](), [data](), [Amazon Web Services]() Tags: [aws](), [aws-s3](), [cloud](), [featured](), [s3](), [scale](), [scribd](), [talk](), [youtube]() ### AI overview Scribd discusses engineering challenges at extreme AWS S3 scale, including checking whether files are damaged, controlling costs, and consolidating objects to reduce spending. The discussion is also available as audio and video. ### Source excerpt Scribd has absolutely fascinating data-at-scale type problems, all the way down to the fundamentals of how we use AWS S3. In my previous post I wrote about the design of Content Crush and how Scribd is consolidating objects in S3 to minimize our costs. Related to that work I was fortunate enough to join the (in)famous Corey Quinn to talk about Engineering around Extreme S3 scale: ## Deploying a Cost-Effective, Scalable PhotoDNA System for CSAM Detection DevFeed: [Deploying a Cost-Effective, Scalable PhotoDNA System for CSAM Detection]() Original publisher: [Read original article]() Author: Anish Kumar Published: 2026-01-20T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Perceptual hashing](), [observability](), [Machine learning](), [Microsoft]() Tags: [auditability](), [aws](), [content-trust-series](), [cost](), [databricks](), [false-positives](), [featured](), [hashing](), [lambda](), [latency](), [legal](), [microsoft](), [observability](), [privacy](), [reporting](), [scale]() ### AI overview Scribd describes a production system for detecting known Child Sexual Abuse Material using PhotoDNA perceptual hashes. The article covers its integration with NCMEC reporting, ingestion-scale architecture, operational model, cost considerations, and safety-related constraints. ### Source excerpt Child safety is a non-negotiable responsibility for any platform that hosts user-generated content. Over the last year, we designed and deployed a production system that detects known Child Sexual Abuse Material (CSAM) using PhotoDNA perceptual hashes, integrates with the National Center for Missing and Exploted Children's (NCMEC) reporting system, and scales efficiently across our ingestion surfaces. This post explains the problem we set out to solve, how PhotoDNA hashing works, the online child-protection ecosystem (NCMEC, Tech Coalition, Project Lantern), our architecture and operational model, cost considerations, and key learnings. ## How Scribd's Content Crush consolidates S3 objects for storage cost management DevFeed: [How Scribd's Content Crush consolidates S3 objects for storage cost management]() Original publisher: [Read original article]() Author: R Tyler Croy Published: 2026-01-12T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Amazon S3](), [Amazon Web Services](), [data](), [parquet](), [cloud-infrastructure]() Tags: [apache-parquet](), [aws](), [aws-s3](), [cost](), [data](), [featured](), [object-storage](), [retrieval](), [rust](), [s3](), [storage]() ### AI overview Scribd describes Content Crush, a system that consolidates the many objects generated from uploaded documents and presentations into a single Apache Parquet file in Amazon S3 while preserving virtualized keyspace and retrieval semantics. The system addresses storage-management complexity and the cost of S3 Intelligent-Tiering monitoring at Scribd's scale. ### Source excerpt Scribd and Slideshare have been using AWS S3 for almost twenty years and store hundreds of billions of objects making storage management quite a challenge. My focus at Scribd has generally been around data and storage but only in the past twelve months have I started to really focus on one of our hardest technology problems: cost-effective storage and availability for the hundreds of billions of objects that represent our content library. ## Using GitHub Actions OIDC to authenticate with AWS without hardcoded IAM credentials DevFeed: [Using GitHub Actions OIDC to authenticate with AWS without hardcoded IAM credentials]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2026-01-06T00:00:00Z Content type: tutorial Language: en Sources: [Scribd Tech]() Topics: [GitHub](), [AWS IAM](), [OpenID connect (OIDC)](), [GitHub Actions](), [Terraform](), [Amazon Web Services]() Tags: [api](), [api-keys](), [authentication](), [cloud](), [github](), [github-actions](), [iam](), [oidc](), [openid-connect](), [terraform]() ### AI overview This tutorial explains how Scribd uses GitHub Actions with AWS and why hardcoded API keys and tokens are difficult to manage securely. It introduces AWS IAM OpenID Connect identity providers and the open-source terraform-oidc-module for establishing trust between GitHub and AWS and configuring deployment permissions. ### Source excerpt Scribd deploys a lot of code from GitHub to AWS using GitHub Actions, which means many of our Actions need to access AWS resources. Managing AWS API keys and tokens for different IAM users is time-consuming, brittle, and insecure. Managing key-distribution between AWS and GitHub also makes it difficult to track which keys go where, when they should be rotated, and what permissions those keys have. Fortunately AWS supports creating OpenID Connect identity providers which is an ideal tool handle this kind of cross-cloud authentication in a more maintainable way. ## Building a Scalable AWS Backup System for S3 Data Warehouses DevFeed: [Building a Scalable AWS Backup System for S3 Data Warehouses]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-09-22T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Amazon S3](), [Amazon Web Services](), [AWS Lambda](), [parquet](), [cloud-infrastructure]() Tags: [aws](), [aws-lambda](), [backup](), [backups](), [cleanup](), [compute](), [cost](), [data](), [data-lake](), [databases](), [datasets](), [deltalake](), [efficiency](), [featured](), [incremental](), [lambda](), [lifecycle](), [parquet](), [pipeline](), [reduce](), [retention](), [s3](), [terraform]() ### AI overview This article describes a monthly backup system for S3 data warehouses that uses incremental copying, S3 Inventory validation, parallel processing, and Glacier retention. It routes smaller workloads to AWS Lambda and larger workloads to ECS Fargate, with lifecycle-based cleanup for eligible files. ### Source excerpt We designed and implemented a scalable, cost-optimized backup system for S3 data warehouses that runs automatically on a monthly schedule. The system handles petabytes of data across multiple databases and uses a hybrid approach: AWS Lambda for small workloads and ECS Fargate for larger ones. At its core, the pipeline performs incremental backups -- copying only new or changed parquet files while always preserving delta logs -- dramatically reducing costs and runtime compared to full backups. Data is validated through S3 Inventory manifests, processed in parallel, and stored in Glacier for long-term retention. To avoid data loss and reduce storage costs, we also implemented a safe deletion workflow. Files older than 90 days, successfully backed up, and no longer present in the source are tagged for lifecycle-based cleanup instead of being deleted immediately. This approach ensures reliability, efficiency, and safety: backups scale seamlessly from small to massive datasets, compute resources are right-sized, and storage is continuously optimized. ## Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools DevFeed: [Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools]() Original publisher: [Read original article]() Author: R Tyler Croy Published: 2025-08-01T00:00:00Z Content type: tutorial Language: en Sources: [Scribd Tech]() Topics: [data](), [Amazon Web Services](), [databricks](), [event driven](), [Amazon Simple Queue Service (SQS)](), [Kafka](), [Open Source](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [architecture](), [aws](), [azure](), [catalog](), [cloud](), [cloud-native](), [data](), [databricks](), [deltalake](), [event-driven](), [featured](), [google-cloud](), [kafka](), [open-source](), [rust](), [sqs]() ### AI overview This article presents Scribd's cloud-native data-ingestion architecture for building large datasets for Delta Lake. It describes using AWS services and open-source tools such as kafka-delta-ingest, oxbow, and Airbyte in a more event-driven and reliable platform, with Databricks and Unity Catalog. The approach can also be adapted to Azure, Google Cloud Platform, or on-premises environments. ### Source excerpt Delta Lake is a fantastic technology for quickly querying massive data sets, but first you need those massive data sets! In this talk from Data and AI Summit 2025 I dive into the cloud-native architecture Scribd has adopted to ingest data from AWS Aurora, SQS, Kinesis Data Firehose and more! ## Terraform module to manage Oxbow Lambda and its components DevFeed: [Terraform module to manage Oxbow Lambda and its components]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-03-14T00:00:00Z Content type: release Language: en Sources: [Scribd Tech]() Topics: [Terraform](), [AWS Lambda](), [infrastructure as code (IAC)](), [Amazon Web Services](), [AWS Glue](), [Amazon Simple Queue Service (SQS)](), [data-processing](), [event driven](), [DynamoDB](), [Amazon S3]() Tags: [aws](), [aws-glue](), [aws-lambda](), [aws-s3](), [deltalake](), [dynamodb](), [event-driven](), [iac](), [oxbow](), [rust](), [security](), [sqs](), [terraform]() ### AI overview The article introduces terraform-oxbow, an open-source Terraform module for deploying and managing an Oxbow AWS Lambda workflow and its supporting components. It describes configurable integrations including AWS Glue, Kinesis Data Firehose, SQS, DynamoDB, IAM policies, and S3 notifications, while noting AWS notification limits and least-privilege considerations. ### Source excerpt Oxbow is a project to take an existing storage location which contains Apache Parquet files into a Delta Lake table. It is intended to run both as an AWS Lambda or as a command line application. We are excited to introduce terraform-oxbow, an open-source Terraform module that simplifies the deployment and management of AWS Lambda and its supporting components. Whether you're working with AWS Glue, Kinesis Data Firehose, SQS, or DynamoDB, this module provides a streamlined approach to infrastructure as code (IaC) in AWS.