# Localytics Join our engineers where we talk about solutions to problems presented by big data and the race for big speed. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Saving Money and Protecting Privacy With Bloom Filters DevFeed: [Saving Money and Protecting Privacy With Bloom Filters]() Original publisher: [Read original article]() Author: Tristan Garwood Published: 2018-08-27T17:08:17Z Content type: article Language: en Sources: [Localytics]() Topics: [DynamoDB](), [data](), [pii](), [NoSQL](), [Amazon S3](), [Amazon Web Services](), [Back end](), [Caching](), [implementation](), [Latency]() Tags: [aws](), [backend](), [bloom-filter](), [cache](), [data](), [dynamodb](), [false-positive](), [filter](), [gdpr](), [implementation](), [latency](), [low-latency](), [memory](), [native](), [nosql](), [pii](), [privacy](), [s3]() ### AI overview The article explains how Localytics used Bloom filters with DynamoDB, S3, and caching to enforce GDPR opt-out checks in a high-throughput ingestion pipeline. The design reduces unnecessary DynamoDB lookups while preserving verification of possible matches and dropping data from opted-out users. ### Source excerpt The EU General Data Protection Regulation (GDPR) went into effect on May 25th, 2018 and stipulates that companies processing personally identifiable information (PII) must carry out requests from end users to be 'forgotten' from their systems. Implementing this right to be forgotten (RTBF) for our products means that ## Introducing Exploranda, a set of tools for exploring complex systems. DevFeed: [Introducing Exploranda, a set of tools for exploring complex systems.]() Original publisher: [Read original article]() Author: Raphael Luckom Published: 2018-01-25T18:02:35Z Content type: article Language: en Sources: [Localytics]() Topics: [Complex Systems](), [data](), [Tool](), [Monitoring](), [Graphs]() Tags: [ci](), [complex-systems](), [data](), [exploration](), [graphs](), [monitoring](), [systems](), [tools]() ### AI overview The article introduces Exploranda, a tool for exploring complex systems and supporting the early stages of data-driven decision-making. It describes how open-ended investigation differs from established tools such as monitoring, CI, and CD, especially when problems or causes are not obvious. ### Source excerpt At Localytics, data is central to our mission. One of our core principles is to "lead with data." When I envision that, I often think of an analyst using graphs to make a case for a business decision, or a principal engineer using benchmarks to advocate for a ## Infrastructure As Code With humidifier-reservoir DevFeed: [Infrastructure As Code With humidifier-reservoir]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2017-08-23T20:00:09Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [Infrastructure as code](), [AWS CloudFormation](), [Amazon Web Services](), [Provisioning]() Tags: [aws](), [aws-cloudformation](), [build](), [examples](), [infrastructure](), [infrastructure-as-code]() ### AI overview Localytics introduces humidifier-reservoir, an open-source tool for building AWS infrastructure with simple configuration files. The article explains its relationship to humidifier, AWS CloudFormation, and other infrastructure tools, and describes how configuration can map resource attributes and custom attributes for deployment. ### Source excerpt A little more than a year ago we made the first commit to the gem that eventually became humidifier. It's evolved quite a bit in the last year, including integrating AWS' resource specification which had the side-effect of greatly stabilizing the API. Here at Localytics, we& ## Connecting to Snowflake with Ruby on Rails DevFeed: [Connecting to Snowflake with Ruby on Rails]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2017-06-09T14:53:09Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [Ruby on Rails](), [API](), [data-processing](), [Cloud](), [Databases](), [SQL]() Tags: [activerecord](), [adapter](), [api](), [cloud](), [data](), [data-processing](), [open-source](), [ruby-on-rails](), [snowflake](), [sql]() ### AI overview A tutorial on connecting a Ruby on Rails API to the Snowflake data warehouse using unixODBC, Snowflake's ODBC driver, and an ActiveRecord adapter. It explains ODBC, DSN configuration, installation, and debugging. ### Source excerpt At Localytics, one of the tools we use for data processing is the Snowflake data warehouse. We connect to Snowflake in a couple different ways, but our main data retrieval application is a Ruby on Rails API. To accomplish this we use a combination of unixODBC (an open-source implementation ## Cogito: An Open-Source Abstraction for AWS IAM Policies DevFeed: [Cogito: An Open-Source Abstraction for AWS IAM Policies]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2017-06-07T16:43:12Z Content type: release Language: en Sources: [Localytics]() Topics: [AWS IAM](), [IAM](), [JSON](), [C](), [AWS CloudFormation]() Tags: [aws](), [aws-iam](), [c](), [cloudformation](), [iam](), [identity-and-access-management](), [json](), [microservices](), [open-source](), [permissions](), [policy](), [repositories](), [security](), [syntax-highlighting]() ### AI overview The article announces Cogito, an open-source abstraction for AWS IAM policy syntax. It introduces a human-readable syntax, a small C library called libcogito for translating between Cogito syntax and AWS JSON policies, local validation, source-control integration, editor syntax highlighting, and CloudFormation integration. ### Source excerpt Today we are open-sourcing Cogito, an abstraction of AWS IAM syntax. IAM (Identity and Access Management) is AWS's module that dictates the ability of various users and resources to mutate other resources within the AWS ecosystem. Permissions are described through a JSON-structured policy document that lives ## Faster Snowflake Queries through Clustering DevFeed: [Faster Snowflake Queries through Clustering]() Original publisher: [Read original article]() Author: Michal Klos Published: 2017-03-13T14:17:01Z Content type: article Language: en Sources: [Localytics]() Topics: [data-processing](), [Databases](), [Query (disambiguation)](), [Amazon Redshift]() Tags: [clustering](), [data-processing](), [database](), [databases](), [join](), [mpp](), [mysql](), [partitioning](), [performance](), [redshift](), [s3](), [scale](), [snowflake](), [time-series]() ### AI overview This article explains how Snowflake clustering and re-clustering can improve query performance by helping the query engine prune unneeded data. It discusses choosing cluster keys, comparisons with MPP database techniques, and the caveat that naturally ordered time-series data may not need clustering. ### Source excerpt At Localytics we have petabytes of data that needs to be served at low latencies and we use Snowflake in our mix of data processing technologies. Snowflake, like many other MPP databases, has a way of partitioning data to optimize read-time performance by allowing the query engine to prune ## ODBC and writing your own ActiveRecord adapter DevFeed: [ODBC and writing your own ActiveRecord adapter]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2017-03-07T22:08:02Z Content type: article Language: en Sources: [Localytics]() Topics: [Ruby on Rails](), [Rails](), [Ruby](), [Object-relational mapping](), [API]() Tags: [activerecord](), [api](), [database](), [rails](), [ruby](), [ruby-on-rails]() ### AI overview The article describes Localytics' open-source ODBC adapter for ActiveRecord, enabling Ruby on Rails applications to communicate with ODBC-compliant databases. It explains the adapter's role in a Rails API upgrade and recounts its development from an older fork, including support for multiple Rails branches and database environments. The supplied text ends before the account is complete. ### Source excerpt Today we are open-sourcing our ODBC adapter for ActiveRecord, which allows Ruby on Rails applications to communicate with ODBC-compliant databases. The impetus for this work was an effort to update one of our APIs to run with the latest Rails and ruby. Along the way we released Rails ## Why the Team Transitioned Its Frontend from Angular to React DevFeed: [Why the Team Transitioned Its Frontend from Angular to React]() Original publisher: [Read original article]() Author: Nate Tenczar Published: 2017-02-02T20:33:59Z Content type: opinion Language: en Sources: [Localytics]() Topics: [React](), [Angular](), [frontend development](), [JavaScript]() Tags: [angular](), [frameworks](), [frontend](), [javascript](), [react]() ### AI overview The article explains how the team evaluated its frontend development environment and decided to transition from Angular to React. It cites React's growing user base, state-management solutions, and existing internal use, while emphasizing that the opinions reflect the team's specific use case. ### Source excerpt If you sit quietly enough while reading this article you'll hear a sound. It's the faint tapping of fingers on a keyboard, off in the distance in parts unknown. Those self-assured fingers are putting together yet another think piece on JavaScript in this--the ## Best Practices and Common Mistakes with Travis CI DevFeed: [Best Practices and Common Mistakes with Travis CI]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2017-01-10T13:46:39Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [Travis CI](), [Continuous integration](), [Testing](), [build performance](), [GitHub]() Tags: [best-practices](), [continuous-integration](), [github](), [parallelism](), [speed](), [testing](), [travis-ci]() ### AI overview This article presents five internal best practices for using Travis CI, focusing on continuous integration, efficient testing, and reducing build time. It explains how Travis CI integrates with GitHub and supports configurable build environments, build matrices, and test parallelism. ### Source excerpt What is Travis At Localytics, we've worked hard to ensure that all of our services are built and deployed using our own internal best practices. One of those core best practices is that the service in question is using continuous integration (CI). CI greatly reduces the amount of ## Reducing DynamoDB Storage Costs for High-Volume Datapoint Deduplication DevFeed: [Reducing DynamoDB Storage Costs for High-Volume Datapoint Deduplication]() Original publisher: [Read original article]() Author: Joanna Solmon Published: 2017-01-04T14:45:00Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [DynamoDB](), [amazon](), [Database](), [Mobile]() Tags: [amazon](), [aws](), [cost](), [data](), [database](), [dedupe](), [devices](), [distributed](), [dynamodb](), [mobile](), [ops](), [storage]() ### AI overview The article describes how Localytics used DynamoDB to deduplicate billions of daily datapoints and examined schema changes to reduce storage costs. It discusses duplicate UUIDs, a 30-day deduplication history, and the limitations of a naive schema. ### Source excerpt Amazon's DynamoDB is a great technology: it's easy to set up and provides all the benefits of an infinitely scalable distributed database without the operational overhead. Unfortunately, DynamoDB can also be very expensive for high volume, high throughput storage. This post will show how we were ## Exploring CLI Best Practices DevFeed: [Exploring CLI Best Practices]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2016-10-11T19:25:23Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [Command-line interface](), [Ruby](), [Thor](), [Tooling](), [User Experience]() Tags: [bash](), [best-practices](), [cli](), [command-line](), [ruby](), [thor](), [tooling](), [user-experience]() ### AI overview Localytics describes best practices for designing intuitive command-line interfaces, based on its experience refining internal Ruby CLIs built with Thor. The recommendations include providing sensible defaults, offering readable options with short aliases, following common command-line conventions, and allowing users to identify files explicitly. ### Source excerpt At Localytics we build command line interfaces that manage our infrastructure and processes. This is our list of best practices that we've learned. ## Six Recipes for Software Managers DevFeed: [Six Recipes for Software Managers]() Original publisher: [Read original article]() Author: Adam Buggia Published: 2016-09-23T19:55:07Z Content type: tutorial Language: en Sources: [Localytics]() Topics: [engineering-culture](), [Software]() Tags: [best-practices](), [email](), [engineering](), [engineering-culture](), [meetings](), [post](), [software](), [survey](), [team]() ### AI overview Localytics engineering managers share practical practices for leading software teams, including weekly one-on-one meetings, preparation, employee-led discussion, action-item follow-through, and periodic surveys. ### Source excerpt The engineering managers at Localytics have been working together for over a year, and through ups and downs, we've grown and learned a lot as a team. The goal of this post is to share some of the lessons we've learned with the community. Some of ## Localytics' Process for Responding to Service Incidents DevFeed: [Localytics' Process for Responding to Service Incidents]() Original publisher: [Read original article]() Author: Tony Wieczorek Published: 2016-09-13T18:24:39Z Content type: article Language: en Sources: [Localytics]() Topics: [incident](), [systems](), [Slack](), [realtime]() Tags: [downtime](), [engineering](), [incident](), [pagerduty](), [postmortem](), [realtime](), [slack](), [systems]() ### AI overview Localytics describes its process for responding to service degradations and downtime. The process uses Slack and PagerDuty to coordinate triage, debugging, incident follow-up, postmortems, and communication with company leaders and customers. ### Source excerpt We believe the highest performing engineering teams have a process to identify, triage, fix and learn from service degradations and downtime. At Localytics, we build highly available and scalable systems, and part of the key to our success is learning from failures. One way we foster a learning culture is ## Humidifier - CloudFormation made easier DevFeed: [Humidifier - CloudFormation made easier]() Original publisher: [Read original article]() Author: Kevin Deisz Published: 2016-06-20T17:19:11Z Content type: release Language: en Sources: [Localytics]() Topics: [AWS CloudFormation](), [Amazon Web Services](), [Ruby](), [Provisioning](), [JSON](), [Terraform]() Tags: [aws](), [aws-cloudformation](), [infrastructure](), [json](), [provisioning](), [ruby](), [terraform]() ### AI overview Localytics open-sources Humidifier, a Ruby gem that generates programmatic objects for AWS CloudFormation resources. It abstracts JSON manipulation, generates Ruby classes from CloudFormation specifications, and supports maintaining and deploying CloudFormation stacks and change sets. ### Source excerpt Today we are open-sourcing Humidifier - one of the tools that we use internally to manage infrastructure on Amazon Web Services (AWS) at Localytics. This Ruby gem allows you to programmatically generate domain objects that represent the AWS CloudFormation (CFN) resources you want, handling the details of JSON manipulation under ## Testing AWS Scala Microservices DevFeed: [Testing AWS Scala Microservices]() Original publisher: [Read original article]() Author: Ben Darfler Published: 2016-04-27T11:56:52Z Content type: article Language: en Sources: [Localytics]() Topics: [Testing](), [Scala](), [Microservices](), [Amazon Web Services](), [Amazon S3](), [Amazon Simple Queue Service (SQS)](), [DynamoDB]() Tags: [architecture](), [aws](), [bugs](), [dynamodb](), [hackathon](), [metrics](), [microservices](), [microservices-architecture](), [s3](), [scala](), [slack](), [sqs](), [testing]() ### AI overview The article describes how Localytics addressed slow, fragile, and overused end-to-end tests in an AWS-based Scala microservices architecture. It introduces three open-sourced SBT plugins--sbt-s3, sbt-dynamodb, and sbt-sqs--and explains the testing problems that motivated them. ### Source excerpt In January and February of this year we open sourced three SBT plugins that have had a huge impact on our ability to test our AWS based Scala microservices architecture: sbt-s3 sbt-dynamodb sbt-sqs This is our story. Stop me if you've heard this one before.