# Curalate

A blog about developing at Curalate. How we handle big data architecture, design for the consumer web, and help our customers get the most out of their imagery.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Safely Modifying Your Hosts File with Gas Mask

DevFeed: [Safely Modifying Your Hosts File with Gas Mask](<https://devfeed.tech/articles/safely-modifying-your-hosts-file-with-gas-mask-26529.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2019/05/30/gas-mask.html>)

Author: Karl Shouler is a Software Engineer

Published: 2019-05-30T00:00:00Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [configuration](<https://devfeed.tech/topics/configuration.md>), [Microservice](<https://devfeed.tech/topics/microservice.md>), [browser](<https://devfeed.tech/topics/browser.md>), [Development](<https://devfeed.tech/topics/development.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [ui](<https://devfeed.tech/topics/ui.md>)

Tags: [browser](<https://devfeed.tech/tags/browser.md>), [cache](<https://devfeed.tech/tags/cache.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [go](<https://devfeed.tech/tags/go.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [ui](<https://devfeed.tech/tags/ui.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

This tutorial explains how to use Gas Mask, a UI-based hosts file manager, to maintain separate hosts files for locally testing microservices. It covers installation, creating and activating a hosts file, redirecting domains to 127.0.0.1, and restoring the original file to avoid unintended request routing.

### Source excerpt

Sometimes the DNS for a specific domain on your machine needs to point somewhere else - at Curalate, we test changes microservices locally before shipping them, which could require redirecting requests to look at that local instance. One way to do this is by adding an entry like 127.0.0.1 some.service.curalate.com to /etc/hosts. Why use a hosts file manager? In most cases, it's not advised to directly modify /etc/hosts. Because it's buried deep into the filesystem, it's easy to forget you've modified it, which can lead to numerous problems ranging from annoying to dangerous. Also, danger aside, it can begin to get messy and complex if you have a lot of entries to manage. Think of even just fifteen lines you're constantly commenting/uncommenting to represent the configuration you need at a given moment. This would be insanity. Gas Mask, a simple UI-based hosts file manager, allows you to set up different hosts files, while making it plainly obvious which hosts file is currently activated on your system via the OS Menu bar. Installation instructions Go to https://github.com/2ndalpha/gasmask and download the latest version. Unpack and install. On first-run, the only hosts file listed will be Original File which is the /etc/hosts file you'll no longer be modifying. Creating your first host file Create a new hosts file and name it something that makes sense. Add the test entry 127.0.0.1 google.com and save. The format of these entries is <target IP address> <URL or IP to redirect>. Activate that hosts file. Gasmask substitutes in this file at /etc/hosts. You may either need to flush your DNS cache or just restart the browser. To test it out, go to google.com in the browser. What happens now is, when the browser goes to get the IP for google.com, the OS sees the matching entry in your hosts file, then refers to 127.0.0.1 (your local computer) to make the request - which will fail. Go ahead and reactivate your Original File, restart the browser, and you should be able to a

## How to Setup a Scheduled Scala Spark Job

DevFeed: [How to Setup a Scheduled Scala Spark Job](<https://devfeed.tech/articles/how-to-setup-a-scheduled-scala-spark-job-26528.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2019/03/27/scheduled-scala-spark-job.html>)

Published: 2019-03-27T00:00:00Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Scala](<https://devfeed.tech/topics/scala.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [SDKs](<https://devfeed.tech/topics/sdks.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [cli](<https://devfeed.tech/tags/cli.md>), [daily](<https://devfeed.tech/tags/daily.md>), [data](<https://devfeed.tech/tags/data.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [emr](<https://devfeed.tech/tags/emr.md>), [github](<https://devfeed.tech/tags/github.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [job](<https://devfeed.tech/tags/job.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scheduled](<https://devfeed.tech/tags/scheduled.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [spark](<https://devfeed.tech/tags/spark.md>), [weekly](<https://devfeed.tech/tags/weekly.md>)

### AI overview

A tutorial for packaging and deploying a Scala Spark job as a fat JAR, uploading it to Amazon S3, and configuring it to run on a schedule through AWS Data Pipeline and EMR.

### Source excerpt

Have you written a Scala Spark job that processes a massive amount of data on an intimidating amount of RAM and you want to run it daily/weekly/monthly on a schedule on AWS? I had to do this recently, and couldn't find a good tutorial on the full process to get the spark job running. Included in this article and accompanying repository is everything you need to get your Scala Spark job running on AWS Data Pipeline and EMR. Code Repo This tutorial is not going to walk you through the process of actually writing your specific Scala Spark job to do whatever number crunching you need. There are already plenty of resources available (1, 2, 3) to get you started on that. The code template for setting up a Spark Scala job is available in this GitHub repo. Assuming that you have already written your Spark Job and are only using the AWS Java SDK to connect to your AWS data stores, drop your code in the Main function of SparkJob.scala and run the deploy.sh script to upload the fat jar to your S3 bucket. If you do take other dependencies, then it may take some extra work on your part. To run a Scala Spark job on AWS you need to compile a fat jar that contains the byte code for your job and all of the libraries it needs to run. This project already has the sbt-assembly plugin setup and a assemblyMergeStrategy set up to package the Spark, Hadoop, and AWS SDK together in the fat jar. If you need to add in other libraries that do not play well with each other, or are using a noncompatible version of Spark for this current repo, there are a few good resources available to help you through the needed build.sbt modifications. Outside of the previously mentioned needed changes you need to set a few parameters in the deploy.sh script. Mainly the deploymentPath to your specific S3 bucket, adding a profile to the AWS CLI command to upload to your specific S3 bucket if it's private, and changing the resulting fat jar name if you please. The deploy script uses the AWS CLI to upload the fat

## Load Testing APIs for Holiday Traffic and Dynamic Scalability

DevFeed: [Load Testing APIs for Holiday Traffic and Dynamic Scalability](<https://devfeed.tech/articles/hey-you-busy-i-have-thousands-of-questions-to-ask-you-26527.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/12/31/holiday-load-prep.html>)

Published: 2018-12-31T17:00:00Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Testing](<https://devfeed.tech/topics/testing.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [Network](<https://devfeed.tech/topics/network.md>), [API](<https://devfeed.tech/topics/api.md>), [data](<https://devfeed.tech/topics/data.md>), [Bash](<https://devfeed.tech/topics/bash.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [availability](<https://devfeed.tech/tags/availability.md>), [bash](<https://devfeed.tech/tags/bash.md>), [data](<https://devfeed.tech/tags/data.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [holiday](<https://devfeed.tech/tags/holiday.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [network](<https://devfeed.tech/tags/network.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

This engineering article explains how Curalate prepared its APIs and infrastructure for increased holiday traffic. It describes using historical traffic data to estimate peak requests per second, load testing with steadily increasing request rates, and dynamic scalability to avoid over-provisioning costs.

### Source excerpt

If you're a brick-and-mortar business owner, you quickly identify patterns in your customer foot traffic, especially around the holidays when achieving your sales goals depends on both timely and quality service. If you're an e-commerce business owner, like many of Curalate's 1,000+ customers, it's really no different: holiday sales are crucial to success and they depend heavily on your site's reliability. At Curalate, we take great pride in maintaining high availability and low latency for our client integrations throughout the year. But over the "Black Fiveday" period--Thanksgiving through Cyber Monday--and the week leading up to and including the day after Christmas, we see a roughly 5x increase in network requests to our APIs from our clients' sites. Therefore it's crucial that we both design our systems to handle that increased load and perform load testing on the systems ahead of time to prove that our designs work. This post describes how we carried out load tests of our infrastructure to prepare for the holiday traffic increase on our APIs. Additionally, it highlights how our approach towards dynamic scalability reduces costs by avoiding over-provisioning. Load Test Planning Our first question was: what volume of traffic can we expect? To answer that, we consulted the last several years of data describing our holiday traffic load pattern. Second: what are the important dimensions of that traffic? For example, do we expect a majority of the traffic to be cached or uncached? Do total requests matter or only instantaneous load? Since a previous blog post discussed cached versus uncached testing, this post focuses on API request rate. Total requests in a day is interesting, but only suggests an average requests-per-second (RPS) rate. The metric we're mostly interested in is the daily peak RPS rate. This gives us an idea of the busiest moment in our day, and if we can handle that rate, we should have confidence that we can handle lesser request rates at other times

## Uploading EC2 Logs to S3 on Shutdown

DevFeed: [Uploading EC2 Logs to S3 on Shutdown](<https://devfeed.tech/articles/uploading-ec2-logs-to-s3-on-shutdown-26526.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/09/04/ec2-log-upload.html>)

Published: 2018-09-04T08:56:00Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [Bash](<https://devfeed.tech/topics/bash.md>), [Ubuntu](<https://devfeed.tech/topics/ubuntu.md>), [chmod](<https://devfeed.tech/topics/chmod.md>), [cURL](<https://devfeed.tech/topics/curl.md>), [SIEM, Security, Observability](<https://devfeed.tech/topics/siem-security-observability.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [bash](<https://devfeed.tech/tags/bash.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [chmod](<https://devfeed.tech/tags/chmod.md>), [curl](<https://devfeed.tech/tags/curl.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [linux](<https://devfeed.tech/tags/linux.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [s3](<https://devfeed.tech/tags/s3.md>), [script](<https://devfeed.tech/tags/script.md>), [ubuntu](<https://devfeed.tech/tags/ubuntu.md>)

### AI overview

This tutorial shows how to use a Linux shutdown script on Ubuntu EC2 instances to upload selected logs to Amazon S3 before an Auto Scaling Group terminates a failed instance. It is intended to preserve evidence for debugging, including errors from C++ code that may bypass JVM or logging services.

### Source excerpt

If you've ever used an auto scaling group (ASG) on AWS, you've probably had an EC2 instance fail and get removed from the ASG. While great for redundancy (the ASG launches a new instance to start handling requests), it makes debugging the failure difficult since the ASG terminates the bad instance, erasing any evidence of what went wrong. Below, I present a script that will upload relevant files to S3 after an instance is triggered to shutdown but before it terminates. To achieve this, we make use of Linux's runlevel scripts. The instructions below are for Ubuntu, but it should be straight forward to migrate to a different distro. First, we make a script /etc/rc0.d/K01upload-logs. This script will run when the system is shutting down. You should change BUCKET, PATH, and LOG_FILE to match your needs. #!/bin/bash source /etc/environment # get strict after sourcing environment since we don't trust it... set -euo pipefail IFS=$'\n\t' # what logs should I upload and where to? LOG_FILE="/var/log/tomcat7/catalina.out" BUCKET="my-logs-bucket" # below we include the instance id in the path. That way it's easily findable. HOST=$(/usr/bin/curl http://169.254.169.254/latest/meta-data/instance-id) PATH="services/logs/$HOST/" # upload the logs /bin/echo "Uploading logs to s3://$BUCKET/$PATH" | /usr/bin/wall /usr/local/bin/aws s3 cp $LOG_FILE s3://$BUCKET/$PATH wait After installing the script, you need to set the permissions: chown root:root /etc/rc0.d/K01upload-logs chmod +x /etc/rc0.d/K01upload-logs And that's it! The script will upload the logs to your S3 bucket when the ASG terminates an instance. We've found this extremely helpful for our deep learning infrastructure that can often contain errors from C++ code (and thus isn't handled by the jvm or sent to our logging services).

## Why Curalate migrated from Caffe to MXNet for deep learning development and deployment

DevFeed: [Why Curalate migrated from Caffe to MXNet for deep learning development and deployment](<https://devfeed.tech/articles/how-curalate-uses-mxnet-on-aws-for-deep-learning-magic-26525.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/08/01/mxnet-case-study.html>)

Published: 2018-08-01T00:00:00Z

Content type: article

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [Framework](<https://devfeed.tech/topics/framework.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [Development](<https://devfeed.tech/topics/development.md>), [Microservice](<https://devfeed.tech/topics/microservice.md>), [ImageNet](<https://devfeed.tech/topics/imagenet.md>), [onnx](<https://devfeed.tech/topics/onnx.md>)

Tags: [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [deeplearning](<https://devfeed.tech/tags/deeplearning.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [development](<https://devfeed.tech/tags/development.md>), [fork](<https://devfeed.tech/tags/fork.md>), [framework](<https://devfeed.tech/tags/framework.md>), [model](<https://devfeed.tech/tags/model.md>), [mxnet](<https://devfeed.tech/tags/mxnet.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [python](<https://devfeed.tech/tags/python.md>), [scala](<https://devfeed.tech/tags/scala.md>)

### AI overview

Curalate describes migrating from Caffe to MXNet for computer vision systems. The post explains how MXNet supports experimentation, pre-trained models, model conversion, and integration into microservices and production environments.

### Source excerpt

This post was simultaneously published to Medium. At Curalate, we use state of the art deep learning and computer vision to add a layer of magic to our products. Intelligent Product Tagging, for example, identifies our clients' products in user-generated photos. Being a startup, we need to build these deep learning and computer vision systems the same way we do the rest of our products: quickly. Our computer vision systems are built in two phases, research and productization, and we require a deep learning framework that accelerates both. During the research phase, we need a framework that's quick to get started with and is flexible enough to experiment with new ideas. Once we have a solution, we need a framework that can easily be integrated into a microservice and deployed to multiple production environments. In the past, we used Caffe for experimentation and our own custom inference interface to deploy the trained models to production. Experimentation was slow due to Caffe's dated Python API, lack of automatic differentiation, unreliable build/install process, and clunky support for advanced layers which required us to maintain our own custom fork. Productization of Caffe was challenging since we had to maintain our own JNI interface. We needed new and modern framework that fulfilled all of our needs while saving us from the shortcomings of Caffe. After a review of all the available options, we decided to move to MXNet. In this post, we'll discuss why we migrated to MXNet as our deep learning framework of choice to facilitate our speed of experimentation, development, and deployment. Training and Experimentation Whenever we are faced with a new computer vision problem, we start by looking at existing state-of-the-art implementations. If we are lucky the functionality of the service we are implementing is similar to an existing pre-trained model for MXNet. MXNet has a fairly fleshed out and maintained Model Zoo that contains all of the standard pre-trained models

## Productionizing AWS ECS for production container workloads

DevFeed: [Productionizing AWS ECS for production container workloads](<https://devfeed.tech/articles/productionalizing-ecs-26524.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/05/16/productionalizing-ecs.html>)

Author: Anton Kropp was a Software Engineer

Published: 2018-05-16T09:45:39Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Containers](<https://devfeed.tech/topics/containers.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Docker](<https://devfeed.tech/topics/docker.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [containers](<https://devfeed.tech/tags/containers.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [docker](<https://devfeed.tech/tags/docker.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [ecs](<https://devfeed.tech/tags/ecs.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>)

### AI overview

This engineering article describes a company's migration from AMI-based virtual machines to AWS ECS containers and the challenges of making ECS production-ready. It covers cluster scaling, cluster image upgrades, container monitoring, cleanup of container artifacts, and remote debugging of JVM processes.

### Source excerpt

In January of last year we decided as a company to move towards containerization and began a migration to move onto AWS ECS. We pushed to move to containers, and off of AMI based VM deployments, in order to speed up our deployments, simplify our build tooling (since it only has to work on containers), get the benefits of being able to run our production code in a sandbox even locally on our dev machines (something you can't really do easily with AMI's), and lower our costs by getting more out of the resources we're already paying for. However, making ECS production ready was actually quite the challenge. In this post I'll discuss: Scaling the underlying ECS cluster Upgrading the backing cluster images Monitoring our containers Cleanup of images, container artifacts Remote debugging of our JVM processes Which is a short summary of the things we encountered and our solutions, finally making ECS a set it and forget it system. Scaling the cluster The first thing we struggled with was how to scale our cluster. ECS is a container orchestrator, analogous to Kubernetes or Rancher, but you still need to have a set of EC2 machines to run as a cluster. The machines all need to have the ECS Docker agent installed on it and ECS doesn't provide a way to automatically scale and manage your cluster for you. While this has changed recently with the announcement of Fargate, Fargate's pricing makes it cost prohibitive for organizations with a lot of containers. The general recommendation that AWS gave with ECS was to scale based on CPU reservation limit OR memory limit. There's no clear way to scale with a combination of the two, since auto scaling rules need to apply to a single CloudWatch metric or you face potential thrashing. Our first attempt on scaling was to try and scale on container placement failures. ECS logs a message when containers are unable to be placed due to constraints (not enough memory on the cluster, or not enough CPU reservation left), but there is no way to act

## Choosing a Deep Learning library for developing and deploying your App/Service

DevFeed: [Choosing a Deep Learning library for developing and deploying your App/Service](<https://devfeed.tech/articles/choosing-a-deep-learning-library-for-developing-and-deploying-your-app-service-26523.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/03/23/DL-lib-for-app-dev-and-prod.html>)

Published: 2018-03-23T10:11:36Z

Content type: article

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Library](<https://devfeed.tech/topics/library.md>), [App](<https://devfeed.tech/topics/app.md>), [Requirements](<https://devfeed.tech/topics/requirements.md>), [Code](<https://devfeed.tech/topics/code.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [browser](<https://devfeed.tech/topics/browser.md>)

Tags: [app](<https://devfeed.tech/tags/app.md>), [caffe](<https://devfeed.tech/tags/caffe.md>), [cntk](<https://devfeed.tech/tags/cntk.md>), [code](<https://devfeed.tech/tags/code.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [learning](<https://devfeed.tech/tags/learning.md>), [library](<https://devfeed.tech/tags/library.md>), [linux](<https://devfeed.tech/tags/linux.md>), [mxnet](<https://devfeed.tech/tags/mxnet.md>), [network](<https://devfeed.tech/tags/network.md>), [neural](<https://devfeed.tech/tags/neural.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

This article discusses how to choose a deep learning library for developing and deploying applications or services. Drawing on Curalate's experience using several libraries in production, it identifies factors such as application needs, deployment platforms, deep network architecture, API language requirements, and codebase quality.

### Source excerpt

Interest in deep learning is growing and growing and, with it at peak hype right now, a lot of people are looking to find the best deep learning library to build their new app or bring their company into the modern age. There are many deep learning toolkits to choose from ranging from the long used, supported, and robust academic libraries to the new state-of-the-art, industry backed platforms. At Curalate, we've been working on deep learning problems since 2014, meaning we've had the chance to watch the deep learning community and its open source libraries grow. We have also had the fortunate (unfortunate?) experience of using a few of the deep learning libraries in our production services and applications, and along the way, we have learned a lot about what to look for in a deep learning library to build reliable, production-ready applications and services. In this post, I'll share our lessons learned knowledge in hopes it will help you in your search for the perfect deep learning library match. You might even find that your best fit is using more than one! Important factors The specifics needs of your application/service The platform you are developing on and deploying to. Develop in OSX? Linux? Windows? Plan on having your application run in a web browser? A smart phone? A massive multi-node GPU cluster? It's not surprising that each of the libraries have prioritized different environments and some will work much better for your specific situation. The specific deep net architecture you are trying to implement If you are just trying to implement a typical, pre-trained classification net, this factor may not be as important for you. Some libraries are more performant and appropriate for certain types of deep nets (LSTMs, RNNs), but more on this later. API language requirements If you already have a code base written in language A, you probably would like to keep it that way without having to figure out some convoluted way to fit a deep net interface in language B

## R&D At Curalate: A Case Study of Deep Metric Embedding

DevFeed: [R&D At Curalate: A Case Study of Deep Metric Embedding](<https://devfeed.tech/articles/r-d-at-curalate-a-case-study-of-deep-metric-embedding-26522.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2018/02/01/deep-metric-embedding.html>)

Published: 2018-02-01T10:11:36Z

Content type: article

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [implementation](<https://devfeed.tech/topics/implementation.md>), [dataset](<https://devfeed.tech/topics/dataset.md>)

Tags: [case-study](<https://devfeed.tech/tags/case-study.md>), [computer](<https://devfeed.tech/tags/computer.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [e-commerce](<https://devfeed.tech/tags/e-commerce.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine](<https://devfeed.tech/tags/machine.md>), [metric](<https://devfeed.tech/tags/metric.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [research](<https://devfeed.tech/tags/research.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

A Curalate engineering case study describes building a visual search engine to identify clients' products in user-generated photos. It explains how a literature review led the team to use deep metric learning, which learns image embeddings that place images of the same product close together in Euclidean space.

### Source excerpt

At Curalate, we make social sell for hundreds of the world's largest brands and retailers. Our Fanreel product is a good example of this; it empowers brands to collect, curate, and publish social user-generated photos to their e-commerce site. A vital step in this pipeline is connecting the user generated content (UGC) to the product on our client's web site. Automating this process requires cutting edge computer vision techniques whose implementation details are not always clear, especially for production use cases. In this post, I review how we leveraged Curalate's R&D principles to build a visual search engine that identifies which of our clients' products are in user generated photos. The resulting system allows our clients to quickly connect user generated content to their e-comm site, enabling the UGC to generate revenue immediately upon distribution. Step 1: Do Your Homework We start every R&D project by hitting the books and catching up on the relevant research. This lets us understand what is feasible, the (rough) computational costs, and any pitfalls of various techniques. In this case, our goal is to find which products are in any UGC image using only the product images from the client's e-comm site. This is extremely difficult: UGC photos have dramatic lighting conditions, generally contain multiple objects or clutter, and may have undergone non rigid transformations (especially if it's a garment). Knowing we had a difficult problem on our hands, we did an extensive literature review on papers from leading computer vision conferences, journals, and even arxiv to ensure we have a good understanding of the state of the art. One approach stood out in the literature review: deep metric learning. Deep metric learning is a deep learning technique that learns an embedding function that, when applied to images of the same product, produces feature vectors that are close together in Euclidean space. This technique is perfect for our use case: we can train the sys

## Load Testing for Expected Increases in Traffic with Vegeta

DevFeed: [Load Testing for Expected Increases in Traffic with Vegeta](<https://devfeed.tech/articles/load-testing-for-expected-increases-in-traffic-with-vegeta-26521.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2017/12/21/expected-traffic-load-testing.html>)

Author: Reid Knight was a Software Engineer

Published: 2017-12-21T13:00:00Z

Content type: tutorial

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [Testing](<https://devfeed.tech/topics/testing.md>), [API](<https://devfeed.tech/topics/api.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [caching](<https://devfeed.tech/tags/caching.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [rate-limiting](<https://devfeed.tech/tags/rate-limiting.md>), [testing](<https://devfeed.tech/tags/testing.md>), [vegeta](<https://devfeed.tech/tags/vegeta.md>)

### AI overview

This article explains how Curalate plans and performs load testing for predictable increases in service and API traffic, including holiday peaks. It discusses dependencies, caching, staging versus production environments, rate limiting, and expected load across services.

### Source excerpt

At Curalate, our service and API traffic is fairly tightly coupled to e-commerce traffic, so any increase is reasonably predictable. We expect an increase in request rate towards the beginning of November each year, with traffic peaking at 10x our steady rate on Black Friday and Cyber Monday. Why Load Test? Curalate works directly with retail brands to drive traffic to their sites. The holiday shopping period is the most important time of the year for most of them, and we need to ensure that our experiences continue to operate at a high standard throughout. More generally, though, load testing is critical for services and APIs, especially in cases where load is expected to increase. It uncovers potential points of failure, during business hours, and hopefully prevents people from needing to wake up at 2 a.m. on a weekend. Creating a Test Plan In cases of expected load increases, it's important to understand as much as possible before diving into it. There are a few questions to ask: Is there any data available so I can understand the expected load? Is it a yearly increase - are previous years a good indication? If it's a brand new launch, what are the expectations? What are the hard and soft dependencies of the service or API that I'm testing? What sort of caching is in place? Does a 10x increase on my service cause a 10x increase on everything downstream, as well? Should we test against the active production environment, or is it feasible to spin up a staging environment with the same scaling behavior? Depending on the breadth of dependencies, it may not be possible to spin up a new duplicated environment. If I test against production, how can I ensure I don't negatively affect live traffic? Am I expecting an increase in load across services? If there are any core dependencies, what does the combined load look like at peak? How much of a buffer do I provide against the expected peak? Does my service have any rate limiting that I need to bypass or keep in mind? How

## Tracing High Volume Services

DevFeed: [Tracing High Volume Services](<https://devfeed.tech/articles/tracing-high-volume-services-26520.md>)

Original publisher: [Read original article](<http://engineering.curalate.com/2017/09/26/tracing-services.html>)

Author: Anton Kropp was a Software Engineer

Published: 2017-09-26T12:11:36Z

Content type: article

Language: en

Sources: [Curalate](<https://devfeed.tech/sources/curalate.md>)

Topics: [tracing](<https://devfeed.tech/topics/tracing.md>), [debugging](<https://devfeed.tech/topics/debugging.md>), [Logging and metrics](<https://devfeed.tech/topics/logging-and-metrics.md>), [Finagle](<https://devfeed.tech/topics/finagle.md>), [HTTP](<https://devfeed.tech/topics/http.md>)

Tags: [debugging](<https://devfeed.tech/tags/debugging.md>), [devops](<https://devfeed.tech/tags/devops.md>), [finagle](<https://devfeed.tech/tags/finagle.md>), [http](<https://devfeed.tech/tags/http.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logging-and-metrics](<https://devfeed.tech/tags/logging-and-metrics.md>), [okhttp](<https://devfeed.tech/tags/okhttp.md>), [scala](<https://devfeed.tech/tags/scala.md>), [services](<https://devfeed.tech/tags/services.md>), [soa](<https://devfeed.tech/tags/soa.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

The article explains how to trace requests across a high-volume service ecosystem by propagating a trace identifier through service calls and including it in logs. It describes Curalate's use of Finagle and custom OkHttp interceptors to place the identifier in HTTP headers, then reuse it in logging contexts and structured JSON logs.

### Source excerpt

We like to think that building a service ecosystem is like stacking building blocks. You start with a function in your code. That function is hosted in a class. That class in a service. That service is hosted in a cluster. That cluster in a region. That region in a data center, etc. At each level there's a myriad of challenges. From the start, developers tend to use things like logging and metrics to debug their systems, but a certain class of problems crops up when you need to debug across services. From a debugging perspective, you'd like to have a higher projection of the view of the system: a linearized view of what requests are doing. I.e. You want to be able to see that service A called service B and service C called service D at the granularity of single requests. Cross Service Logging The simplest solution to this is to require that every call from service to service comes with some sort of trace identifier. Incoming requests into the system, either from public API's or client side requests, or even from async daemon invoked timers/schedules/etc generates a trace. This trace then gets propagated through the entire system. If you use this trace in all your log statements you can now correlate cross service calls. How is this accomplished at Curalate? For the most part we use Finagle based services and the Twitter ecosystem has done a good job of providing the concept of a thread local TraceId and automatically propagating it to all other twitter-* components (yet another reason we like Finatra!). All of our service clients automatically pull this thread local trace id out and populate a known HTTP header field that services then pick up and re-assume. For Finagle based clients this is auto-magick'd for you. For other clients that we use, like OkHttp, we had to add custom interceptors that pulled the trace from the thread local and set it on the request. Here is an example of the header being sent automatically as part of Zipkin based headers (which we re-use