# Target

Target Techologists blogging about what they do

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## REDstack

DevFeed: [REDstack](<https://devfeed.tech/articles/redstack-20397.md>)

Original publisher: [Read original article](<https://target.github.io/big%20data%20infrastructure/REDstack-Hadoop-as-a-Service>)

Author: Target Brands, Inc

Published: 2017-12-07T06:00:00Z

Content type: release

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [openstack](<https://devfeed.tech/topics/openstack.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apis](<https://devfeed.tech/tags/apis.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [big-data-infrastructure](<https://devfeed.tech/tags/big-data-infrastructure.md>), [chef](<https://devfeed.tech/tags/chef.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [components](<https://devfeed.tech/tags/components.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [customizable](<https://devfeed.tech/tags/customizable.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [development](<https://devfeed.tech/tags/development.md>), [docker-image](<https://devfeed.tech/tags/docker-image.md>), [druid](<https://devfeed.tech/tags/druid.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [https](<https://devfeed.tech/tags/https.md>), [kerberos](<https://devfeed.tech/tags/kerberos.md>), [ldap](<https://devfeed.tech/tags/ldap.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [openstack](<https://devfeed.tech/tags/openstack.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [provisioning](<https://devfeed.tech/tags/provisioning.md>), [python](<https://devfeed.tech/tags/python.md>), [sandbox](<https://devfeed.tech/tags/sandbox.md>)

### AI overview

REDstack is an open-source tool for provisioning secured, Kerberized Hadoop clusters on OpenStack, developed at Target for Big Data sandboxing and Chef cookbook integration testing. Its hdp-cloud cookbook installs components and configures clusters, and can also be used independently for manual provisioning. The Python orchestration application provisions resources through OpenStack APIs, coordinates parallel Chef deployments, and manages component installation over HTTPS. A bundled Docker image provides a local setup for configuration and dependencies. Build files allow customization of cluster and node sizes and roles, Hadoop settings, components, users, passwords, and secure assets; the excerpt also notes preconfigured LDAP, Kerberos, and user accounts. The supplied text is excerpted and cuts off during the project history.

### Source excerpt

REDstack is Now Open Source! We are officially open sourcing REDstack, our sandbox tool for Big Data development at Target. What is REDstack? REDstack is a tool for provisioning kerberized clusters on OpenStack. We created it with four goals in mind: Provide a secured environment, with the ability to leverage preconfigured LDAP and Kerberos servers. Out of the box usability, allowing you to log in with preconfigured user accounts. Custom user management utilities to administer the cluster. Provide a fully customizable experience, everything is a configuration option in your build files: Cluster size, node sizes, types of nodes and node roles, Hadoop configurations, heap sizes, and components, All users, passwords, and secure assets. Components REDstack is made up of two major components: hdp-cloud - The cookbook The cookbook is used by the application itself to install components and lay down cluster configuration. The cookbook can be used independently of REDstack to manually provision a cluster. REDstack - The orchestration component REDstack is a python application that performs all of the high-level complexities and timings associated with a full Hadoop installation: Orchestrates the provisioning of resources over OpenStack APIs, Controls and monitors parallel Chef deployment across the cluster, Manages and monitors cluster component install over HTTPS requests. REDstack is bundled with a Docker image, where the configs are set up locally before an installation, and all of the dependencies are updated and configured. How to Get Started Head over to the repository at https://github.com/target/redstack and follow along. The repo has instructions on how to build and configure the clusters using the included Docker image. History of the Project Target's Big Data Platform Team manages multiple Big Data environments, with hundreds of nodes and many PB's of data. As mentioned in our prior blog posts, we depend heavily on Chef as a core part of our CI/CD pipeline. Durin

## K8Guard

DevFeed: [K8Guard](<https://devfeed.tech/articles/k8guard-20409.md>)

Original publisher: [Read original article](<https://target.github.io/infrastructure/k8guard-the-guardian-angel-for-kuberentes>)

Author: Target Brands, Inc

Published: 2017-06-27T05:00:00Z

Content type: release

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [K8Guard](<https://devfeed.tech/topics/k8guard.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [audit](<https://devfeed.tech/tags/audit.md>), [automation](<https://devfeed.tech/tags/automation.md>), [availability](<https://devfeed.tech/tags/availability.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [container](<https://devfeed.tech/tags/container.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [docker](<https://devfeed.tech/tags/docker.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [email](<https://devfeed.tech/tags/email.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [ingress](<https://devfeed.tech/tags/ingress.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [k8guard](<https://devfeed.tech/tags/k8guard.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [monitors](<https://devfeed.tech/tags/monitors.md>), [namespaces](<https://devfeed.tech/tags/namespaces.md>), [notifications](<https://devfeed.tech/tags/notifications.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [platform](<https://devfeed.tech/tags/platform.md>), [pods](<https://devfeed.tech/tags/pods.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [security](<https://devfeed.tech/tags/security.md>)

### AI overview

K8Guard is an open-source Kubernetes auditing system that checks resources such as deployments, pods, jobs, ingresses, and namespaces for violations involving security, efficiency, availability, or stability. It notifies namespace owners and, after a configurable number of notices, can scale deployments to zero, suspend jobs, or delete ingresses; safe mode limits it to notifications. Its discover service can publish violations to Kafka or serve them through an API without Kafka. The action service processes Kafka messages and records actions in Cassandra, while the report service provides searchable records of past actions. The excerpt also describes 19 discover API metrics, including resource and violation counts, plus API-call timing measurements for Prometheus and Grafana dashboards.

### Source excerpt

K8Guard is Officially Open Source I am happy to announce that Target has open sourced K8Guard. I have been part of designing and developing it for the past few months, and I'm going to share a little more about it. What is K8Guard? K8Guard is an auditing system for Kubernetes clusters. It monitors different entities on your cluster for possible violations. K8Guard notifies the violators and then takes action on them. It also provides metrics and dashboards about violations in the cluster through Prometheus. How to Pronounce It? Like Kate Guard - the guardian angel for your Kubernetes clusters. Why? If you have large size kuberentes clusters and you care about security, efficiency, availability and stability, you need a tool to detect violations and do appropriate actions on them. What Kind of Violations Does It Discover? Violation Why Example Image Size Efficiency 5 GB image size Image Repo Security Downloading image from a shady repo Extra Capabilities Security Setting UID/GUID Privileged Mode Security Root containers Single Replica Availability Not 12-factor app Invalid Ingress Security/Stability Having *" in ingress Mount Host Vols Security/Stability Mounting Kubernetes system files No Owner Security No owner annotation for namespace What Kind of Entities Does It Monitor? Any entities which deployed to your kubernetes cluster such as Deployments, Pods, Jobs/CronJobs, Ingresses and namespaces. What Kind of Actions Does It Take? Notifies the namespace owner (email, hipchat, ...). After X amount of notifications, it will do a hard action such as: Scale bad deployments down to zero. Suspend bad jobs. Delete bad ingress Note that there is a safe mode - which only notifies and does not do hard actions. The K8Guard Design K8Guard has 3 main microservices (discover, action, report) Discover service, when in messaging mode, finds violations and puts them on a kafka topic. and also discover API mode, is able to serve without depending on kafka. you can hit the end points to

## Measuring the Performance of our OpenStack Cloud

DevFeed: [Measuring the Performance of our OpenStack Cloud](<https://devfeed.tech/articles/measuring-the-performance-of-our-openstack-cloud-20398.md>)

Original publisher: [Read original article](<https://target.github.io/cloudpunch>)

Author: Target Brands, Inc

Published: 2017-06-20T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [openstack](<https://devfeed.tech/topics/openstack.md>), [cloud-infrastructure](<https://devfeed.tech/topics/cloud-infrastructure.md>), [Service-Level Agreements](<https://devfeed.tech/topics/service-level-agreements.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloudpunch](<https://devfeed.tech/tags/cloudpunch.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [http](<https://devfeed.tech/tags/http.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linux](<https://devfeed.tech/tags/linux.md>), [load](<https://devfeed.tech/tags/load.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [openstack](<https://devfeed.tech/tags/openstack.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-testing](<https://devfeed.tech/tags/performance-testing.md>), [storage](<https://devfeed.tech/tags/storage.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

Target sought a way to measure instance and hardware performance in its private OpenStack cloud, both to evaluate hardware or driver changes and address customer concerns about performance. Rally focused on API functionality and stability; although it could boot instances and run commands, coordinating performance software and collecting results proved impractical. KloudBuster tested performance inside instances: HTTP requests per second and latency, plus storage IOPS and bandwidth using FIO. Its limited test extensibility and environment configuration, along with stability problems in Target's cloud, made it unsuitable. The excerpt says Target therefore chose to build a flexible performance framework, but does not provide further implementation details or results.

### Source excerpt

Here at Target, we run our own private OpenStack cloud and have never been able to accurately measure the performance of our hardware. This lack of measurement prevents the evaluation of performance improvements of new hardware or alternative technologies running as drivers inside OpenStack. It also prevents us from providing a Service Level Agreement (SLA) to our customers. Recently we have been striving to improve our OpenStack service which led us to talk to our consumers directly. One of the major feedback points provided by talking with our consumers was the performance of the OpenStack cloud was lower than expected. Because we have not measured the performance of our cloud in the past, we have been unable to know if new hardware or configuration changes improves consumer-facing performance. With our new OpenStack environment builds we focused on changing this. But first we needed a tool to do the job. Searching for a Tool The first tool we looked at was Rally. Rally does performance testing of an OpenStack cloud. However, Rally focuses on the OpenStack API only. It is mainly used to test functionality (via Tempest) and stability of the API under large amounts of load. Rally does contain a resource to boot an instance and run Linux CLI commands via user data. This was tested as a way to provide the staging of instances to run performance software. However, starting the software on each instance at the same time and collecting the results from said software was difficult and not viable. Because of this, we deemed Rally was not suitable for our needs. The next tool we looked at was KloudBuster. KloudBuster is a tool that does performance testing inside an OpenStack instance. At the time of writing it provides two sets of tests: HTTP and storage. The HTTP test uses a traffic generator to measure requests per second and latency between instances. The storage test uses FIO to measure read/write IOPs and bandwidth. KloudBuster does what we were looking for, measuring

## Target and Elasticsearch: Maintaining an ELK stack over Peak Season

DevFeed: [Target and Elasticsearch: Maintaining an ELK stack over Peak Season](<https://devfeed.tech/articles/target-and-elasticsearch-maintaining-an-elk-stack-over-peak-season-20410.md>)

Original publisher: [Read original article](<https://target.github.io/logging%20and%20metrics/elasticsearch-cloud>)

Author: Target Brands, Inc

Published: 2017-05-25T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [kibana](<https://devfeed.tech/topics/kibana.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [demand](<https://devfeed.tech/tags/demand.md>), [designing](<https://devfeed.tech/tags/designing.md>), [easy](<https://devfeed.tech/tags/easy.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [elk](<https://devfeed.tech/tags/elk.md>), [elk-stack](<https://devfeed.tech/tags/elk-stack.md>), [environments](<https://devfeed.tech/tags/environments.md>), [fast](<https://devfeed.tech/tags/fast.md>), [fields](<https://devfeed.tech/tags/fields.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [load](<https://devfeed.tech/tags/load.md>), [log-aggregation](<https://devfeed.tech/tags/log-aggregation.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logging-and-metrics](<https://devfeed.tech/tags/logging-and-metrics.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

Target's cloud ELK platform handled peak-season growth across 14 production Elasticsearch clusters, with more than 83 billion documents and nearly 100 terabytes of data. Teams used it for simple field searches, near-real-time monitoring and analytics through Kibana, and shared multi-tenant logging that pooled infrastructure across applications. Simple searches were described as relatively light, while the main challenge was serving teams with differing logging and metrics needs through a reliable shared platform. The excerpt also names Kafka, Consul, Chef, and Spinnaker among the tools used, and says the original Elasticsearch deployment began on version 1.4.4; it does not include the later operational lessons or practices.

### Source excerpt

One of the strongest benefits of launching an application into the cloud is the pure on-demand scalability that it provides. I've had the privilege of working with the ELK stack (Elasticsearch, Logstash, Kibana) for purposes of log aggregation for the past two years. When we started at that time, we were pleased with our performance on search and query times with 10's of gigabytes of data in the cluster in production. When Peak time hit, we reveled as our production clusters successfully managed half a terabyte of data(!). During peak, Target hosted 14 Elasticsearch clusters in the cloud containing more than 83 billion documents across nearly 100 terabytes in production environments alone. Consumers of these logs are able to get access to queries in blazing fast times with excellent reliability. It wasn't always that way though, and our team learned much about Elasticsearch in the process. What's The Use Case At Target? In a word, "vast." The many teams that use our platform for log aggregation and search are often times looking for a variety of things. Simple Search This one is easy, and the least resource intensive. Simply doing a match query and searching for fields within our data. Metrics / Analytics This one can be harder to accommodate at times, but some teams use our Elasticsearch clusters for near-realtime monitoring and Analytics using Kibana dashboards. Multi-tenant Logging Not necessarily consumer facing, but an interesting use for Elasticsearch is that we can aggregate many teams and applications into one cluster. In essence, this saves money over individual applications paying for infrastructure to log themselves. Simple search is the least of our concerns here. Queries add marginal load on the cluster, but often they are one-offs or otherwise infrequently used. However, the largest challenge faced here is multi-tenant demand. Different teams have very different needs for logging/metrics; designing a robust and reliable 'one-size-fits-all' platform is

## Surviving (and thriving) Through Peak Season 2016 on the Digital Observability Team

DevFeed: [Surviving (and thriving) Through Peak Season 2016 on the Digital Observability Team](<https://devfeed.tech/articles/surviving-and-thriving-through-peak-season-2016-on-the-digital-observability-team-20393.md>)

Original publisher: [Read original article](<https://target.github.io/agile/surviving-retail-peak-season-on-the-digital-observability-platform-team-a-scrum-master-s-perspective>)

Author: Target Brands, Inc

Published: 2017-05-23T05:00:00Z

Content type: opinion

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [engineering-leadership](<https://devfeed.tech/topics/engineering-leadership.md>), [Self-organizing Team](<https://devfeed.tech/topics/self-organizing-team.md>)

Tags: [2016](<https://devfeed.tech/tags/2016.md>), [agile](<https://devfeed.tech/tags/agile.md>), [application](<https://devfeed.tech/tags/application.md>), [black-friday](<https://devfeed.tech/tags/black-friday.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [culture](<https://devfeed.tech/tags/culture.md>), [devops](<https://devfeed.tech/tags/devops.md>), [elastic](<https://devfeed.tech/tags/elastic.md>), [forums](<https://devfeed.tech/tags/forums.md>), [health](<https://devfeed.tech/tags/health.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [logging](<https://devfeed.tech/tags/logging.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [observability](<https://devfeed.tech/tags/observability.md>), [organizational](<https://devfeed.tech/tags/organizational.md>), [overtime](<https://devfeed.tech/tags/overtime.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-tests](<https://devfeed.tech/tags/performance-tests.md>), [recognition](<https://devfeed.tech/tags/recognition.md>), [resources](<https://devfeed.tech/tags/resources.md>), [team](<https://devfeed.tech/tags/team.md>), [updates](<https://devfeed.tech/tags/updates.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

### AI overview

The observability team completed a major Elastic logging cluster upgrade intended to help digital teams monitor application health for Black Friday; the work involved overtime, late calls, and cross-team performance tests. The author credits leadership support--resources, trust, help with organizational blockers, and recognition--and a respectful, cooperative team culture. Senior engineers helped junior colleagues learn the full stack, while teammates covered on-call shifts and supported one another during incidents and personal demands. The excerpt begins a section on visibility but ends before its details.

### Source excerpt

It is 7:30 AM on a Monday morning in late October. I am waiting in line at Cafe Donuts to bring my team breakfast for our mandated 'no work for one hour'. We just wrapped up a strenuous week of implementing a major upgrade to our Elastic logging cluster. Many digital teams are relying on this upgrade to position themselves to confidently monitor their application health during the most important day in retail - Black Friday. It was a successful, much anticipated upgrade that resulted in many hours of overtime, late night calls, and cross-team performance tests. Morale is high, but it hasn't always been. It has been over three months since the team disassembled to work on new initiatives. I have since had a chance to reflect on the qualities of the team that made it so unique. I can't speak for my colleagues, but the accomplishments we made over six months are my proudest achievements in my five year tenure at Target. Leadership support Our leaders truly supported and enabled our work to be successful. They gave us the resources and confidence to confront organizational blockers and competing priorities. They helped us celebrate wins by providing recognition through updates to high level executives and advertising our achievements in forums. They took risks for us, trusted us to make decisions, and went to bat for us even when there was doubt that we could deliver on our promises. Respect This quality is near and dear to my heart. I never witnessed competition, resentment, distrust, or judgement among the team. Everyone truly respected one another and their unique perspectives. We were all aligned, working towards the same goals, and were in it together. Lead engineers partnered with junior engineers to ensure they understood the entire tech stack and could independently support our customers. I personally felt like my opinions and ideas mattered. On call schedules were set, but we supported one another through escalations and incidents. As a primary on call member,

## Hadoop Rolling Upgrades

DevFeed: [Hadoop Rolling Upgrades](<https://devfeed.tech/articles/hadoop-rolling-upgrades-20408.md>)

Original publisher: [Read original article](<https://target.github.io/infrastructure/hadoop-rolling-upgrades>)

Author: Target Brands, Inc

Published: 2017-05-22T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [zero downtime deployment](<https://devfeed.tech/topics/zero-downtime-deployment.md>)

Tags: [2016](<https://devfeed.tech/tags/2016.md>), [admin](<https://devfeed.tech/tags/admin.md>), [administration](<https://devfeed.tech/tags/administration.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [core](<https://devfeed.tech/tags/core.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [major](<https://devfeed.tech/tags/major.md>), [monthly](<https://devfeed.tech/tags/monthly.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [order](<https://devfeed.tech/tags/order.md>), [outage](<https://devfeed.tech/tags/outage.md>), [outages](<https://devfeed.tech/tags/outages.md>), [pattern](<https://devfeed.tech/tags/pattern.md>), [upgrades](<https://devfeed.tech/tags/upgrades.md>)

### AI overview

The team aimed to move from full-cluster Hadoop outages to rolling maintenance so it could deploy changes faster, reduce risk from bundling monthly changes, and avoid planned downtime for users. The excerpt covers a Hadoop Core 2.7.1-to-2.7.3 cluster running Hive, Tez, MapReduce, Spark, Pig, HBase, and Storm. They used a cluster administration tool to orchestrate upgrades and REDStack to repeatedly build and tear down secure clusters with production-like configurations for testing. One issue was HiveServer2 restarting on a new port during a multi-day upgrade; the team modified the stack upgrade XML to pause the process and shut down HS2 instances, reducing disruption. The excerpt ends before explaining the complete mitigation or upgrade results.

### Source excerpt

Hadoop upgrades over the last few years meant long outages where the Big Data platform team would shutdown the cluster, perform the upgrade, start services and then complete validation before notifying users it was ok to resume activity. This approach is a typical pattern for major upgrades even outside Target and reduces the complexity and risks associated with the upgrade. While this worked great for the platform team, it was not ideal for the hundreds of users and thousands of jobs that were dependent on the platform. That is why we decided to shake things up and go all in for rolling maintenance. Cluster Details: Hadoop (Core) 2.7.1 to 2.7.3 Mixed cluster workload running Hive, Tez, MR, Spark, Pig, HBase, Storm Goal Our March 2016 Hadoop upgrade was the turning point for rolling maintenance. With a large outage for the upgrade and monthly maintenance windows leading up to it, we decided to challenge ourselves with rolling maintenance to reach our uptime goals for core components. This would allow the platform team to deploy changes faster, reduce maintenance risk by not bundling changes together every month and more importantly not impact users with planned downtime. Drawing the line in the sand for rolling maintenance meant that it was time to get to work on our upgrade strategy. Reviewing the Playbook We started reviewing the upgrade process in November 2016 with the goal to upgrade our first admin cluster in December. The short turnaround time meant that we would leverage our cluster administration tool to handle the upgrade orchestration. The focus shifted towards understanding the order of events and evaluating the process for potential impacts. Running through the upgrades required a quick way to iterate through Hadoop cluster deployments and then tearing them down to retest. This is where REDStack, which is our internal cloud provisioning tool to build out a secure Hadoop cluster with production like configurations, came into play. After going through the

## How (and Why) We Moved to Spinnaker

DevFeed: [How (and Why) We Moved to Spinnaker](<https://devfeed.tech/articles/how-and-why-we-moved-to-spinnaker-20404.md>)

Original publisher: [Read original article](<https://target.github.io/how-and-why-we-moved-to-spinnaker>)

Author: Target Brands, Inc

Published: 2017-04-07T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [spinnaker](<https://devfeed.tech/topics/spinnaker.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [internal developer platform](<https://devfeed.tech/topics/internal-developer-platform.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>)

Tags: [automation](<https://devfeed.tech/tags/automation.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-platform](<https://devfeed.tech/tags/cloud-platform.md>), [code](<https://devfeed.tech/tags/code.md>), [continuous-deployment](<https://devfeed.tech/tags/continuous-deployment.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [git](<https://devfeed.tech/tags/git.md>), [green](<https://devfeed.tech/tags/green.md>), [image](<https://devfeed.tech/tags/image.md>), [images](<https://devfeed.tech/tags/images.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [jenkins](<https://devfeed.tech/tags/jenkins.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [layer](<https://devfeed.tech/tags/layer.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [multiple](<https://devfeed.tech/tags/multiple.md>), [spinnaker](<https://devfeed.tech/tags/spinnaker.md>)

### AI overview

Target chose Spinnaker after finding its homegrown cloud deployment platform cumbersome and out of step with cloud best practices. Spinnaker integrates with Jenkins, supports major public cloud providers, and uses immutable images containing code and configuration. To make the platform supportable and highly available for internal teams, engineers built packages from upstream Spinnaker components and used Terraform to manage separate stacks for blue-green updates without downtime. They also replaced Front50's default Cassandra persistence with cloud storage to reduce configuration overhead, and used the provider's cache service instead of managing a highly available Redis cluster.

### Source excerpt

Background Just after the middle of last year, Target expanded beyond its on-prem infrastructure and began deploying portions of target.com to the cloud. The deployment platform was homegrown (codename Houston), and was backed wholly by our public cloud provider. While in some aspects that platform was on par with other prominent continuous deployment offerings, the actual method of deploying code was cumbersome and not adherent to cloud best practices. These shortcomings led to a brief internal evaluation of various CI/CD platforms, which in turn led us to Spinnaker. We chose Spinnaker because it integrates with CI tools we already use at scale (Jenkins), supports deploying to all major public cloud providers, and compels software deployment best practices - all deployments are performed via immutable images, a snapshot of config + code. Supporting a Platform The primary goal of Target's cloud platform is to enable product teams to deploy and manage their applications across multiple cloud providers. We provide CI/CD, monitoring, and service discovery as services, and any application deployed via our platform gets those capabilities via a base image that is pre-configured for connectivity to each service's respective endpoint. Since these components are essentially products we provide to internal customers, we had to ensure the new CD platform was operationally supportable and highly-available. So, as soon as we decided on Spinnaker, a handful of engineers from the Cloud Platform group set about making this happen. Default Spinnaker scripts make it easy to standup a single self-contained server with the microservices and persistence layer all together, but that wasn't conducive to doing blue-green deployments - allowing updates of Spinnaker without downtime to our internal customers. We built jobs for building packages based off the master branch of each Spinnaker component's upstream git repository, and wrote Terraform plans to manage the deployment of each stack.

## Win the cloud with Winnaker!

DevFeed: [Win the cloud with Winnaker!](<https://devfeed.tech/articles/win-the-cloud-with-winnaker-20405.md>)

Original publisher: [Read original article](<https://target.github.io/infrastructure/Win_the_cloud_with_Winnaker>)

Author: Target Brands, Inc

Published: 2017-02-13T06:00:00Z

Content type: release

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [spinnaker](<https://devfeed.tech/topics/spinnaker.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [automation](<https://devfeed.tech/tags/automation.md>), [cd](<https://devfeed.tech/tags/cd.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [continuous-delivery](<https://devfeed.tech/tags/continuous-delivery.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [deployment-strategies](<https://devfeed.tech/tags/deployment-strategies.md>), [developers](<https://devfeed.tech/tags/developers.md>), [end-user](<https://devfeed.tech/tags/end-user.md>), [error](<https://devfeed.tech/tags/error.md>), [functionality](<https://devfeed.tech/tags/functionality.md>), [health-checks](<https://devfeed.tech/tags/health-checks.md>), [identity-management](<https://devfeed.tech/tags/identity-management.md>), [image](<https://devfeed.tech/tags/image.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [limit](<https://devfeed.tech/tags/limit.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [platform](<https://devfeed.tech/tags/platform.md>), [security](<https://devfeed.tech/tags/security.md>), [spinnaker](<https://devfeed.tech/tags/spinnaker.md>), [winnaker](<https://devfeed.tech/tags/winnaker.md>)

### AI overview

Winnaker automates end-to-end audits of Spinnaker deployments from an end user's perspective. The excerpt says component unit tests and health checks alone may miss failures involving connectivity between components, cloud provider API rate limits, or base infrastructure settings. Winnaker tests the whole deployment process with a sample app across accounts and regions, which the authors say is too time-consuming and repetitive to do manually. The excerpt also introduces automated troubleshooting but cuts off before describing how it works.

### Source excerpt

Win the cloud with Winnaker! I am happy to announce that we, at Target, decided to open source a tool called Winnaker. This tool will allow the user to audit Spinnaker from an end user point of view. But first what is Spinnaker? The first time I heard the word Spinnaker, my reaction was, "wait, what does that even mean in English?" Shortly after, I found myself implementing a demo of Spinnaker as a potential replacement for our internal cloud deployment tool. Spinnaker is a cloud agnostic continuous delivery tool, which means we can push our code to any cloud provider we like. In fact, Spinnaker takes agnosticism to the next level by introducing three abstractions. Load balancers Server groups Security groups By enforcing this level of simplicity, it allows the implementation of deployment strategies such as Highlander, Red/Black on a vast different type of infrastructure (VM, Container, Kubernetes, public cloud, private cloud) with a high level of confidence and an incredible level of ease of use for the app developers. Spinnaker also roots for the immutable infrastructure design pattern. Baking your image once and deploying the image everywhere is another bold move that differentiates Spinnaker from the other tools. Why Winnaker ? Short answer is because of automation! Test the functionality of the CD system as a whole. Spinnaker has different components (CloudDriver, Rosco, Deck,...). Each of these components have their own unit tests and health checks that can be monitored. We learned the hard way that relying only on component health checks is not effective enough to ensure developers won't face any error when they deploy their apps. A few things can go wrong when off monitoring radar: Connectivity between the separate components Maxing out cloud provider API rate limit Base infrastructure configurations (subnet address space, identity management roles) So we decided to audit Spinnaker and cloud's whole functionality with a sample app. If baking and deploying our

## How Target Performance Tunes Machine Learning Applications

DevFeed: [How Target Performance Tunes Machine Learning Applications](<https://devfeed.tech/articles/how-target-performance-tunes-machine-learning-applications-20400.md>)

Original publisher: [Read original article](<https://target.github.io/data%20science%20and%20engineering/numspark>)

Author: Target Brands, Inc

Published: 2016-09-29T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [applications](<https://devfeed.tech/tags/applications.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science-and-engineering](<https://devfeed.tech/tags/data-science-and-engineering.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [engine](<https://devfeed.tech/tags/engine.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [file-system](<https://devfeed.tech/tags/file-system.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [leverage](<https://devfeed.tech/tags/leverage.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [open-source-software](<https://devfeed.tech/tags/open-source-software.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [science](<https://devfeed.tech/tags/science.md>)

### AI overview

Target describes evaluating large-scale matrix multiplication to improve the speed and relevance of machine-learning algorithms built on billions of guest interactions and item data. The excerpt says this work requires computation across multiple nodes, high-throughput access to distributed data, storage and compute that scale with added resources, and fault tolerance as failures become more common. It names Apache Hadoop for storing and interacting with data, Apache Spark as the compute engine and a distributed machine-learning tool, and Apache Mahout for building and applying distributed algorithms. The article begins its practical guidance with paper-and-pencil examples: try matrices of different shapes and properties, choose compatible dimensions, and work through operations such as multiplication, inversion, and transposition. The excerpt cuts off during this list.

### Source excerpt

At Target we aim to make shopping more fun and relevant for our guests through extensive use of data - and believe me, we have lots of data! Tens of millions of guests and hundreds of thousands of items lead to billions of transactions and interactions. We regularly employ a number of different machine learning techniques on such large datasets for dozens of algorithms. We are constantly looking for ways to improve speed and relevance of our algorithms and one such quest brought us to carefully evaluate matrix multiplications at scale - since that forms the bedrock for most algorithms. If we make matrix multiplication more efficient, we can speed up most of our algorithms! Before we dig in, let me describe some properties of the landscape we will be working in. First, what do I mean by large scale? A large scale application, at a minimum, will require its computation to be spread over multiple nodes of a distributed computing environment to finish in a reasonable amount of time. These calculations will use existing data that are stored on a distributed file system that provides high-throughput access from the computing environment. Scalability, in terms of storage and compute, should grow as we add to these resources. As the system grows larger and more complex, failures will become more commonplace. Thus, software should be fault-tolerant. Fortunately, there is a lot of existing open-source software that we can leverage to work in such an environment, particularly Apache Hadoop for storing and interacting with our data, Apache Spark as the compute engine, and both Apache Spark and Apache Mahout for applying and building distributed machine learning algorithms. There are many other tools that we can add to the mix as well, but for the purposes of this post we will limit our discussion to these three. With that out of the way, lets dig in! Don't Forget the Basics Begin with good old paper and pencil. Yeah, I know this is about large scale matrix operations that you c

## (Data) Science or Witchcraft?

DevFeed: [(Data) Science or Witchcraft?](<https://devfeed.tech/articles/data-science-or-witchcraft-20399.md>)

Original publisher: [Read original article](<https://target.github.io/data%20science%20and%20engineering/dse-intro-one>)

Author: Target Brands, Inc

Published: 2016-03-01T06:00:00Z

Content type: opinion

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Data Science](<https://devfeed.tech/topics/data-science.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [computational complexity](<https://devfeed.tech/topics/computational-complexity.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [brain](<https://devfeed.tech/tags/brain.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-science-and-engineering](<https://devfeed.tech/tags/data-science-and-engineering.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [fashion](<https://devfeed.tech/tags/fashion.md>), [focus](<https://devfeed.tech/tags/focus.md>), [guests](<https://devfeed.tech/tags/guests.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [high-performance-computing](<https://devfeed.tech/tags/high-performance-computing.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [progress](<https://devfeed.tech/tags/progress.md>), [science](<https://devfeed.tech/tags/science.md>), [supervised-learning](<https://devfeed.tech/tags/supervised-learning.md>), [target](<https://devfeed.tech/tags/target.md>), [unsupervised-learning](<https://devfeed.tech/tags/unsupervised-learning.md>)

### AI overview

The excerpt presents data science as scientific modeling that can seem like magic, while grounding it in how people learn from experience and instruction to anticipate outcomes. It says advances in large-scale, high-performance computing have made complex calculations more practical on demand. At Target, the stated aim is to identify signals from each guest and tailor experiences to individual needs, rather than treating guests as statistics or relying only on popular trends. The excerpt ends mid-sentence, so it does not establish how this personalization is implemented.

### Source excerpt

On my first encounter with it, around early 2010's, I was mystified. It sounded like witchcraft and I imagined the practitioners to be a coven of witches and wizards, all holding Ph.D.s in the dark art of "Data Science" and being respectfully addressed as "Data Scientists". It was believed they would magically transform haystacks into gold and then ask for your first-born in return as a reward for their service (a la Rumpelstiltskin) There is no denying the fact that the title "Data Scientist" is the most coveted one these days and has a nice ring to it. It's also true that data science has traditionally been a monopoly of mathematicians and statisticians. Obviously, developing statistical models and machine learning algorithms requires years of training and practice to specialize. In my opinion it is more of an art form driven by science and can easily be mistaken for magic. It's common knowledge that the more experienced in life we get, the easier it is for us to make up our mind. For instance, "What diner to pick for a boy's-night-out?", "When to stay off highways to avoid being stuck in a traffic-jam?", "When to buy a house? When NOT to buy?", are all such decions we make everyday. This ability comes as result of years of learning from implicit experience (a.k.a unsupervised learning) and explicit instructions from parents, teachers, friends, family and media (a.k.a supervised learning.) Our brain builds models of the world, of the situations we have been in, of banal and extraordinary, of nice and not-so-nice, of appropriate and inappropriate etc. These models facilitate judgment, govern behavior and enable anticipation of likely outcomes. That's basically data science. The recent progress in large scale and high performance computing has opened doors for such complex calculations to be performed on-demand and much more efficiently than was possible before. Hence, the buzz! At Target we operate in a guest-centric universe. We don't treat our guests as a statist

## Bare Metal Big Data Builds

DevFeed: [Bare Metal Big Data Builds](<https://devfeed.tech/articles/bare-metal-big-data-builds-20406.md>)

Original publisher: [Read original article](<https://target.github.io/infrastructure/bare-metal-big-data-builds>)

Author: Target Brands, Inc

Published: 2016-01-13T06:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>)

Tags: [2014](<https://devfeed.tech/tags/2014.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [ambari](<https://devfeed.tech/tags/ambari.md>), [api](<https://devfeed.tech/tags/api.md>), [artifactory](<https://devfeed.tech/tags/artifactory.md>), [automated](<https://devfeed.tech/tags/automated.md>), [automation](<https://devfeed.tech/tags/automation.md>), [bare-metal](<https://devfeed.tech/tags/bare-metal.md>), [base](<https://devfeed.tech/tags/base.md>), [bash](<https://devfeed.tech/tags/bash.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [boot](<https://devfeed.tech/tags/boot.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [build](<https://devfeed.tech/tags/build.md>), [builds](<https://devfeed.tech/tags/builds.md>), [chef](<https://devfeed.tech/tags/chef.md>), [client](<https://devfeed.tech/tags/client.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [command](<https://devfeed.tech/tags/command.md>), [components](<https://devfeed.tech/tags/components.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [ironic](<https://devfeed.tech/tags/ironic.md>)

### AI overview

Target describes reducing Hadoop cluster builds from months to hours and adding hundreds of servers over a couple of weeks. After adopting Chef, Artifactory and Jenkins for OS configuration and operations, it automated the remaining server provisioning with OpenStack Ironic and the Nova Python client. Nova build parameters and user data start a bootstrap script that installs Chef, configures public keys and runs knife bootstrap. Chef role wrapper cookbooks then specify node types such as data, control and edge nodes, allowing attributes and run lists to be managed by role and environment. The excerpt explains the build foundation and server configuration flow but ends before detailing the full role build.

### Source excerpt

When you first think about scaling an on-premise Hadoop cluster your mind jumps to the process and the teams involved in building the servers, the time needed for configuring them and then the stability required while getting them into the cluster. Here at Target that process used to be measured in months. The story below outlines our journey around scaling our Hadoop cluster, taking the months to hours and adding hundreds of servers in a couple weeks. The Need Early 2013 taught us the lesson that manually managing Hadoop clusters, no matter how small is a time consuming and very repetitive task. Our next cluster build in 2014 drove the adoption of Chef, Artifactory and Jenkins to help with cluster operations. We stood up those components and created new role cookbooks to manage everything on the OS (configurations, storage, Kerberos, MySQL, etc.). While this was a step in the right direction, it left us with a manual process to still create the initial base server build and then add it to the cluster after configuring it with Chef. Build Foundation Closing the gap in our automation meant finding a way to deliver on true end to end builds, from an initial bootstrap to running jobs in your cluster. OpenStack's Ironic project was the first piece of the puzzle. Ironic gives us the ability to provision bare metal servers, similar to how OpenStack automated the VM build process. With Ironic as the foundation, we leveraged the Nova client to manage our instance builds. The nova python client interacts with the Compute service's API, giving us an easy way to specify our build parameters and spinning up an instance on one of our physical servers. The other key piece with the nova client is the ability to send boot information to the server using user data. The bash script sent executes several commands to install the Chef client, setup public keys and run the initial knife bootstrap to set the run list for the build. Example nova boot command: nova boot --image $image_name

## Real-time Big Data at Target

DevFeed: [Real-time Big Data at Target](<https://devfeed.tech/articles/real-time-big-data-at-target-20394.md>)

Original publisher: [Read original article](<https://target.github.io/analytics/big-data-storm>)

Author: Target Brands, Inc

Published: 2015-11-11T06:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [hdfs](<https://devfeed.tech/topics/hdfs.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [ambari](<https://devfeed.tech/tags/ambari.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-projects](<https://devfeed.tech/tags/apache-projects.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-components](<https://devfeed.tech/tags/open-source-components.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [resiliency](<https://devfeed.tech/tags/resiliency.md>), [storm](<https://devfeed.tech/tags/storm.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

Target's Big Data platform team sought a resilient, low-latency way to stream varied message formats from Apache Kafka into Hadoop and make the data immediately usable, using adaptable open source components. Apache Storm fit the team's preference for managing streaming through Apache Ambari. In a test that disabled and restored HDFS, however, the Storm topology became unrecoverable and needed a manual restart. Storm's one-message-at-a-time processing can provide low latency, but destination bolts often batch writes; batches set too large can delay delivery until they fill. The excerpt ends as the team examines Storm's source code, so it does not show the eventual solution or outcome.

### Source excerpt

An enterprise as large as Target generates a lot of data and on my Big Data platform team we want to make it as easy as possible for our users to get it into Hadoop in real-time. I want to discuss how we are starting to approach this problem, what we've done so far, and what is still to come. Our requirements We wanted to build a system with flexible open source components. Our experience with proprietary products on Hadoop is that they tend to be inflexible and only work with a narrow set of use cases. That can also be true of open source products, but we have found them to be easier to adapt to our needs. Indeed, that ended up being the case as we went through this journey and made several contributions to Apache projects. More specifically, we wanted to create a system that: was highly resilient to the failure of any individual component supported a variety of message formats delivered data to Hadoop with very low latency made the data immedaiately usable streamed data from Apache Kafka sources A Streaming Framework There are many excellent comparisons of streaming frameworks available, and I won't attempt to recreate them here. The first criteria we considered was what tooling was needed to monitor and administer the streaming framework, with a strong preference to use our primary Hadoop administration tool, Apache Ambari. Apache Storm fit that bill and was also a proven solution for stream processing. If Storm could meet our other requirements, it would be our first choice. To test its resiliency we ran a simple scenario: start a data stream into Hadoop, disable HDFS, and then reenable it. Streaming would obviously fail while HDFS was disabled, but we needed the system to recover gracefully when HDFS came back online. Unfortunately our first test of this scenario left our Storm topology in an unrecoverable state, which required a manual restart. That's not something we could live with. We also needed very fine control over the latency of arriving data. In gener

## The Dojo

DevFeed: [The Dojo](<https://devfeed.tech/articles/the-dojo-20403.md>)

Original publisher: [Read original article](<https://target.github.io/devops/the-dojo>)

Author: Target Brands, Inc

Published: 2015-08-14T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [software-development](<https://devfeed.tech/topics/software-development.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [Bootcamp](<https://devfeed.tech/topics/bootcamp.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [engineering-culture](<https://devfeed.tech/topics/engineering-culture.md>)

Tags: [2015](<https://devfeed.tech/tags/2015.md>), [agile](<https://devfeed.tech/tags/agile.md>), [automation](<https://devfeed.tech/tags/automation.md>), [building](<https://devfeed.tech/tags/building.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [change](<https://devfeed.tech/tags/change.md>), [channel](<https://devfeed.tech/tags/channel.md>), [chef](<https://devfeed.tech/tags/chef.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [culture](<https://devfeed.tech/tags/culture.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [develop](<https://devfeed.tech/tags/develop.md>), [development](<https://devfeed.tech/tags/development.md>), [devops](<https://devfeed.tech/tags/devops.md>), [dojo](<https://devfeed.tech/tags/dojo.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [experts](<https://devfeed.tech/tags/experts.md>), [faster](<https://devfeed.tech/tags/faster.md>), [learning](<https://devfeed.tech/tags/learning.md>), [leverage](<https://devfeed.tech/tags/leverage.md>), [openstack](<https://devfeed.tech/tags/openstack.md>)

### AI overview

Target's Dojo is a dedicated space where teams build expertise by working alongside resident subject matter experts, addressing limited expert availability as a barrier to scaling learning. Automation, Scrum, OpenStack, Chef, and Kafka specialists can support colocated project or product teams for days, weeks, or longer, then move between teams as training needs arise. The excerpt reports capacity for up to eight or nine teams at a time, a significant increase over Target's prior learning reach, but does not specify the previous capacity.

### Source excerpt

At Target we're always looking for ways to move forward in becoming the best omni-channel retailer that we can be. This journey demands that we enable change in most every part of our technology organization. Our culture, delivery model, technology selections, working arrangements and org structures are all levers we can pull to help us be more responsive. A key question that we continually ask ourselves is "how can we move faster"? Introducing change in a large enterprise can take a long time if we don't challenge ourselves to be creative around constraints (like not enough "experts" to go around). Traditional learning approaches might not be fast enough, so we've started acting on some innovative ideas to break down those barriers. While this progress has been great, we've continually been asking ourselves how we could go faster. ###Building Capacity One of the newest things we doing is expanding our capacity for others to learn the new ways of doing things. Since we still have relatively few subject matter experts to go around, we are finding that access to these experts is a constraint that makes it difficult for us to scale the learning process for new teams. To address this constraint, we are standing up a learning "Dojo." The Miriam-Webster dictionary defines a Dojo as "a school for training in various arts of self-defense (as judo or karate)." At ChefConf 2015, Adam Jacobs delivered a very interesting talk titled Chef Style DevOps Kungfu. This presentation talked about building your practice through repetition and development of skills. We needed a place to develop our own special DevOps Kungfu. Cue the Dojo! The Dojo is a dedicated space in which our subject matter experts take up residence for an extended time. This is the home base for automation engineers, advanced Scrum leaders, OpenStack engineers, Chef experts, Kafka engineers, etc. Project teams and product teams looking to leverage these experts to build their own expertise can colocate their teams

## Modernizing our IT Environment at Target

DevFeed: [Modernizing our IT Environment at Target](<https://devfeed.tech/articles/modernizing-our-it-environment-at-target-20396.md>)

Original publisher: [Read original article](<https://target.github.io/architecture/modernization_Intro>)

Author: Target Brands, Inc

Published: 2015-04-15T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [it modernization](<https://devfeed.tech/topics/it-modernization.md>), [Legacy Modernization](<https://devfeed.tech/topics/legacy-modernization.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>)

Tags: [2014](<https://devfeed.tech/tags/2014.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [build](<https://devfeed.tech/tags/build.md>), [business](<https://devfeed.tech/tags/business.md>), [data](<https://devfeed.tech/tags/data.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [devops](<https://devfeed.tech/tags/devops.md>), [digital](<https://devfeed.tech/tags/digital.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [integration](<https://devfeed.tech/tags/integration.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [legacy-systems](<https://devfeed.tech/tags/legacy-systems.md>), [modernization](<https://devfeed.tech/tags/modernization.md>), [modernizing](<https://devfeed.tech/tags/modernizing.md>), [organizations](<https://devfeed.tech/tags/organizations.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [release](<https://devfeed.tech/tags/release.md>), [target](<https://devfeed.tech/tags/target.md>), [teams](<https://devfeed.tech/tags/teams.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

A 2014 Marketing and Digital project illustrates how legacy systems slowed Target's effort to deliver newer capabilities. About 75% of project time went to integrating systems due to be retired, and the estimated legacy integration effort equaled 34 full-time workers for a year. Setting up the distributed environment took more than 20 weeks; source systems limited exposed data to 24 hours old; each release required six weeks of fully manual testing; and decisions involved eight teams across four organizations. The author links these burdens to accumulated technology debt and pressure to do more with less, move faster, and meet changing retail and digitally aware guest expectations while keeping existing systems running. The excerpt begins to introduce modernization shifts but does not provide their details.

### Source excerpt

In 2014, I worked on a project to build out a system to provide new business capabilities for our Marketing and Digital spaces. This initiative was not unlike other initiatives we undertake at Target, but it provides a great example of the importance of us investing in an effort to modernize the way we operate, the technology stacks we use and the way we think about systems and systems architecture. The project took in data in near real-time from multiple systems, processed the data and exposed it to consumers. The closer we came to real-time data the larger the benefit to our guests. There was only one problem - we had to keep one of our current systems running in parallel prior to switching on the new capabilities. In this scenario, there were a couple of key frustrations I experienced which are symbolic of what is driving our need to modernize: 75% of the time spent on the project was integration with systems which would be shut off shortly after we went live. The estimate for integration work with legacy systems was enough work to keep 34 full-time workers busy for an entire year. The majority of the processing for this system was in a distributed environment, standing up the environment took over 20 weeks. The data which would be exposed through this system would be 24 hours old due to limitations of the providing systems. Every time we went through a release, we would initiate a fully manual testing process which spanned six weeks. Decision making for this effort spanned over eight teams and four different organizations. From what I have seen, Target is not that different from the majority of large enterprises which have been in business for half a century. We have grown over time, transitioning from one technology paradigm to the next, adding more and more technology debt into our environment as we move quickly to provide business value to our internal business partners and our guests. I assume we are faced with the same pressures as other large enterprises:

## Robotics Analytics @ Target

DevFeed: [Robotics Analytics @ Target](<https://devfeed.tech/articles/robotics-analytics-target-20395.md>)

Original publisher: [Read original article](<https://target.github.io/analytics/robotics-analytics>)

Author: Target Brands, Inc

Published: 2015-04-09T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [FIRST](<https://devfeed.tech/topics/first.md>), [FIRST Robotics Competition](<https://devfeed.tech/topics/first-robotics-competition.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [automated](<https://devfeed.tech/tags/automated.md>), [automation](<https://devfeed.tech/tags/automation.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [competition](<https://devfeed.tech/tags/competition.md>), [complex](<https://devfeed.tech/tags/complex.md>), [control](<https://devfeed.tech/tags/control.md>), [curiosity](<https://devfeed.tech/tags/curiosity.md>), [data](<https://devfeed.tech/tags/data.md>), [dc](<https://devfeed.tech/tags/dc.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [electrical-engineering](<https://devfeed.tech/tags/electrical-engineering.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [equipment](<https://devfeed.tech/tags/equipment.md>), [first](<https://devfeed.tech/tags/first.md>), [interest](<https://devfeed.tech/tags/interest.md>), [labor](<https://devfeed.tech/tags/labor.md>), [machine](<https://devfeed.tech/tags/machine.md>), [robotics](<https://devfeed.tech/tags/robotics.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

Target's automated distribution centers produce machine data that could help improve operations, but engineers described a cumbersome process for collecting and interpreting it. Data was scattered across locations with duplicate file names, required server access, and had to be moved manually before spreadsheets and macros could parse and visualize it. The author introduced Splunk as a tool for bringing data from different sources into one place with built-in visualization. The excerpt does not describe the implementation or its results.

### Source excerpt

One important part of a retailer is their supply chain network. In order to sell a product in our stores, or online at Target.com, we need to have a well run and maintained network to move all of those products from one place to another. It may not sound like a complex problem to ship products from a distribution center to a store, but as soon as you start to have multiple vendors with multiple stores (not to mention online orders that go directly to a guest's home) it becomes increasingly complex. Retailers get products from companies, referred to as vendors, and then distribute the products to stores through central locations, referred to as distribution centers (DCs). In order to improve throughput of our supply chain network we expand our distribution network by building new distribution centers or we increasing the efficiency of the current network. A great way to improve performance of a DC is to put robots and other automation equipment in it. High tech equipment like this requires high skilled labor to maintain and manage the equipment. These highly automated systems run on servers and other control equipment and they produce a lot of machine data that can lead to valuable insights. Robotics has always been an interest of mine, starting with my experience in high school with the FIRST Robotics Competition and continuing today as I volunteer with FIRST. When I first joined Target right out of school with a degree in Electrical Engineering my curiosity and interest in robotics led me to get to know some of the engineers who worked on the robotics systems within Target's DCs. I mentioned to them that I was working with a tool called Splunk that did a good job of aggregating data from different sources to a central location with a visualization platform built in. The engineers said they had a very cumbersome, inconsistent manual process which involved multiple spreadsheets and home grown macros to gather and visualize data. The data was spread out through multip

## Outage Resolution Through Automation

DevFeed: [Outage Resolution Through Automation](<https://devfeed.tech/articles/outage-resolution-through-automation-20401.md>)

Original publisher: [Read original article](<https://target.github.io/devops/outage-resolution-through-automation>)

Author: Target Brands, Inc

Published: 2014-12-29T06:00:00Z

Content type: news

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [infrastructure monitoring](<https://devfeed.tech/topics/infrastructure-monitoring.md>), [Monitoring & Alerting](<https://devfeed.tech/topics/monitoring-alerting.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [DevOps](<https://devfeed.tech/topics/devops.md>), [Jenkins](<https://devfeed.tech/topics/jenkins.md>)

Tags: [2014](<https://devfeed.tech/tags/2014.md>), [agile](<https://devfeed.tech/tags/agile.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [automation](<https://devfeed.tech/tags/automation.md>), [chef](<https://devfeed.tech/tags/chef.md>), [code](<https://devfeed.tech/tags/code.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data](<https://devfeed.tech/tags/data.md>), [devops](<https://devfeed.tech/tags/devops.md>), [feature](<https://devfeed.tech/tags/feature.md>), [flashbuilds](<https://devfeed.tech/tags/flashbuilds.md>), [github](<https://devfeed.tech/tags/github.md>), [health](<https://devfeed.tech/tags/health.md>), [jenkins](<https://devfeed.tech/tags/jenkins.md>), [kanban](<https://devfeed.tech/tags/kanban.md>), [kanban-boards](<https://devfeed.tech/tags/kanban-boards.md>), [key](<https://devfeed.tech/tags/key.md>), [kitchen](<https://devfeed.tech/tags/kitchen.md>), [managed](<https://devfeed.tech/tags/managed.md>), [metric](<https://devfeed.tech/tags/metric.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [outage](<https://devfeed.tech/tags/outage.md>)

### AI overview

Project Argus used a monitoring-as-code setup to automate KPI collection and dashboards. Each Chef cookbook represented a business function's KPIs, connected to data sources, created processing scripts, and scheduled them through crontab. Processing ran on separate on-site servers, while dashboards were decoupled and cookbook definitions were dashboard agnostic, allowing presentation changes or a dashboard switch. Dashboards also detected missing data and sent chat alerts. When the monitoring servers went down on 2014-12-06, the dashboards went blank and the health monitors alerted within minutes. The excerpt ends before describing how the outage was resolved.

### Source excerpt

Recently we launched Project Argus, a 30-day "monitoring challenge" to improve visibility of key performance indicators (KPIs) across our technology stack prior to our peak retail season (in Greek Mythology, Argus is a 100-eyed giant). This effort was structured as a mix between a FlashBuild and agile. We used two day sprints, twice a day stand-ups, and feature tracking through Kanban boards. Quickly in this effort we decided to build our product as monitoring-as-code. Through the use of tools such as GitHub, Chef, Kitchen, Jenkins, and Ruby, we were able to quickly build several monitoring and dashboard solutions for use within our Technology Operations Center. These dashboards and the iterative process we use to continue delivering more content have been embraced by our support teams who now heavily rely on them to proactively detect and resolve issues. One deliverable from Argus includes a cookbook per core business function that represents the KPIs identified by each business product owner. When run, the cookbooks establish connections to our data sources, build the scripts that process our data, and create crontab entries to automatically run the scripts at our predefined intervals. All of these resources and actions are defined in code. We designed our solution so that the dashboards are decoupled from our centralized processing and metric creation. Additionally, our cookbooks are written to be dashboard agnostic and independent so that anyone can make changes to the appearance and we can switch dashboard solutions easily. The processing of our metrics is centralized and managed on-site with each cookbook receiving its own server. Finally, our dashboard solution detects lapses in data and generates alerts that post into our persistent chat client. This allows us to close the loop on the health of the monitoring system, which can be a painful endeavor! ##The Problem Unfortunately, on 2014-12-06 at around 1:30pm the servers we use to process the monitoring data

## Target FlashBuilds

DevFeed: [Target FlashBuilds](<https://devfeed.tech/articles/target-flashbuilds-20402.md>)

Original publisher: [Read original article](<https://target.github.io/devops/target-flashbuilds>)

Author: Target Brands, Inc

Published: 2014-11-10T06:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [DevOps](<https://devfeed.tech/topics/devops.md>), [Self-organizing Team](<https://devfeed.tech/topics/self-organizing-team.md>), [engineering-leadership](<https://devfeed.tech/topics/engineering-leadership.md>)

Tags: [agile](<https://devfeed.tech/tags/agile.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [devops](<https://devfeed.tech/tags/devops.md>), [flashbuilds](<https://devfeed.tech/tags/flashbuilds.md>), [fun](<https://devfeed.tech/tags/fun.md>), [innovation](<https://devfeed.tech/tags/innovation.md>), [it](<https://devfeed.tech/tags/it.md>), [journey](<https://devfeed.tech/tags/journey.md>), [organization](<https://devfeed.tech/tags/organization.md>), [people](<https://devfeed.tech/tags/people.md>), [philosophies](<https://devfeed.tech/tags/philosophies.md>), [principles](<https://devfeed.tech/tags/principles.md>), [problems](<https://devfeed.tech/tags/problems.md>), [process](<https://devfeed.tech/tags/process.md>), [questions](<https://devfeed.tech/tags/questions.md>), [radius](<https://devfeed.tech/tags/radius.md>), [scrum](<https://devfeed.tech/tags/scrum.md>), [service](<https://devfeed.tech/tags/service.md>), [stuff](<https://devfeed.tech/tags/stuff.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

Target describes FlashBuilds as one-day sessions with two agile sprints, the needed participants, and a facilitator, aimed at tackling difficult business challenges while improving communication and relationships across teams. The approach is intended to reduce organizational and process barriers and give teams more control over how work gets done. Target says sessions can produce new products and faster problem solving, but not every session yields a finished, viable product; even unsuccessful sessions provide learnings and greater appreciation for partner teams. This overview is based on the supplied excerpt.

### Source excerpt

At a couple of recent conferences around DevOps (MSP DevOps Days and DevOps Enterprise Summit), Heather Mickman and Ross Clanton from Target explained some high level concepts and actions we're taking to break our mold of working on complex problems. When presented with an opportunity to work differently and challenge our normal delivery of IT assets - from plan to decommission - what would that look like? First, we asked some questions: What can we do to address the barriers within our organization; can we remove that impact on our ability to get things done? What can we do to remove time spent moving from meeting to meeting, often recapping the same discussions that you had in your last meeting 2 weeks ago? What process barriers exist that can be included in the delivery and built as part of the 'service'? And then we asked ... What if we could alleviate all of these problems (and more) all at the same time, all while you build team morale and actually have fun (gasp!) building stuff? What if? Two simple words that can: spark amazing conversations; generate innovation and excitement; challenge the status quo; break through to show how DevOps philosophies break through - even with the big horses. A ton of this post is simply about getting stuff done (GSD) but also to get into some specifics around how we approached fixing a business problem in some new ways - at least for us. So let's embark on some early details around Target's journey to challenge the traditional approaches to get stuff done and take control of the 'how' ... to redefine our working philosophies, principles, and minimize the impact radius of our efforts while still going deep with the teams and persons engaged. Any endeavor such as this has to have a name ... We affectionately call this process FlashBuilds! What is a FlashBuild?! Glad you asked! FlashBuild: A one-day session that includes 2 agile sprints, all the smart people that you need to build stuff, a facilitator, and a whole lot of positive focus

## Target Joins the W3C

DevFeed: [Target Joins the W3C](<https://devfeed.tech/articles/target-joins-the-w3c-20412.md>)

Original publisher: [Read original article](<https://target.github.io/standards/target-joins-w3c>)

Author: Target Brands, Inc

Published: 2014-10-15T05:00:00Z

Content type: news

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [W3C](<https://devfeed.tech/topics/w3c.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>), [web-standards](<https://devfeed.tech/topics/web-standards.md>), [developer-relations](<https://devfeed.tech/topics/developer-relations.md>)

Tags: [2014](<https://devfeed.tech/tags/2014.md>), [2015](<https://devfeed.tech/tags/2015.md>), [content-distribution](<https://devfeed.tech/tags/content-distribution.md>), [customers](<https://devfeed.tech/tags/customers.md>), [developer](<https://devfeed.tech/tags/developer.md>), [developer-community](<https://devfeed.tech/tags/developer-community.md>), [inc](<https://devfeed.tech/tags/inc.md>), [internet-of-things](<https://devfeed.tech/tags/internet-of-things.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [location](<https://devfeed.tech/tags/location.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [retail](<https://devfeed.tech/tags/retail.md>), [services](<https://devfeed.tech/tags/services.md>), [standards](<https://devfeed.tech/tags/standards.md>), [target](<https://devfeed.tech/tags/target.md>), [technology](<https://devfeed.tech/tags/technology.md>), [w3c](<https://devfeed.tech/tags/w3c.md>), [web](<https://devfeed.tech/tags/web.md>), [web-services](<https://devfeed.tech/tags/web-services.md>), [world-wide-web](<https://devfeed.tech/tags/world-wide-web.md>)

### AI overview

Target joined the W3C to collaborate with other organizations on technology it sees as important to retail and customer experiences. Its initial interests are web services, location-based content distribution--including iBeacons and smart stores--and the Internet of Things. Target says it plans to engage in open dialogue and collaborative work, contribute ideas, work with the developer community, and share progress.

### Source excerpt

Tuesday, October 14th 2015, Target affirmed its commitment to technology and leadership in the retail space by joining the World Wide Web Consortium (W3C). As a company that cares about delivering great experiences for customers, we realize those experiences are now steeped in technology. One way Target can have a significant impact on shaping technology is open engagement with other companies and organizations at the heart of emerging development - the W3C enables us to do just that. So you joined - now what? There are so many areas of technology important to retail and associated businesses. Initially, we're interested in engaging with tech leaders, businesses, and organizations on web services, location-based content distribution (iBeacons, smart stores, etc.), and the Internet of Things. Through open dialogue and collaborative work in the W3C, we seek to push these areas of technology forward to enable new and better experiences for our customers. More to come soon! This is just the beginning. As an organization we will be engaging with the developer community in new and different ways, proposing ideas on the technologies that matter to us, and providing feedback of our progress. Now let's get to work! Target Joins the W3C was originally published by Target Brands, Inc at target tech on October 15, 2014.