# Target Target Techologists blogging about what they do This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack DevFeed: [REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-12-07T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Provisioning](), [Hadoop](), [openstack](), [Orchestration](), [Python](), [Open Source](), [Docker]() Tags: [big-data](), [big-data-infrastructure](), [chef](), [cloud](), [docker](), [druid](), [elasticsearch](), [hadoop](), [open-source](), [openstack](), [orchestration](), [provisioning](), [python]() ### AI overview REDstack is an open-source sandbox tool for Big Data development that provisions kerberized Hadoop clusters on OpenStack. It combines a cookbook for installing and configuring cluster components with a Python orchestration application that manages resource provisioning, Chef deployment, and component installation. ### Source excerpt REDstack is Now Open Source! We are officially open sourcing REDstack, our sandbox tool for Big Data development at Target. What is REDstack? REDstack is a tool for provisioning kerberized clusters on OpenStack. We created it with four goals in mind: Provide a secured environment, with the ability to leverage preconfigured LDAP and Kerberos servers. Out of the box usability, allowing you to log in with preconfigured user accounts. Custom user management utilities to administer the cluster. Provide a fully customizable experience, everything is a configuration option in your build files: Cluster size, node sizes, types of nodes and node roles, Hadoop configurations, heap sizes, and components, All users, passwords, and secure assets. Components REDstack is made up of two major components: hdp-cloud - The cookbook The cookbook is used by the application itself to install components and lay down cluster configuration. The cookbook can be used independently of REDstack to manually provision a cluster. REDstack - The orchestration component REDstack is a python application that performs all of the high-level complexities and timings associated with a full Hadoop installation: Orchestrates the provisioning of resources over OpenStack APIs, Controls and monitors parallel Chef deployment across the cluster, Manages and monitors cluster component install over HTTPS requests. REDstack is bundled with a Docker image, where the configs are set up locally before an installation, and all of the dependencies are updated and configured. How to Get Started Head over to the repository at https://github.com/target/redstack and follow along. The repo has instructions on how to build and configure the clusters using the included Docker image. History of the Project Target's Big Data Platform Team manages multiple Big Data environments, with hundreds of nodes and many PB's of data. As mentioned in our prior blog posts, we depend heavily on Chef as a core part of our CI/CD pipeline. Durin ## K8Guard: An Open-Source Auditing System for Kubernetes Clusters DevFeed: [K8Guard: An Open-Source Auditing System for Kubernetes Clusters]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-06-27T05:00:00Z Content type: release Language: en Sources: [Target]() Topics: [Kubernetes](), [Security](), [Monitoring](), [Microservices](), [Prometheus](), [Grafana](), [API](), [Kafka](), [Database]() Tags: [api](), [audit](), [automation](), [cassandra](), [cloud](), [container](), [dashboards](), [docker](), [grafana](), [infrastructure](), [k8guard](), [kafka](), [kubernetes](), [microservices](), [monitoring](), [platform](), [prometheus](), [security]() ### AI overview Target open-sourced K8Guard, an auditing system for Kubernetes clusters. It monitors deployments, pods, jobs, ingresses, and namespaces for violations, notifies owners, and can take automated actions such as scaling deployments down, suspending jobs, or deleting ingresses. It also provides API access, Prometheus metrics, and Grafana dashboards. ### Source excerpt K8Guard is Officially Open Source I am happy to announce that Target has open sourced K8Guard. I have been part of designing and developing it for the past few months, and I'm going to share a little more about it. What is K8Guard? K8Guard is an auditing system for Kubernetes clusters. It monitors different entities on your cluster for possible violations. K8Guard notifies the violators and then takes action on them. It also provides metrics and dashboards about violations in the cluster through Prometheus. How to Pronounce It? Like Kate Guard - the guardian angel for your Kubernetes clusters. Why? If you have large size kuberentes clusters and you care about security, efficiency, availability and stability, you need a tool to detect violations and do appropriate actions on them. What Kind of Violations Does It Discover? Violation Why Example Image Size Efficiency 5 GB image size Image Repo Security Downloading image from a shady repo Extra Capabilities Security Setting UID/GUID Privileged Mode Security Root containers Single Replica Availability Not 12-factor app Invalid Ingress Security/Stability Having *" in ingress Mount Host Vols Security/Stability Mounting Kubernetes system files No Owner Security No owner annotation for namespace What Kind of Entities Does It Monitor? Any entities which deployed to your kubernetes cluster such as Deployments, Pods, Jobs/CronJobs, Ingresses and namespaces. What Kind of Actions Does It Take? Notifies the namespace owner (email, hipchat, ...). After X amount of notifications, it will do a hard action such as: Scale bad deployments down to zero. Suspend bad jobs. Delete bad ingress Note that there is a safe mode - which only notifies and does not do hard actions. The K8Guard Design K8Guard has 3 main microservices (discover, action, report) Discover service, when in messaging mode, finds violations and puts them on a kafka topic. and also discover API mode, is able to serve without depending on kafka. you can hit the end points to ## Measuring the Performance of our OpenStack Cloud DevFeed: [Measuring the Performance of our OpenStack Cloud]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-06-20T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [openstack](), [Performance Testing](), [Cloud](), [Hardware](), [HTTP](), [Latency](), [Command-line interface](), [Linux]() Tags: [cli](), [cloudpunch](), [configuration](), [hardware](), [http](), [latency](), [linux](), [openstack](), [performance](), [performance-testing]() ### AI overview This article describes Target's effort to measure performance in its private OpenStack cloud. It evaluates Rally and KloudBuster, finding that Rally focuses on API testing while KloudBuster supports HTTP and storage tests but lacks sufficient extensibility, configuration, and stability. The authors therefore decide to create their own flexible performance framework. ### Source excerpt Here at Target, we run our own private OpenStack cloud and have never been able to accurately measure the performance of our hardware. This lack of measurement prevents the evaluation of performance improvements of new hardware or alternative technologies running as drivers inside OpenStack. It also prevents us from providing a Service Level Agreement (SLA) to our customers. Recently we have been striving to improve our OpenStack service which led us to talk to our consumers directly. One of the major feedback points provided by talking with our consumers was the performance of the OpenStack cloud was lower than expected. Because we have not measured the performance of our cloud in the past, we have been unable to know if new hardware or configuration changes improves consumer-facing performance. With our new OpenStack environment builds we focused on changing this. But first we needed a tool to do the job. Searching for a Tool The first tool we looked at was Rally. Rally does performance testing of an OpenStack cloud. However, Rally focuses on the OpenStack API only. It is mainly used to test functionality (via Tempest) and stability of the API under large amounts of load. Rally does contain a resource to boot an instance and run Linux CLI commands via user data. This was tested as a way to provide the staging of instances to run performance software. However, starting the software on each instance at the same time and collecting the results from said software was difficult and not viable. Because of this, we deemed Rally was not suitable for our needs. The next tool we looked at was KloudBuster. KloudBuster is a tool that does performance testing inside an OpenStack instance. At the time of writing it provides two sets of tests: HTTP and storage. The HTTP test uses a traffic generator to measure requests per second and latency between instances. The storage test uses FIO to measure read/write IOPs and bandwidth. KloudBuster does what we were looking for, measuring ## Target and Elasticsearch: Maintaining an ELK stack over Peak Season DevFeed: [Target and Elasticsearch: Maintaining an ELK stack over Peak Season]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-05-25T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [elasticsearch](), [Logging](), [Cloud](), [Monitoring](), [dashboards](), [Scalability](), [Kafka]() Tags: [apache](), [apache-kafka](), [cloud](), [clusters](), [consul](), [contribute](), [dashboards](), [elasticsearch](), [elk](), [hashicorp](), [infrastructure](), [kafka](), [logging](), [logging-and-metrics](), [logs](), [make](), [metrics](), [monitoring](), [open](), [open-source](), [scalability]() ### AI overview This article describes how Target operated and evolved an Elasticsearch-based ELK stack for large-scale log aggregation, search, analytics, and multi-tenant logging during peak production periods. It discusses cloud scalability, operational challenges, and the open-source tools used alongside Elasticsearch. ### Source excerpt One of the strongest benefits of launching an application into the cloud is the pure on-demand scalability that it provides. I've had the privilege of working with the ELK stack (Elasticsearch, Logstash, Kibana) for purposes of log aggregation for the past two years. When we started at that time, we were pleased with our performance on search and query times with 10's of gigabytes of data in the cluster in production. When Peak time hit, we reveled as our production clusters successfully managed half a terabyte of data(!). During peak, Target hosted 14 Elasticsearch clusters in the cloud containing more than 83 billion documents across nearly 100 terabytes in production environments alone. Consumers of these logs are able to get access to queries in blazing fast times with excellent reliability. It wasn't always that way though, and our team learned much about Elasticsearch in the process. What's The Use Case At Target? In a word, "vast." The many teams that use our platform for log aggregation and search are often times looking for a variety of things. Simple Search This one is easy, and the least resource intensive. Simply doing a match query and searching for fields within our data. Metrics / Analytics This one can be harder to accommodate at times, but some teams use our Elasticsearch clusters for near-realtime monitoring and Analytics using Kibana dashboards. Multi-tenant Logging Not necessarily consumer facing, but an interesting use for Elasticsearch is that we can aggregate many teams and applications into one cluster. In essence, this saves money over individual applications paying for infrastructure to log themselves. Simple search is the least of our concerns here. Queries add marginal load on the cluster, but often they are one-offs or otherwise infrequently used. However, the largest challenge faced here is multi-tenant demand. Different teams have very different needs for logging/metrics; designing a robust and reliable 'one-size-fits-all' platform is ## Team Qualities Behind a Digital Observability Team's Peak-Season Upgrade DevFeed: [Team Qualities Behind a Digital Observability Team's Peak-Season Upgrade]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-05-23T05:00:00Z Content type: opinion Language: en Sources: [Target]() Topics: [observability](), [Logging](), [upgrade]() Tags: [agile](), [culture](), [devops](), [elastic](), [leadership](), [logging](), [observability](), [on-call](), [organizational](), [quality](), [support](), [team](), [upgrade]() ### AI overview A reflection on a digital observability team after a major Elastic logging-cluster upgrade ahead of Black Friday. The author identifies leadership support, mutual respect, and cooperation during on-call work as qualities that contributed to the team's accomplishments. ### Source excerpt It is 7:30 AM on a Monday morning in late October. I am waiting in line at Cafe Donuts to bring my team breakfast for our mandated 'no work for one hour'. We just wrapped up a strenuous week of implementing a major upgrade to our Elastic logging cluster. Many digital teams are relying on this upgrade to position themselves to confidently monitor their application health during the most important day in retail - Black Friday. It was a successful, much anticipated upgrade that resulted in many hours of overtime, late night calls, and cross-team performance tests. Morale is high, but it hasn't always been. It has been over three months since the team disassembled to work on new initiatives. I have since had a chance to reflect on the qualities of the team that made it so unique. I can't speak for my colleagues, but the accomplishments we made over six months are my proudest achievements in my five year tenure at Target. Leadership support Our leaders truly supported and enabled our work to be successful. They gave us the resources and confidence to confront organizational blockers and competing priorities. They helped us celebrate wins by providing recognition through updates to high level executives and advertising our achievements in forums. They took risks for us, trusted us to make decisions, and went to bat for us even when there was doubt that we could deliver on our promises. Respect This quality is near and dear to my heart. I never witnessed competition, resentment, distrust, or judgement among the team. Everyone truly respected one another and their unique perspectives. We were all aligned, working towards the same goals, and were in it together. Lead engineers partnered with junior engineers to ensure they understood the entire tech stack and could independently support our customers. I personally felt like my opinions and ideas mattered. On call schedules were set, but we supported one another through escalations and incidents. As a primary on call member, ## Hadoop Rolling Upgrades DevFeed: [Hadoop Rolling Upgrades]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-05-22T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Hadoop](), [big-data](), [Provisioning](), [Orchestration](), [Cloud](), [Apache Spark]() Tags: [big-data](), [cloud](), [hadoop](), [infrastructure](), [orchestration](), [provisioning](), [spark](), [upgrades]() ### AI overview This article describes Target's move from outage-based Hadoop upgrades to rolling maintenance. It covers the upgrade from Hadoop 2.7.1 to 2.7.3, the goal of avoiding planned downtime, and the use of cluster administration and REDStack provisioning tools to test and orchestrate the process. ### Source excerpt Hadoop upgrades over the last few years meant long outages where the Big Data platform team would shutdown the cluster, perform the upgrade, start services and then complete validation before notifying users it was ok to resume activity. This approach is a typical pattern for major upgrades even outside Target and reduces the complexity and risks associated with the upgrade. While this worked great for the platform team, it was not ideal for the hundreds of users and thousands of jobs that were dependent on the platform. That is why we decided to shake things up and go all in for rolling maintenance. Cluster Details: Hadoop (Core) 2.7.1 to 2.7.3 Mixed cluster workload running Hive, Tez, MR, Spark, Pig, HBase, Storm Goal Our March 2016 Hadoop upgrade was the turning point for rolling maintenance. With a large outage for the upgrade and monthly maintenance windows leading up to it, we decided to challenge ourselves with rolling maintenance to reach our uptime goals for core components. This would allow the platform team to deploy changes faster, reduce maintenance risk by not bundling changes together every month and more importantly not impact users with planned downtime. Drawing the line in the sand for rolling maintenance meant that it was time to get to work on our upgrade strategy. Reviewing the Playbook We started reviewing the upgrade process in November 2016 with the goal to upgrade our first admin cluster in December. The short turnaround time meant that we would leverage our cluster administration tool to handle the upgrade orchestration. The focus shifted towards understanding the order of events and evaluating the process for potential impacts. Running through the upgrades required a quick way to iterate through Hadoop cluster deployments and then tearing them down to retest. This is where REDStack, which is our internal cloud provisioning tool to build out a secure Hadoop cluster with production like configurations, came into play. After going through the ## How (and Why) We Moved to Spinnaker DevFeed: [How (and Why) We Moved to Spinnaker]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-04-07T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [CI/CD](), [Cloud](), [Jenkins](), [Provisioning](), [Terraform](), [Microservices](), [Apache Cassandra](), [Persistence](), [Cache](), [Git]() Tags: [automation](), [cassandra](), [ci-cd](), [cloud](), [deployment](), [engineering](), [git](), [jenkins](), [microservices](), [redis](), [spinnaker](), [terraform]() ### AI overview Target describes moving portions of target.com from a homegrown cloud deployment platform to Spinnaker after evaluating CI/CD platforms. The team selected Spinnaker for its Jenkins integration, multi-cloud deployment support, and use of immutable images, then adapted its architecture with Terraform, cloud storage, and managed caching to support resilient, highly available operation. ### Source excerpt Background Just after the middle of last year, Target expanded beyond its on-prem infrastructure and began deploying portions of target.com to the cloud. The deployment platform was homegrown (codename Houston), and was backed wholly by our public cloud provider. While in some aspects that platform was on par with other prominent continuous deployment offerings, the actual method of deploying code was cumbersome and not adherent to cloud best practices. These shortcomings led to a brief internal evaluation of various CI/CD platforms, which in turn led us to Spinnaker. We chose Spinnaker because it integrates with CI tools we already use at scale (Jenkins), supports deploying to all major public cloud providers, and compels software deployment best practices - all deployments are performed via immutable images, a snapshot of config + code. Supporting a Platform The primary goal of Target's cloud platform is to enable product teams to deploy and manage their applications across multiple cloud providers. We provide CI/CD, monitoring, and service discovery as services, and any application deployed via our platform gets those capabilities via a base image that is pre-configured for connectivity to each service's respective endpoint. Since these components are essentially products we provide to internal customers, we had to ensure the new CD platform was operationally supportable and highly-available. So, as soon as we decided on Spinnaker, a handful of engineers from the Cloud Platform group set about making this happen. Default Spinnaker scripts make it easy to standup a single self-contained server with the microservices and persistence layer all together, but that wasn't conducive to doing blue-green deployments - allowing updates of Spinnaker without downtime to our internal customers. We built jobs for building packages based off the master branch of each Spinnaker component's upstream git repository, and wrote Terraform plans to manage the deployment of each stack. ## Distributed Troubleshooting DevFeed: [Distributed Troubleshooting]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-04-05T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [incident](), [SIEM, Security, Observability](), [systems](), [elasticsearch](), [Open Source](), [telemetry]() Tags: [big-data](), [data](), [distributed](), [elasticsearch](), [elk](), [incident](), [infrastructure](), [kibana](), [logs](), [logstash](), [metrics](), [open-source](), [open-source-components](), [troubleshooting]() ### AI overview The article describes Target's Distributed Troubleshooting Platform for investigating issues across a large open source big data platform. It aggregates service logs and metrics so engineers can inspect information from many machines and services in one place, using open-source components and the Elasticsearch, Logstash, and Kibana stack. ### Source excerpt Target's open source big data platform contains a vast array of clustered technologies or ecosystems working together. Troubleshooting an issue within a single ecosystem is a difficult task let alone an issue that spans several ecosystems. It is impractical for a single human to individually investigate ecosystems one at a time for potential problems. The house will burn to the ground long before an engineer can find the cause of an issue and resolve it without quick access to aggregated system metrics and logs. The Solution How to identify, troubleshoot and resolve a distributed issue? Fight fire with fire of course! Big data issues must be solved with big data solutions. At Target, we are constantly expanding our Distributed Troubleshooting Platform to encapsulate every log and metric from every service in every ecosystem of our big data platform. Aggregating this data into a single troubleshooting platform enables an engineer to view error logs and system metrics across hundreds of machines and services with a single click. A troubleshooting platform like the one described above is not a new idea. Systems like Splunk have been doing it for years. Splunk however, has restrictions on the amount of data that can be ingested without an enterprise license. The larger we scale; the more money we pay for systems like Splunk. We created our Distributed Troubleshooting Platform from open-source components and without enterprise licenses. This allows us to utilize it on every server in the big data platform without worrying about the volume of data it is processing and re-negotiating enterprise licenses. It becomes a given, not a variable. Our Distributed Troubleshooting Platform is similar to the black box recorder on an aircraft. A majority of the time, the contents are never viewed. When the plane crashes however, the contents of the black box are the only way to reconstruct what happened and learn from the incident. Running a big data platform without enterprise licens ## Target open-sources Winnaker to audit Spinnaker deployments DevFeed: [Target open-sources Winnaker to audit Spinnaker deployments]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-02-13T06:00:00Z Content type: release Language: en Sources: [Target]() Topics: [Cloud](), [Tool](), [Continuous Delivery (CD)](), [Automation](), [Open Source](), [Deployment Strategies](), [Monitoring](), [Kubernetes]() Tags: [api](), [audit](), [automation](), [cloud](), [deployment](), [deployment-strategies](), [health-checks](), [identity-management](), [infrastructure](), [kubernetes](), [monitoring](), [open-source](), [platform](), [spinnaker](), [winnaker]() ### AI overview Target announces Winnaker, an open-source tool for auditing Spinnaker deployments from an end-user perspective. The article explains how Winnaker automates whole-system deployment testing and helps identify connectivity, API rate-limit, and infrastructure-configuration problems. ### Source excerpt Win the cloud with Winnaker! I am happy to announce that we, at Target, decided to open source a tool called Winnaker. This tool will allow the user to audit Spinnaker from an end user point of view. But first what is Spinnaker? The first time I heard the word Spinnaker, my reaction was, "wait, what does that even mean in English?" Shortly after, I found myself implementing a demo of Spinnaker as a potential replacement for our internal cloud deployment tool. Spinnaker is a cloud agnostic continuous delivery tool, which means we can push our code to any cloud provider we like. In fact, Spinnaker takes agnosticism to the next level by introducing three abstractions. Load balancers Server groups Security groups By enforcing this level of simplicity, it allows the implementation of deployment strategies such as Highlander, Red/Black on a vast different type of infrastructure (VM, Container, Kubernetes, public cloud, private cloud) with a high level of confidence and an incredible level of ease of use for the app developers. Spinnaker also roots for the immutable infrastructure design pattern. Baking your image once and deploying the image everywhere is another bold move that differentiates Spinnaker from the other tools. Why Winnaker ? Short answer is because of automation! Test the functionality of the CD system as a whole. Spinnaker has different components (CloudDriver, Rosco, Deck,...). Each of these components have their own unit tests and health checks that can be monitored. We learned the hard way that relying only on component health checks is not effective enough to ensure developers won't face any error when they deploy their apps. A few things can go wrong when off monitoring radar: Connectivity between the separate components Maxing out cloud provider API rate limit Base infrastructure configurations (subnet address space, identity management roles) So we decided to audit Spinnaker and cloud's whole functionality with a sample app. If baking and deploying our ## How Target Performance Tunes Machine Learning Applications DevFeed: [How Target Performance Tunes Machine Learning Applications]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-09-29T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Machine learning](), [distributed-systems](), [Apache Spark](), [Hadoop](), [Scalability](), [Filesystems](), [datasets](), [Algorithms](), [Transactions](), [Open Source]() Tags: [algorithms](), [apache-spark](), [data](), [data-science-and-engineering](), [distributed](), [engineering](), [hadoop](), [machine-learning](), [open-source](), [scalability](), [science]() ### AI overview The article discusses improving the performance and relevance of machine-learning applications by evaluating matrix multiplication at scale. It describes distributed computing requirements, distributed file storage, scalability, fault tolerance, and the use of Apache Hadoop, Apache Spark, and Apache Mahout. ### Source excerpt At Target we aim to make shopping more fun and relevant for our guests through extensive use of data - and believe me, we have lots of data! Tens of millions of guests and hundreds of thousands of items lead to billions of transactions and interactions. We regularly employ a number of different machine learning techniques on such large datasets for dozens of algorithms. We are constantly looking for ways to improve speed and relevance of our algorithms and one such quest brought us to carefully evaluate matrix multiplications at scale - since that forms the bedrock for most algorithms. If we make matrix multiplication more efficient, we can speed up most of our algorithms! Before we dig in, let me describe some properties of the landscape we will be working in. First, what do I mean by large scale? A large scale application, at a minimum, will require its computation to be spread over multiple nodes of a distributed computing environment to finish in a reasonable amount of time. These calculations will use existing data that are stored on a distributed file system that provides high-throughput access from the computing environment. Scalability, in terms of storage and compute, should grow as we add to these resources. As the system grows larger and more complex, failures will become more commonplace. Thus, software should be fault-tolerant. Fortunately, there is a lot of existing open-source software that we can leverage to work in such an environment, particularly Apache Hadoop for storing and interacting with our data, Apache Spark as the compute engine, and both Apache Spark and Apache Mahout for applying and building distributed machine learning algorithms. There are many other tools that we can add to the mix as well, but for the purposes of this post we will limit our discussion to these three. With that out of the way, lets dig in! Don't Forget the Basics Begin with good old paper and pencil. Yeah, I know this is about large scale matrix operations that you c ## What Data Science Is and How It Supports Personalized Experiences DevFeed: [What Data Science Is and How It Supports Personalized Experiences]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-03-01T06:00:00Z Content type: opinion Language: en Sources: [Target]() Topics: [Data Science](), [Machine learning](), [Algorithms, Complexity](), [High-Performance Computing]() Tags: [algorithms](), [data](), [data-science](), [data-science-and-engineering](), [engineering](), [high-performance-computing](), [machine-learning](), [science]() ### AI overview The article explains data science through everyday decision-making, describing how experience and instruction help people build models that guide judgment and anticipate outcomes. It connects these ideas to statistical models, machine learning algorithms, and Target's goal of creating personalized guest experiences. ### Source excerpt On my first encounter with it, around early 2010's, I was mystified. It sounded like witchcraft and I imagined the practitioners to be a coven of witches and wizards, all holding Ph.D.s in the dark art of "Data Science" and being respectfully addressed as "Data Scientists". It was believed they would magically transform haystacks into gold and then ask for your first-born in return as a reward for their service (a la Rumpelstiltskin) There is no denying the fact that the title "Data Scientist" is the most coveted one these days and has a nice ring to it. It's also true that data science has traditionally been a monopoly of mathematicians and statisticians. Obviously, developing statistical models and machine learning algorithms requires years of training and practice to specialize. In my opinion it is more of an art form driven by science and can easily be mistaken for magic. It's common knowledge that the more experienced in life we get, the easier it is for us to make up our mind. For instance, "What diner to pick for a boy's-night-out?", "When to stay off highways to avoid being stuck in a traffic-jam?", "When to buy a house? When NOT to buy?", are all such decions we make everyday. This ability comes as result of years of learning from implicit experience (a.k.a unsupervised learning) and explicit instructions from parents, teachers, friends, family and media (a.k.a supervised learning.) Our brain builds models of the world, of the situations we have been in, of banal and extraordinary, of nice and not-so-nice, of appropriate and inappropriate etc. These models facilitate judgment, govern behavior and enable anticipation of likely outcomes. That's basically data science. The recent progress in large scale and high performance computing has opened doors for such complex calculations to be performed on-demand and much more efficiently than was possible before. Hence, the buzz! At Target we operate in a guest-centric universe. We don't treat our guests as a statist ## Bare Metal Big Data Builds DevFeed: [Bare Metal Big Data Builds]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-01-13T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Hadoop](), [Automation](), [openstack](), [Nova](), [Jenkins](), [Bash](), [Python](), [API]() Tags: [ambari](), [api](), [automation](), [bare-metal](), [bash](), [big-data](), [chef](), [hadoop](), [infrastructure](), [ironic](), [jenkins](), [nova](), [openstack](), [python]() ### AI overview This engineering article describes Target's transition from manually building on-premise Hadoop clusters to an automated bare-metal provisioning workflow. It combines OpenStack Ironic and the Nova client with user-data bash scripts and Chef cookbooks to bootstrap servers, configure node roles, and add them to clusters. ### Source excerpt When you first think about scaling an on-premise Hadoop cluster your mind jumps to the process and the teams involved in building the servers, the time needed for configuring them and then the stability required while getting them into the cluster. Here at Target that process used to be measured in months. The story below outlines our journey around scaling our Hadoop cluster, taking the months to hours and adding hundreds of servers in a couple weeks. The Need Early 2013 taught us the lesson that manually managing Hadoop clusters, no matter how small is a time consuming and very repetitive task. Our next cluster build in 2014 drove the adoption of Chef, Artifactory and Jenkins to help with cluster operations. We stood up those components and created new role cookbooks to manage everything on the OS (configurations, storage, Kerberos, MySQL, etc.). While this was a step in the right direction, it left us with a manual process to still create the initial base server build and then add it to the cluster after configuring it with Chef. Build Foundation Closing the gap in our automation meant finding a way to deliver on true end to end builds, from an initial bootstrap to running jobs in your cluster. OpenStack's Ironic project was the first piece of the puzzle. Ironic gives us the ability to provision bare metal servers, similar to how OpenStack automated the VM build process. With Ironic as the foundation, we leveraged the Nova client to manage our instance builds. The nova python client interacts with the Compute service's API, giving us an easy way to specify our build parameters and spinning up an instance on one of our physical servers. The other key piece with the nova client is the ability to send boot information to the server using user data. The bash script sent executes several commands to install the Chef client, setup public keys and run the initial knife bootstrap to set the run list for the build. Example nova boot command: nova boot --image $image_name ## Real-time Big Data at Target DevFeed: [Real-time Big Data at Target]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-11-11T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Hadoop](), [stream-processing](), [streaming-data-processing](), [Kafka](), [Latency](), [Low Latency](), [Open Source]() Tags: [analytics](), [apache-kafka](), [batch](), [big-data](), [data](), [hadoop](), [kafka](), [latency](), [low-latency](), [open-source](), [real-time](), [storm](), [stream-processing](), [streaming]() ### AI overview Target's Big Data platform team describes its effort to move data into Hadoop in real time using flexible open source components. The article outlines requirements for resilience, message-format support, usability, and low latency, then evaluates Apache Storm and identifies recovery and batching-related latency issues in testing. ### Source excerpt An enterprise as large as Target generates a lot of data and on my Big Data platform team we want to make it as easy as possible for our users to get it into Hadoop in real-time. I want to discuss how we are starting to approach this problem, what we've done so far, and what is still to come. Our requirements We wanted to build a system with flexible open source components. Our experience with proprietary products on Hadoop is that they tend to be inflexible and only work with a narrow set of use cases. That can also be true of open source products, but we have found them to be easier to adapt to our needs. Indeed, that ended up being the case as we went through this journey and made several contributions to Apache projects. More specifically, we wanted to create a system that: was highly resilient to the failure of any individual component supported a variety of message formats delivered data to Hadoop with very low latency made the data immedaiately usable streamed data from Apache Kafka sources A Streaming Framework There are many excellent comparisons of streaming frameworks available, and I won't attempt to recreate them here. The first criteria we considered was what tooling was needed to monitor and administer the streaming framework, with a strong preference to use our primary Hadoop administration tool, Apache Ambari. Apache Storm fit that bill and was also a proven solution for stream processing. If Storm could meet our other requirements, it would be our first choice. To test its resiliency we ran a simple scenario: start a data stream into Hadoop, disable HDFS, and then reenable it. Streaming would obviously fail while HDFS was disabled, but we needed the system to recover gracefully when HDFS came back online. Unfortunately our first test of this scenario left our Storm topology in an unrecoverable state, which required a manual restart. That's not something we could live with. We also needed very fine control over the latency of arriving data. In gener ## Target's Dojo Expands Access to DevOps Expertise and Training DevFeed: [Target's Dojo Expands Access to DevOps Expertise and Training]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-08-14T05:00:00Z Content type: opinion Language: en Sources: [Target]() Topics: [Development](), [DevOps](), [Automation](), [Learning](), [Kafka](), [openstack]() Tags: [agile](), [automation](), [developers](), [devops](), [kafka](), [learning](), [openstack](), [scrum]() ### AI overview Target describes a learning "Dojo" that brings subject matter experts together with project and product teams to expand DevOps skills and make training easier to scale. The dedicated space can support up to 8 or 9 teams at a time. ### Source excerpt At Target we're always looking for ways to move forward in becoming the best omni-channel retailer that we can be. This journey demands that we enable change in most every part of our technology organization. Our culture, delivery model, technology selections, working arrangements and org structures are all levers we can pull to help us be more responsive. A key question that we continually ask ourselves is "how can we move faster"? Introducing change in a large enterprise can take a long time if we don't challenge ourselves to be creative around constraints (like not enough "experts" to go around). Traditional learning approaches might not be fast enough, so we've started acting on some innovative ideas to break down those barriers. While this progress has been great, we've continually been asking ourselves how we could go faster. ###Building Capacity One of the newest things we doing is expanding our capacity for others to learn the new ways of doing things. Since we still have relatively few subject matter experts to go around, we are finding that access to these experts is a constraint that makes it difficult for us to scale the learning process for new teams. To address this constraint, we are standing up a learning "Dojo." The Miriam-Webster dictionary defines a Dojo as "a school for training in various arts of self-defense (as judo or karate)." At ChefConf 2015, Adam Jacobs delivered a very interesting talk titled Chef Style DevOps Kungfu. This presentation talked about building your practice through repetition and development of skills. We needed a place to develop our own special DevOps Kungfu. Cue the Dojo! The Dojo is a dedicated space in which our subject matter experts take up residence for an extended time. This is the home base for automation engineers, advanced Scrum leaders, OpenStack engineers, Chef experts, Kafka engineers, etc. Project teams and product teams looking to leverage these experts to build their own expertise can colocate their teams ## Modernizing our IT Environment at Target DevFeed: [Modernizing our IT Environment at Target]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-04-15T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [systems](), [real-time](), [data](), [Testing]() Tags: [architecture](), [data](), [devops](), [distributed](), [integration](), [legacy](), [modernization](), [real-time](), [teams](), [technology](), [testing]() ### AI overview The article describes Target's effort to modernize its IT environment, using a project involving near-real-time data from multiple systems as an example. It identifies integration with legacy systems, distributed-environment setup, stale source data, lengthy manual testing, and coordination across teams as modernization challenges. ### Source excerpt In 2014, I worked on a project to build out a system to provide new business capabilities for our Marketing and Digital spaces. This initiative was not unlike other initiatives we undertake at Target, but it provides a great example of the importance of us investing in an effort to modernize the way we operate, the technology stacks we use and the way we think about systems and systems architecture. The project took in data in near real-time from multiple systems, processed the data and exposed it to consumers. The closer we came to real-time data the larger the benefit to our guests. There was only one problem - we had to keep one of our current systems running in parallel prior to switching on the new capabilities. In this scenario, there were a couple of key frustrations I experienced which are symbolic of what is driving our need to modernize: 75% of the time spent on the project was integration with systems which would be shut off shortly after we went live. The estimate for integration work with legacy systems was enough work to keep 34 full-time workers busy for an entire year. The majority of the processing for this system was in a distributed environment, standing up the environment took over 20 weeks. The data which would be exposed through this system would be 24 hours old due to limitations of the providing systems. Every time we went through a release, we would initiate a fully manual testing process which spanned six weeks. Decision making for this effort spanned over eight teams and four different organizations. From what I have seen, Target is not that different from the majority of large enterprises which have been in business for half a century. We have grown over time, transitioning from one technology paradigm to the next, adding more and more technology debt into our environment as we move quickly to provide business value to our internal business partners and our guests. I assume we are faced with the same pressures as other large enterprises: ## Robotics Analytics @ Target DevFeed: [Robotics Analytics @ Target]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-04-09T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Robotics](), [data](), [Automation](), [systems](), [FIRST Robotics Competition]() Tags: [analytics](), [automation](), [big-data](), [data](), [distribution](), [machine](), [network](), [robotics](), [servers](), [supply-chain](), [visualization]() ### AI overview The article describes robotics and automation in Target distribution centers, where automated systems generate machine data. It explains that engineers faced a cumbersome manual process involving spreadsheets, macros, distributed files, and difficult server access to gather and visualize that data. ### Source excerpt One important part of a retailer is their supply chain network. In order to sell a product in our stores, or online at Target.com, we need to have a well run and maintained network to move all of those products from one place to another. It may not sound like a complex problem to ship products from a distribution center to a store, but as soon as you start to have multiple vendors with multiple stores (not to mention online orders that go directly to a guest's home) it becomes increasingly complex. Retailers get products from companies, referred to as vendors, and then distribute the products to stores through central locations, referred to as distribution centers (DCs). In order to improve throughput of our supply chain network we expand our distribution network by building new distribution centers or we increasing the efficiency of the current network. A great way to improve performance of a DC is to put robots and other automation equipment in it. High tech equipment like this requires high skilled labor to maintain and manage the equipment. These highly automated systems run on servers and other control equipment and they produce a lot of machine data that can lead to valuable insights. Robotics has always been an interest of mine, starting with my experience in high school with the FIRST Robotics Competition and continuing today as I volunteer with FIRST. When I first joined Target right out of school with a degree in Electrical Engineering my curiosity and interest in robotics led me to get to know some of the engineers who worked on the robotics systems within Target's DCs. I mentioned to them that I was working with a tool called Splunk that did a good job of aggregating data from different sources to a central location with a visualization platform built in. The engineers said they had a very cumbersome, inconsistent manual process which involved multiple spreadsheets and home grown macros to gather and visualize data. The data was spread out through multip ## Target DevOps Days In Pictures DevFeed: [Target DevOps Days In Pictures]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-02-26T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [DevOps](), [engineering-culture](), [Monitoring](), [Open Source]() Tags: [devops](), [engineering](), [engineering-culture](), [gallery](), [images](), [monitoring](), [open-source](), [photos](), [sample-post](), [test]() ### AI overview A photo-based recap of Target DevOps Days, featuring speakers, attendance, engineering culture, monitoring, and open source discussions. ### Source excerpt Setting the stage Celebrity guests in the house Jez Humble opens the day with a bang.The best in the business. Ducy opens with a goat. To the shock of many, the rest of the talk was goat free. Deming's obligatory appearance Record attendance Heather Mickman and Ross Clanton opening and closing the day! Greg Larson delivers an excellent talk on building a strong engineering culture. Brent Nelson, the man behind Target DevOps Days, talks tech clubbing. Matt Helgen talks monitoring and open source Target DevOps Days In Pictures was originally published by Target Brands, Inc at target tech on February 26, 2015. ## Outage Resolution Through Automation DevFeed: [Outage Resolution Through Automation]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2014-12-29T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [dashboards](), [Monitoring](), [Automation](), [GitHub](), [Jenkins](), [Ruby](), [Agile](), [data](), [Script]() Tags: [agile](), [automation](), [code](), [dashboards](), [devops](), [flashbuilds](), [github](), [jenkins](), [metrics](), [monitoring](), [outage](), [ruby](), [server](), [support]() ### AI overview The article describes Project Argus, a monitoring-as-code initiative that used GitHub, Chef, Kitchen, Jenkins, and Ruby to build dashboards, automate metric processing, and alert support teams about missing data. It also introduces an outage in the monitoring data-processing servers that caused the dashboards to go blank. ### Source excerpt Recently we launched Project Argus, a 30-day "monitoring challenge" to improve visibility of key performance indicators (KPIs) across our technology stack prior to our peak retail season (in Greek Mythology, Argus is a 100-eyed giant). This effort was structured as a mix between a FlashBuild and agile. We used two day sprints, twice a day stand-ups, and feature tracking through Kanban boards. Quickly in this effort we decided to build our product as monitoring-as-code. Through the use of tools such as GitHub, Chef, Kitchen, Jenkins, and Ruby, we were able to quickly build several monitoring and dashboard solutions for use within our Technology Operations Center. These dashboards and the iterative process we use to continue delivering more content have been embraced by our support teams who now heavily rely on them to proactively detect and resolve issues. One deliverable from Argus includes a cookbook per core business function that represents the KPIs identified by each business product owner. When run, the cookbooks establish connections to our data sources, build the scripts that process our data, and create crontab entries to automatically run the scripts at our predefined intervals. All of these resources and actions are defined in code. We designed our solution so that the dashboards are decoupled from our centralized processing and metric creation. Additionally, our cookbooks are written to be dashboard agnostic and independent so that anyone can make changes to the appearance and we can switch dashboard solutions easily. The processing of our metrics is centralized and managed on-site with each cookbook receiving its own server. Finally, our dashboard solution detects lapses in data and generates alerts that post into our persistent chat client. This allows us to close the loop on the health of the monitoring system, which can be a painful endeavor! ##The Problem Unfortunately, on 2014-12-06 at around 1:30pm the servers we use to process the monitoring data ## Target FlashBuilds DevFeed: [Target FlashBuilds]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2014-11-10T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [DevOps](), [Agile]() Tags: [agile](), [communication](), [devops](), [flashbuilds](), [process](), [scrum](), [team]() ### AI overview Target describes FlashBuilds, one-day sessions combining two agile sprints, needed participants, and facilitation to address difficult business and delivery problems. The approach aims to improve delivery speed, collaboration, communication, and organizational learning, although some sessions may not produce a finished product. ### Source excerpt At a couple of recent conferences around DevOps (MSP DevOps Days and DevOps Enterprise Summit), Heather Mickman and Ross Clanton from Target explained some high level concepts and actions we're taking to break our mold of working on complex problems. When presented with an opportunity to work differently and challenge our normal delivery of IT assets - from plan to decommission - what would that look like? First, we asked some questions: What can we do to address the barriers within our organization; can we remove that impact on our ability to get things done? What can we do to remove time spent moving from meeting to meeting, often recapping the same discussions that you had in your last meeting 2 weeks ago? What process barriers exist that can be included in the delivery and built as part of the 'service'? And then we asked ... What if we could alleviate all of these problems (and more) all at the same time, all while you build team morale and actually have fun (gasp!) building stuff? What if? Two simple words that can: spark amazing conversations; generate innovation and excitement; challenge the status quo; break through to show how DevOps philosophies break through - even with the big horses. A ton of this post is simply about getting stuff done (GSD) but also to get into some specifics around how we approached fixing a business problem in some new ways - at least for us. So let's embark on some early details around Target's journey to challenge the traditional approaches to get stuff done and take control of the 'how' ... to redefine our working philosophies, principles, and minimize the impact radius of our efforts while still going deep with the teams and persons engaged. Any endeavor such as this has to have a name ... We affectionately call this process FlashBuilds! What is a FlashBuild?! Glad you asked! FlashBuild: A one-day session that includes 2 agile sprints, all the smart people that you need to build stuff, a facilitator, and a whole lot of positive focus ## Target Joins the W3C DevFeed: [Target Joins the W3C]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2014-10-15T05:00:00Z Content type: release Language: en Sources: [Target]() Topics: [Web](), [Internet of things](), [Internet](), [coding-community]() Tags: [internet-of-things](), [location-based](), [open-source](), [standards](), [w3c](), [web](), [web-services]() ### AI overview Target announced its membership in the World Wide Web Consortium (W3C) and described plans to collaborate on web services, location-based content distribution, and the Internet of Things for retail-related technology. ### Source excerpt Tuesday, October 14th 2015, Target affirmed its commitment to technology and leadership in the retail space by joining the World Wide Web Consortium (W3C). As a company that cares about delivering great experiences for customers, we realize those experiences are now steeped in technology. One way Target can have a significant impact on shaping technology is open engagement with other companies and organizations at the heart of emerging development - the W3C enables us to do just that. So you joined - now what? There are so many areas of technology important to retail and associated businesses. Initially, we're interested in engaging with tech leaders, businesses, and organizations on web services, location-based content distribution (iBeacons, smart stores, etc.), and the Internet of Things. Through open dialogue and collaborative work in the W3C, we seek to push these areas of technology forward to enable new and better experiences for our customers. More to come soon! This is just the beginning. As an organization we will be engaging with the developer community in new and different ways, proposing ideas on the technologies that matter to us, and providing feedback of our progress. Now let's get to work! Target Joins the W3C was originally published by Target Brands, Inc at target tech on October 15, 2014.