# openstack Published articles for openstack. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Community Update - Week 25, 2026 DevFeed: [Community Update - Week 25, 2026]() Original publisher: [Read original article]() Author: jnsamyak Published: 2026-06-23T10:12:46Z Content type: news Language: en Sources: [Fedora Community Blog]() Topics: [Fedora](), [release engineering](), [Cloud](), [openstack](), [qemu](), [Amazon Web Services]() Tags: [2026](), [aws](), [ci](), [cloud](), [community-update](), [fedora-project-community](), [infrastructure](), [maintainers](), [migration](), [openstack](), [qemu](), [release-engineering](), [report](), [test]() ### AI overview A Fedora CLE Team report covering infrastructure and CentOS operations during 15-19 June 2026, including service moves, operating-system upgrades, repository security work, migrations, load-balancer changes, release engineering, and quality initiatives. ### Source excerpt This is a report created by CLE Team, which is a team containing community members working in various Fedora groups for example Infrastructure, Release Engineering, Quality etc. This team is also moving forward some initiatives inside Fedora project. Week: 15 - 19 June 2026 (Flock Week!!) Fedora Infrastructure This team is taking care of day [...] The post Community Update - Week 25, 2026 appeared first on Fedora Community Blog. ## Posts from the Past, October 2025 DevFeed: [Posts from the Past, October 2025]() Original publisher: [Read original article]() Author: Scott Lowe Published: 2025-10-22T18:00:00Z Content type: article Language: en Sources: [Scott's Weblog]() Topics: [Amazon Elastic Kubernetes Service](), [Kubernetes](), [Amazon Web Services](), [Infrastructure as code](), [Azure]() Tags: [2025](), [amazon-elastic-kubernetes-service](), [ansible](), [aws](), [azure](), [cilium](), [cli](), [cloud](), [cni](), [containers](), [cri-o](), [devops](), [docker](), [go](), [iac](), [japan](), [k8s](), [kubernetes](), [kubernetes-clusters](), [linux](), [networking](), [oci](), [openstack](), [security](), [vagrant]() ### AI overview A retrospective roundup revisiting the author's Kubernetes, cloud infrastructure, and container-related posts published in October across several years. It highlights work involving Pulumi, Amazon EKS, Bottlerocket OS, Azure Kubernetes Service, Cluster API, AWS, kubeadm, Ansible, Vagrant, and OpenStack. ### Source excerpt Every now and then, I publish one of these "Posts from the Past" articles that looks back on content I've created and posted over the life of this site. This year marks 20 years of content--I can hardly believe it! Don't worry, though; you won't have to go through 20 years of past posts. Here is a selection of posts from mid- to late October over the last decade or so. I hope you find something useful, informative, or at least entertaining! October 2024 Last year I shared information on how to use Pulumi to stand up an Amazon Elastic Kubernetes Service (EKS) cluster with Bottlerocket OS on the Kubernetes nodes--without using any higher-level Pulumi components. October 2022 In 2022, after getting irritated with what I felt was a poor user experience when accessing Azure Kubernetes Service (AKS) clusters created with Pulumi, I published this post on how to change the Kubeconfig file for a more streamlined user experience. October 2021 Cluster API is the name of the game for multiple posts in October 2021. First I wrote this article on kustomize transformer configurations for Cluster API v1beta1 (so that you can use kustomize to manipulate Cluster API manifests), followed up later that month with an article on influencing Cluster API AMI selection. I also touched upon using the external (out of tree) cloud provider for AWS that month, a topic I am revisiting soon as I explore integrating Talos Linux with AWS. October 2020 More Cluster API content--this time discussing IaC considerations for Cluster API (think things like integrating workload clusters with existing AWS workloads or services). October 2019 In October 2019 I explored using jk to programmatically create Kubernetes manifests, and discussed how to use kustomize with kubeadm configuration files. October 2018 Plenty of articles discuss the use of kubeadm to bootstrap Kubernetes clusters (including a few I wrote!), but what of talking about using kubeadm to stand up an etcd cluster? I've got you covered! October 2 ## CentOS8 images published in Pouta as a tech preview DevFeed: [CentOS8 images published in Pouta as a tech preview]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2020-02-17T05:48:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Cloud]() Tags: [centos](), [centos8](), [cloud-data](), [diskimage](), [glance](), [image](), [openstack](), [pouta](), [preview](), [published](), [tech](), [ubuntu]() ### AI overview CentOS 8 images are available in Pouta as a tech preview. The article notes minor upstream issues, including a resolv.conf nameserver issue, and describes temporary image modifications to address one issue. ### Source excerpt There are now CentOS-8 images available in Pouta! There are some minor issues with the upstream CentOS8 images, so, for now, they are considered to be in "tech preview". We have solved the one we have found so far by temporarily modifying the image to use "cloud-user" and remove the resolv.conf leftovers. Basic information about our images can be found on docs.csc.fi One issue is that /etc/resolv.conf sometimes has a nameserver defined from the build of the image. There is an open CentOS bug report about this: https://bugs.centos.org/view.php?id=16948 ## Migrating cPouta Object Storage to the Allas Ceph Service DevFeed: [Migrating cPouta Object Storage to the Allas Ceph Service]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2019-12-27T11:23:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [migration](), [Data Management](), [data](), [cloud-infrastructure]() Tags: [ceph](), [data-management](), [migration](), [object-storage](), [openstack](), [radosgw](), [rgw](), [storage]() ### AI overview This article describes CSC's migration from the cPouta object storage service to Allas, a standalone Ceph-based object storage service. The plan aimed to move user data without requiring user changes or breaking existing data links, using a new Ceph cluster and new data pools. ### Source excerpt We have released our new Allas object storage service. We firmly believe it will play a growing role for data management, at CSC and for the whole academic field in Finland. Alas, the road to Allas was not completely without pain. CSC also hosts the cPouta IaaS service. It provided its own object storage service. Our goal with Allas was to build on this, and transform the object storage portion to a standalone service. This would raise the profile of the service, and make it easier to scale, both human resource wise and where it comes to the platform size. So, in brief, our aim was: Replace cPouta object storage with Allas Make sure all the user data from cPouta object storage moves over. Make sure cPouta object storage users don't have to do any changes. Don't break any existing links to data in cPouta object storage. How did we intend to do this, you may ask? Migration plan What were our options? We had a large bunch of new hardware, and a high level plan. For those of you who know Ceph, you might have already thought of how you'd do this. Add more hardware to the same storage pool, slap on an additional domain name to access the data, and call it a day. Create additional data pools, make sure radosgw uses old and new pools for data, slap on an additional domain name, and call it a day. Create a new Ceph cluster with new pools, migrate the data over, add the old and new domain names, and call it a day. Number 3. is definitely the most work, but we went with that. The main reason for that was that if we want to achieve all the benefits of separating Allas into its own service, we actually need to be a bit separate. For example, the old object storage pools were in the same cluster as the block storage pools used by our OpenStack virtual machines. This not only locked us to the current version of Ceph running there, but created a lot on interdependencies between the services. To make it easier, I'll use the terms rgw-old (cPouta object storage) and rgw-new (Allas) t ## cPouta Cloud Object Storage Service has been updated to run CEPH Luminous! DevFeed: [cPouta Cloud Object Storage Service has been updated to run CEPH Luminous!]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-02-06T10:03:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [Amazon S3](), [Cloud](), [configuration](), [Command-line interface]() Tags: [amazon-s3](), [api](), [automatic](), [ceph](), [cli](), [configuration](), [cpouta](), [format](), [object-storage](), [openstack](), [radosgw](), [s3](), [storage](), [upgrade](), [xml]() ### AI overview The CSC Pouta Cloud Team describes an upgrade of its Ceph Object Storage Gateway from Jewel to Luminous. The update enables AWS4 signatures by default and adds support for several S3 features, including bucket lifecycle, website hosting, and request payment. The article also begins an example of configuring an S3 lifecycle policy with s3cmd and XML. ### Source excerpt We (the CSC Pouta Cloud Team) has finally upgraded CEPH Object Storage Gateway from Jewel to Luminous! There may come more blog posts about other improvements to other of our Openstack services (cinder and glance) which use CEPH. No need to use s3 --signature-v2 anymore! Before this upgrade, anyone using the S3 protocol needed to use the "AWS2" for most actions to work. Now that we run Luminous it is no longer necessary and the default AWS4 signature is working. Yay! In general things should be faster, shinier and also more API calls are supported. Comparing http://docs.ceph.com/docs/luminous/radosgw/s3/ and http://docs.ceph.com/docs/jewel/radosgw/s3/ these features* have gone from unsupported to supported: Bucket Lifecycle Supported Removing expired files is supported Bucket Website Supported Bucket Request Payment Supported Which got me a bit excited! Now to learn a bit about these and see if they work on our setup and if possible if we can make them work without breaking anything for existing objects & users: Bucket Lifecycle This is about expiration/automatic removal of objects at a certain time or after a certain time has passed. Let's start with S3 Some more details about the policies from Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html https://docs.aws.amazon.com/AmazonS3/latest/dev/set-lifecycle-cli.html Telefonica open cloud has some nice documentation with s3cmd examples: https://support.telefonicaopencloud.com/en-us/ugs3cmd/obs/en-us_topic_0051518501.html I have not found anything better on docs.ceph.com -- Step 1. Create a lifecycle policy with s3cmd and in the XML format. (there is a JSON style with awscli and "aws s3api get-bucket-lifecycle-configuration --bucket mybucketname" ): Call the file expiry.xml and make the content something like this: ExampleRule expire_these_ Enabled 1 ) Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-12-31T06:50:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [mount](), [Filesystems](), [Linux](), [Cloud]() Tags: [cinder](), [cloud](), [cpouta](), [filesystem](), [fstab](), [image](), [linux](), [maintenance](), [mount](), [openstack](), [pouta](), [server](), [storage](), [systemd](), [volume](), [volumes]() ### AI overview This article explains how to mount block-device volumes in Linux instances and why relying on standard /etc/fstab entries can prevent an instance from booting when a volume is missing or differs after an image-based deployment. It recommends mount options such as nofail and discusses noauto. ### Source excerpt Some background: As part of the CSC Pouta Cloud Openstack installation we provide the cinder service. This allows our customers to attach block devices to their instances. A common way to illustrate what this means is to say: ".. it's like attaching a USB drive to the instance. " - Some Openstack trainer After attaching the block devices one has to format it with a filesystem and mount it. The usual way to mount block devices (in most Linux based distributions I've seen, if you know other ways please comment below!) every time the server reboots is to put something like this into /etc/fstab: /dev/vdb1 /mnt ext4 defaults 0 0 All clear so far I hope. It mounts block device /dev/vdb to /mnt as filesystem ext4 with the defaults mount options and the last two fields (numbers) are about dumping and file system checks. (Sometimes it's wise to use UUID or LABEL and not /dev/vdb1 in case you have more than two volumes.) But what happens if disk with is not there anymore? Well, with the above fstab your instance will not boot into the operating system. You'll be stuck having to perhaps enter single user mode or attaching a volume and then rebooting and modifying fstab. Single user mode is probably OK sometimes (it's inconvenient and takes time). But let's say you took a snapshot of an instance in Openstack with the glance service. This creates an image of the root disk. If you then spawn a new instance with that image as a root disk it will still have that /dev/vdb1 in /etc/fstab. Hey ho, the new instance booted from the image will not boot. Here you could of course just create a new volume and attach it. But if you used UUID or LABEL in fstab then it wouldn't find those and also break! So what is the best(tm) way? By using some nice fstab mount options. For example nofail UUID=1c7fc2f-f9d9-4e01-8d0f-190f05416831 /mnt ext4 nofail,defaults 0 0 There's another mount option called "noauto". The difference is put quite nicely in this stackoverflow question which has a piece from ## ePouta Flavour Halloween 2018 Communiqué DevFeed: [ePouta Flavour Halloween 2018 Communiqué]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-10-19T10:51:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [GPU](), [cpu](), [Nvidia]() Tags: [applications](), [cpu](), [deprecated](), [epouta](), [espoo](), [flavors](), [gpu](), [hpc](), [newton](), [numa](), [nvidia](), [openstack](), [performance](), [pike](), [pouta](), [standard](), [westmere]() ### AI overview The communiqué announces changes to ePouta flavors, including public availability of the gpu.2.1gpu flavor with an NVIDIA Tesla V100 and NUMA-aware CPU and memory behavior. It also announces the deprecation of hpc.*.westmere flavors and recommends replacement flavors for new instances. ### Source excerpt We are making some changes to our flavor offering . Here's a summary of the changes: We now have even more GPUs in ePouta. The gpu.2.1gpu flavor is now publicly available and each has one NVIDIA Tesla V100 via PCI pass through. This flavor was already added in July but could only be accessed after requesting it from servicedesk. Now we are making them publicly available. The gpu.2.1gpu flavor is our first NUMA aware flavor. This should speed up CPU & memory heavy applications. Our current version (Newton) of OpenStack is not able to guarantee PCI devices from the same NUMA node too, this is possible to come when we get to Pike or later. Deprecation of hpc.*.westmere flavors This does not affect running instances but in the future you might see changes in the performance and therefor we suggest that you resize your flavor to a newer flavor type. You won't be able to launch new instances with the deprecated flavors. Flavors that will be deprecated and suggested replacement: hpc.medium.westmere - standard.xlarge hpc.large.westmere - hpc.4.10core hpc.xlarge.westmere - hpc.4.10core or hpc.4.20core hpc.largemem.westmere - hpc.4.20core or hpc.3.28core More information about the different flavors can be found here: https://research.csc.fi/pouta-flavours No known Copyright Please acknowledge 'Sir George Grey Special Collections, Auckland Libraries, 1-W682' when re-using this image. ## Ubuntu 18.04 images published in cPouta and ePouta DevFeed: [Ubuntu 18.04 images published in cPouta and ePouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-10-17T07:30:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Ubuntu](), [Cloud](), [ssh](), [Command-line interface](), [releases]() Tags: [cli](), [cloud](), [cpouta](), [diskimage](), [ed25519](), [epouta](), [horizon](), [image](), [lts](), [openstack](), [release](), [ssh](), [ubuntu]() ### AI overview Ubuntu 18.04 LTS images are available in the cPouta and ePouta cloud environments. The article explains that the images use Canonical's unmodified image, the default SSH user is ubuntu, and users can upload other Ubuntu cloud images or configure users with cloud-init and the OpenStack CLI. ### Source excerpt Hello Ubuntu fan boys and girls! An image containing latest LTS Ubuntu release (18.04 Bionic Beaver) is now available in c and ePouta! This time we opted to use the image provided by Canonical without any modifications. This means the user used to ssh in is now "ubuntu" instead of "cloud-user". In any case, trying to ssh in as root will tell you which user to ssh in as In case you want to try even newer versions of Ubuntu, say the "in-between" releases like 18.10 Cosmic Cuttlefish - head over to http://cloud-images.ubuntu.com/ , select the release and fetch the "cloudimg-amd64.img" - for example "cosmic-server-cloudimg-amd64.img" - which is a QCOW2 image. This you can then upload as an image in your c/ePouta project. Our user guide has some pictures and instructions how to work with and upload new images: https://research.csc.fi/pouta-adding-images To add your ssh key under a new "cloud-user" (in case you don't want to use ubuntu user) you can do something like below. First create a file that has the cloud-init user data. Then use that file as input to the openstack CLI tool. $ cat cloud-init.cfg #cloud-config users: - default: - default_user: name: cloud-user gecos: Cloud User sudo: ["ALL=(ALL) NOPASSWD:ALL"] ssh_authorized_keys: - ssh-ed25519 AAAAMYKEYQ user@email.example.org And then the openstack CLI command : $ openstack server create ubuntu1804 --flavor standard.small --image Ubuntu-18.04 --key mykey --security-group default --security-group ssh-group --user-data cloud-init.cfg It is if course possible to use horizon to supply the user data as well. p.s. Yes, ed25519 keys works this way! The web interface openstack horizon still does not support them yet. ## ePouta adds new VM flavors and server hardware DevFeed: [ePouta adds new VM flavors and server hardware]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-07-03T09:23:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cloud-infrastructure](), [virtual machines](), [Hardware](), [cpu](), [Nvidia]() Tags: [availability](), [cpu](), [epouta](), [espoo](), [flavors](), [gpu](), [hardware](), [hpc](), [nvidia](), [openstack](), [pouta](), [virtual-machines]() ### AI overview CSC announces new VM flavors and server hardware for ePouta, including standard flavors, Skylake-based HPC flavors, a 735 GB big-memory flavor, and NVIDIA Tesla V100 GPU instances available by request. ### Source excerpt We are pleased to announce the immediate availability of new lovely VM flavors in ePouta backed by new server hardware! A summary of the changes: introduced standard-flavors to ePouta; these are the most popular flavors in cPouta. introduced a new generation of hpc-flavors which have Skylake CPUs. a new generation of big memory flavor with 700GB of RAM. GPUs in ePouta. The flavors have 1 NVIDIA Tesla V100. These are only available per request to CSC Service Desk. deprecating hpc.mini, hpc.small and all Haswell hpc-flavors except hpc.fullnode.haswell. This does not affect the io.haswell-flavors. As always the current description of flavors can be found on https://research.csc.fi/pouta-flavours New Flavors The new standard flavors are of the same configuration that already exists in cPouta. This flavor is really cheap but not suitable for heavy workloads since the CPU is oversubscribed. This means that many virtual machines have to share the same CPUs. These instances are very good for running IT-services, web-servers, and doing development with. There will be standard.*-flavors (for example standard.small) with between 1GB and 16GB of RAM. These are the cheapest flavors we have in ePouta. These will most-likely be your go-to flavor for non-heavy workloads. We are introducing hpc4 flavors, the next generation of HPC flavors. They are named hpc.4.80cores, hpc.4.40cores, hpc.4.20cores and hpc.4.10cores. These are very similar to the hpc.3.* flavors, with additional sizes and the newer Skylake CPU generation. The biggest instance has 50% more cores and memory (45, 90, 180 and 360 GB) than the previous generation. These flavors will probably be your go-to flavor for computational workloads. The new big memory flavor is called tb.4.735RAM and has 80 hyper-threaded Skylake cores, 735GB of memory and a whopping 2500GB of local SSD disk space. The SSD disks are configured in a RAID0 configuration which means that they are really fast but if the disk fails your data will be lo ## Object Storage Use Cases Part 2: Storing your reveal.js presentation in Object Storage DevFeed: [Object Storage Use Cases Part 2: Storing your reveal.js presentation in Object Storage]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-05-09T11:00:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Command-line interface](), [ceph](), [Cloud](), [Amazon Web Services]() Tags: [aws](), [ceph](), [cli](), [demo](), [installation](), [object-storage](), [openstack](), [presentation](), [python](), [radosgw](), [s3](), [storage]() ### AI overview A technical tutorial in the CSC Pouta Cloud Team series explains how to upload a reveal.js presentation to Pouta Object Storage using the awscli tool with Ceph RadosGW's S3-like implementation. It covers URL requirements, credentials, virtual-environment setup, installation, and configuration. ### Source excerpt This is part of a technical series of blog post where the CSC Pouta Cloud Team illustrates with words and sometimes figures how to do things with our object storage. The pevious installation in this series was the magnificent NFS + ZFS + Encryption blog post Topic for this post: we will experiment with Pouta Object Storage, the aws tool and reveal.js to upload a presentation at a nice https://mypresentation20180423.object.pouta.csc.fi/index.html URL. At the time of writing we have not enabled any automatic redirection to index.html with HAProxy if there is no filename specified, so "index.html" is needed or you will see the S3 XML details about the bucket (or AccessDenied). Also https:// at the beginning is needed as we do not yet redirect from TCP 80 to TCP 443. This blog post has two tracks, either you continue with the content on this blog post or you read the content in a reveal.js format on https://mypresentation20180423.object.pouta.csc.fi/index.html Step 1: Configure aws-cli We need something to write files to object storage. We will try the "industry standard" awscli tool to illustrate that it does work even though we are using Ceph's RadosGW which has an s3-like implementation. There are many clients one can use with S3. awscli comes with an s3 client inside which is working with our version for Ceph RadosGW. For some clients we have rudimentary documentation how to use them. You will need OpenStack ec2 credentials to use S3. You get those with this command: $ openstack ec2 credentials create Setting up a python virtual environment and install awscli inside it (so we don't mess with system libraries): $ virtualenv venv_aws $ source venv_aws/bin/activate $ pip install -U setuptools # this was needed on my CentOS7 laptop $ pip install awscli # don't confuse this with "aws" in pip Configuring it: $ aws configure AWS Access Key ID [None]: Enteryouraccesskeyidhere AWS Secret Access Key [None]: Enteryoursecretaccesskeyhere Default region name [None]: # press ENTE ## Admin Stories: Implement Object Storage in CSC's cPouta DevFeed: [Admin Stories: Implement Object Storage in CSC's cPouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-02-12T12:58:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [Amazon S3](), [Cloud](), [API]() Tags: [backend](), [ceph](), [cpouta](), [infrastructure](), [object](), [object-storage](), [openstack](), [rados](), [radosgw](), [rgw](), [s3](), [server](), [servers](), [storage](), [volume]() ### AI overview This article explains the decisions behind implementing object storage in CSC's cPouta cloud. It describes using Ceph RADOS Gateway with existing Ceph clusters and clarifies the relationship between S3 buckets, Swift containers, accounts, and OpenStack project IDs. ### Source excerpt In our previous user survey we found out that you were interested in "OpenStack admin stories". This will be our first attempt in doing that. User guide: https://research.csc.fi/pouta-user-guide While implementing object storage we made a few decisions. This blog post is about highlighting the decisions and their backing thought process. Some background information: cPouta uses OpenStack as the underlying cloud middleware, and Ceph for storing volumes and images. And going forward, objects, too. Terminology RADOS Gateway/RadosGW/RGW/Ceph RGW/Ceph radosgw/radosgw. We mean the same thing. It's a piece of software which exposes an API for storing and retrieving objects. Ceph - the storage system which RadosGW daemons access as a client. Buckets in S3 and Containers in Swift are synonymous. The Ceph radosgw-admin tool also operates on buckets. The latter are mostly the same, but can contain a bit more information i.e. OpenStack project ID. If we write about buckets, the reader can presume we are referring to S3 buckets and Swift containers unless otherwise noted. Account is something that RGW binds usage data into. In some of the radosgw-admin CLI calls, accounts are referred to as users or UIDs. These users and/or accounts get a mapping to OpenStack project IDs in our configuration. So be it account/user/project, we are always talking about the same thing. - Why Ceph RadosGW The other option we briefly looked at for an Object Storage server was the OpenStack Swift server. Yes, there is an API called Swift and a piece of sofware called Swift which serves the Swift API. Not to mention the programming language Swift, the ISO standard SWIFT or the Scottish potato variant Swift. In the end we used none of these for backend because we prefer to reuse existing infrastructure. We already have Ceph clusters used for Virtual Machine image and block (volume) storage in our clouds. Installing some Ceph RadosGW servers and pointing those to an existing Ceph cluster utilizes a lot o ## REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack DevFeed: [REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-12-07T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Provisioning](), [Hadoop](), [openstack](), [Orchestration](), [Python](), [Open Source](), [Docker]() Tags: [big-data](), [big-data-infrastructure](), [chef](), [cloud](), [docker](), [druid](), [elasticsearch](), [hadoop](), [open-source](), [openstack](), [orchestration](), [provisioning](), [python]() ### AI overview REDstack is an open-source sandbox tool for Big Data development that provisions kerberized Hadoop clusters on OpenStack. It combines a cookbook for installing and configuring cluster components with a Python orchestration application that manages resource provisioning, Chef deployment, and component installation. ### Source excerpt REDstack is Now Open Source! We are officially open sourcing REDstack, our sandbox tool for Big Data development at Target. What is REDstack? REDstack is a tool for provisioning kerberized clusters on OpenStack. We created it with four goals in mind: Provide a secured environment, with the ability to leverage preconfigured LDAP and Kerberos servers. Out of the box usability, allowing you to log in with preconfigured user accounts. Custom user management utilities to administer the cluster. Provide a fully customizable experience, everything is a configuration option in your build files: Cluster size, node sizes, types of nodes and node roles, Hadoop configurations, heap sizes, and components, All users, passwords, and secure assets. Components REDstack is made up of two major components: hdp-cloud - The cookbook The cookbook is used by the application itself to install components and lay down cluster configuration. The cookbook can be used independently of REDstack to manually provision a cluster. REDstack - The orchestration component REDstack is a python application that performs all of the high-level complexities and timings associated with a full Hadoop installation: Orchestrates the provisioning of resources over OpenStack APIs, Controls and monitors parallel Chef deployment across the cluster, Manages and monitors cluster component install over HTTPS requests. REDstack is bundled with a Docker image, where the configs are set up locally before an installation, and all of the dependencies are updated and configured. How to Get Started Head over to the repository at https://github.com/target/redstack and follow along. The repo has instructions on how to build and configure the clusters using the included Docker image. History of the Project Target's Big Data Platform Team manages multiple Big Data environments, with hundreds of nodes and many PB's of data. As mentioned in our prior blog posts, we depend heavily on Chef as a core part of our CI/CD pipeline. Durin ## A peek at OpenStack Neutron DevFeed: [A peek at OpenStack Neutron]() Original publisher: [Read original article]() Published: 2017-10-06T00:00:00Z Content type: tutorial Language: en Sources: [Gaurav Sarma's Blog]() Topics: [openstack](), [networking](), [virtual machines](), [Cloud]() Tags: [cloud](), [datacenter](), [gre](), [linux](), [networking](), [networks](), [nova](), [openstack](), [virtual-machines](), [vlan](), [vxlan]() ### AI overview This tutorial examines OpenStack Neutron, its networking service, and how it differs from the legacy Nova networking service. It introduces Linux networking concepts and describes network types including local, flat, VLAN, and VXLAN/GRE overlays, along with Linux Bridge and OVS plugins. ### Source excerpt From the OpenStack official [website](https://www. openstack... ## Measuring the Performance of our OpenStack Cloud DevFeed: [Measuring the Performance of our OpenStack Cloud]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-06-20T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [openstack](), [Performance Testing](), [Cloud](), [Hardware](), [HTTP](), [Latency](), [Command-line interface](), [Linux]() Tags: [cli](), [cloudpunch](), [configuration](), [hardware](), [http](), [latency](), [linux](), [openstack](), [performance](), [performance-testing]() ### AI overview This article describes Target's effort to measure performance in its private OpenStack cloud. It evaluates Rally and KloudBuster, finding that Rally focuses on API testing while KloudBuster supports HTTP and storage tests but lacks sufficient extensibility, configuration, and stability. The authors therefore decide to create their own flexible performance framework. ### Source excerpt Here at Target, we run our own private OpenStack cloud and have never been able to accurately measure the performance of our hardware. This lack of measurement prevents the evaluation of performance improvements of new hardware or alternative technologies running as drivers inside OpenStack. It also prevents us from providing a Service Level Agreement (SLA) to our customers. Recently we have been striving to improve our OpenStack service which led us to talk to our consumers directly. One of the major feedback points provided by talking with our consumers was the performance of the OpenStack cloud was lower than expected. Because we have not measured the performance of our cloud in the past, we have been unable to know if new hardware or configuration changes improves consumer-facing performance. With our new OpenStack environment builds we focused on changing this. But first we needed a tool to do the job. Searching for a Tool The first tool we looked at was Rally. Rally does performance testing of an OpenStack cloud. However, Rally focuses on the OpenStack API only. It is mainly used to test functionality (via Tempest) and stability of the API under large amounts of load. Rally does contain a resource to boot an instance and run Linux CLI commands via user data. This was tested as a way to provide the staging of instances to run performance software. However, starting the software on each instance at the same time and collecting the results from said software was difficult and not viable. Because of this, we deemed Rally was not suitable for our needs. The next tool we looked at was KloudBuster. KloudBuster is a tool that does performance testing inside an OpenStack instance. At the time of writing it provides two sets of tests: HTTP and storage. The HTTP test uses a traffic generator to measure requests per second and latency between instances. The storage test uses FIO to measure read/write IOPs and bandwidth. KloudBuster does what we were looking for, measuring ## Bare Metal Big Data Builds DevFeed: [Bare Metal Big Data Builds]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-01-13T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Hadoop](), [Automation](), [openstack](), [Nova](), [Jenkins](), [Bash](), [Python](), [API]() Tags: [ambari](), [api](), [automation](), [bare-metal](), [bash](), [big-data](), [chef](), [hadoop](), [infrastructure](), [ironic](), [jenkins](), [nova](), [openstack](), [python]() ### AI overview This engineering article describes Target's transition from manually building on-premise Hadoop clusters to an automated bare-metal provisioning workflow. It combines OpenStack Ironic and the Nova client with user-data bash scripts and Chef cookbooks to bootstrap servers, configure node roles, and add them to clusters. ### Source excerpt When you first think about scaling an on-premise Hadoop cluster your mind jumps to the process and the teams involved in building the servers, the time needed for configuring them and then the stability required while getting them into the cluster. Here at Target that process used to be measured in months. The story below outlines our journey around scaling our Hadoop cluster, taking the months to hours and adding hundreds of servers in a couple weeks. The Need Early 2013 taught us the lesson that manually managing Hadoop clusters, no matter how small is a time consuming and very repetitive task. Our next cluster build in 2014 drove the adoption of Chef, Artifactory and Jenkins to help with cluster operations. We stood up those components and created new role cookbooks to manage everything on the OS (configurations, storage, Kerberos, MySQL, etc.). While this was a step in the right direction, it left us with a manual process to still create the initial base server build and then add it to the cluster after configuring it with Chef. Build Foundation Closing the gap in our automation meant finding a way to deliver on true end to end builds, from an initial bootstrap to running jobs in your cluster. OpenStack's Ironic project was the first piece of the puzzle. Ironic gives us the ability to provision bare metal servers, similar to how OpenStack automated the VM build process. With Ironic as the foundation, we leveraged the Nova client to manage our instance builds. The nova python client interacts with the Compute service's API, giving us an easy way to specify our build parameters and spinning up an instance on one of our physical servers. The other key piece with the nova client is the ability to send boot information to the server using user data. The bash script sent executes several commands to install the Chef client, setup public keys and run the initial knife bootstrap to set the run list for the build. Example nova boot command: nova boot --image $image_name ## Target's Dojo Expands Access to DevOps Expertise and Training DevFeed: [Target's Dojo Expands Access to DevOps Expertise and Training]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2015-08-14T05:00:00Z Content type: opinion Language: en Sources: [Target]() Topics: [Development](), [DevOps](), [Automation](), [Learning](), [Kafka](), [openstack]() Tags: [agile](), [automation](), [developers](), [devops](), [kafka](), [learning](), [openstack](), [scrum]() ### AI overview Target describes a learning "Dojo" that brings subject matter experts together with project and product teams to expand DevOps skills and make training easier to scale. The dedicated space can support up to 8 or 9 teams at a time. ### Source excerpt At Target we're always looking for ways to move forward in becoming the best omni-channel retailer that we can be. This journey demands that we enable change in most every part of our technology organization. Our culture, delivery model, technology selections, working arrangements and org structures are all levers we can pull to help us be more responsive. A key question that we continually ask ourselves is "how can we move faster"? Introducing change in a large enterprise can take a long time if we don't challenge ourselves to be creative around constraints (like not enough "experts" to go around). Traditional learning approaches might not be fast enough, so we've started acting on some innovative ideas to break down those barriers. While this progress has been great, we've continually been asking ourselves how we could go faster. ###Building Capacity One of the newest things we doing is expanding our capacity for others to learn the new ways of doing things. Since we still have relatively few subject matter experts to go around, we are finding that access to these experts is a constraint that makes it difficult for us to scale the learning process for new teams. To address this constraint, we are standing up a learning "Dojo." The Miriam-Webster dictionary defines a Dojo as "a school for training in various arts of self-defense (as judo or karate)." At ChefConf 2015, Adam Jacobs delivered a very interesting talk titled Chef Style DevOps Kungfu. This presentation talked about building your practice through repetition and development of skills. We needed a place to develop our own special DevOps Kungfu. Cue the Dojo! The Dojo is a dedicated space in which our subject matter experts take up residence for an extended time. This is the home base for automation engineers, advanced Scrum leaders, OpenStack engineers, Chef experts, Kafka engineers, etc. Project teams and product teams looking to leverage these experts to build their own expertise can colocate their teams ## Meetup PostgreSQL à Paris DevFeed: [Meetup PostgreSQL à Paris]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2014-04-17T11:28:00Z Content type: opinion Language: fr Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [openstack](), [Object-relational mapping]() Tags: [meetup](), [openstack](), [orm](), [postgresql]() ### AI overview A report on the third PostgreSQL Meetup in Paris covers PostgreSQL use in production by France's Ministry of the Interior, OpenStack's use of PostgreSQL, ORM integration concerns, and a Tsung presentation. ### Source excerpt Hier soir se déroulait le troisième Meetup PostgreSQL à Paris, et je crois pouvoir dire que tous les participants étaient ravis de cette édition. Malheureusement, quelques uns sont restés bloqués du mauvais côté de la porte hier soir, j'en suis navré. Nous essayerons d'étendre nos horaires d'accueil lors des prochaines rencontres, dans la mesure du possible. Les sponsors de la soirée Un grand merci à nos sponsors : hier soir eNovance nous accueillait dans une superbe salle de conférence et Hegoa s'est chargée de mettre à disposition des participants un buffet très apprécié.