# CSC - IT Center For Science - Cloud Team Published articles for CSC - IT Center For Science - Cloud Team. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Docker Hub pull rate limits hit Rahti DevFeed: [Docker Hub pull rate limits hit Rahti]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2021-03-04T08:59:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Docker Hub](), [incident](), [Availability](), [Caching](), [Back end]() Tags: [availability](), [deployment](), [docker-container](), [docker-hub](), [incident](), [rate-limiting](), [registry]() ### AI overview This technical article describes how Docker Hub pull rate limits affected Rahti deployments and builds, and how administrators responded by recommending alternative repositories and caching or migrating images to Rahti's registry. The impact was compounded by availability problems in Rahti's own registry during a storage incident. ### Source excerpt November 2020 was an eventful month for Rahti. Our backend storage had incident that affected our production clusters image repository for many weeks. That event is well described in Allas downtime November 2020 - technical deep-dive. Just before our storage incident, 2nd of November, Docker applied "rate limiting for Docker container pulls for some users". When we tried to prepare for this in advance, it was very hard to figure out what the exact effect would be for Rahti. It took longer to resolve than we anticipated. Limits beganWe were relieved that the first day after pull rates were enforced was peaceful. Rahti was still up and running as it should. Users didn't complain about failing deployment in large amounts. Admins survived another upstream change. Eventually we received few tickets on failing builds. Our first approach was to instruct people to change their images away from Docker Hub to alternative repositories which do not have rate limiting enabled. Some users were able to migrate their images but this was not feasible solution for everyone. Still, it was a good opportunity to remind users to check what images they use. The error looks like this: Pulling image "docker.io/centos/python-38-centos7@sha256:da83741689a8d7fe1548fefe7e001c45bcc56a08bc03fd3b29a5636163ca0353" ... pulling image error : toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Cumulative issues cause exponential griefWe gave our users instructions how to move their dependencies from public Docker Hub to Rahti image registry, How to manually cache images in Rahti's registry. This was harder in practice, as we did have issues with our own registry due to our storage incident. There were days when users suffered from both Docker Hub and Rahti internal registry availability. Those days were hard for our Rahti admins, who struggled to meet customer expectations of highly available ser ## Allas November 2020 incident details DevFeed: [Allas November 2020 incident details]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2020-12-22T12:31:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [incident](), [Amazon S3](), [rocksdb](), [NVMe](), [Processes](), [Vapor]() Tags: [ceph](), [deep-dive](), [incident](), [nvme](), [processes](), [rocksdb](), [s3](), [software](), [storage](), [swift](), [technical]() ### AI overview A technical deep dive into the November 2020 Allas downtime incident. It describes Allas as a Ceph cluster and explains its storage nodes, OSD processes, S3 and SWIFT access, replicated metadata pools, RocksDB metadata, placement groups, and erasure-coded object storage. ### Source excerpt Allas downtime November 2020 - technical deep-dive We use Ceph (https://ceph.io/) to provide software defined storage in our Allas service. I'll try to speak some Ceph here. There may be a lot of unfamiliar terms for most readers here as we dive deep into a technology. I'll try to clarify where I can, but architectural knowledge of Ceph helps when reading this. Allas is a Ceph cluster with 48 storage nodes (OSD nodes). Each has one nvme, 24 spinning disks, and 192 GiB of memory. Each OSD node runs one OSD process per disk. The Allas storage is provided over SWIFT and S3 protocols through radosgw daemons. The radowsgw layer is provided by separate internet facing servers. In addition we have 5 nodes running monitor processes (they control the state of the cluster), which also run Ceph manager (handles more advanced monitoring views into the cluster state). The object store metadata pools are 3-way replicated pools (data is stored in 3 different copies) on the nvmes across the OSD nodes. This metadata contains e.g. owners of buckets and objects, permissions, versions, etc. Everything that has to do with S3/SWIFT. The nvme also hosts the rocksdb (ceph internal metadata, as opposed to object store metadata) for the OSDs. This metadata is internal Ceph clustster status. Data on placement groups, etc. I'll use the term placement group (or pg) quite a lot in the post as they are central for Ceph. Basically when you store data objects into Ceph, they are mapped into placement groups, which define where in the cluster the data is are stored. The object store data is stored on an 8+3 erasure coded (data is split into 8 chunks + 3 parity chunks) pool on the spinning disks. The total amount of placement groups for the data pool is 16 384. In the graph the amount ~18 000 is shown, as the graph contains other pools too. The beginning Side note: I love that we collect trending data for our cluster (collectd+graphite in our case). As we had the data stored from a long time, this al ## CentOS8 images published in Pouta as a tech preview DevFeed: [CentOS8 images published in Pouta as a tech preview]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2020-02-17T05:48:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Cloud]() Tags: [centos](), [centos8](), [cloud-data](), [diskimage](), [glance](), [image](), [openstack](), [pouta](), [preview](), [published](), [tech](), [ubuntu]() ### AI overview CentOS 8 images are available in Pouta as a tech preview. The article notes minor upstream issues, including a resolv.conf nameserver issue, and describes temporary image modifications to address one issue. ### Source excerpt There are now CentOS-8 images available in Pouta! There are some minor issues with the upstream CentOS8 images, so, for now, they are considered to be in "tech preview". We have solved the one we have found so far by temporarily modifying the image to use "cloud-user" and remove the resolv.conf leftovers. Basic information about our images can be found on docs.csc.fi One issue is that /etc/resolv.conf sometimes has a nameserver defined from the build of the image. There is an open CentOS bug report about this: https://bugs.centos.org/view.php?id=16948 ## Migrating cPouta Object Storage to the Allas Ceph Service DevFeed: [Migrating cPouta Object Storage to the Allas Ceph Service]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2019-12-27T11:23:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [migration](), [Data Management](), [data](), [cloud-infrastructure]() Tags: [ceph](), [data-management](), [migration](), [object-storage](), [openstack](), [radosgw](), [rgw](), [storage]() ### AI overview This article describes CSC's migration from the cPouta object storage service to Allas, a standalone Ceph-based object storage service. The plan aimed to move user data without requiring user changes or breaking existing data links, using a new Ceph cluster and new data pools. ### Source excerpt We have released our new Allas object storage service. We firmly believe it will play a growing role for data management, at CSC and for the whole academic field in Finland. Alas, the road to Allas was not completely without pain. CSC also hosts the cPouta IaaS service. It provided its own object storage service. Our goal with Allas was to build on this, and transform the object storage portion to a standalone service. This would raise the profile of the service, and make it easier to scale, both human resource wise and where it comes to the platform size. So, in brief, our aim was: Replace cPouta object storage with Allas Make sure all the user data from cPouta object storage moves over. Make sure cPouta object storage users don't have to do any changes. Don't break any existing links to data in cPouta object storage. How did we intend to do this, you may ask? Migration plan What were our options? We had a large bunch of new hardware, and a high level plan. For those of you who know Ceph, you might have already thought of how you'd do this. Add more hardware to the same storage pool, slap on an additional domain name to access the data, and call it a day. Create additional data pools, make sure radosgw uses old and new pools for data, slap on an additional domain name, and call it a day. Create a new Ceph cluster with new pools, migrate the data over, add the old and new domain names, and call it a day. Number 3. is definitely the most work, but we went with that. The main reason for that was that if we want to achieve all the benefits of separating Allas into its own service, we actually need to be a bit separate. For example, the old object storage pools were in the same cluster as the block storage pools used by our OpenStack virtual machines. This not only locked us to the current version of Ceph running there, but created a lot on interdependencies between the services. To make it easier, I'll use the terms rgw-old (cPouta object storage) and rgw-new (Allas) t ## The Bastion host: How to set up and benefits of using it. DevFeed: [The Bastion host: How to set up and benefits of using it.]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-07-03T07:24:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Security](), [ssh](), [OpenSSH](), [Ansible](), [Network](), [servers]() Tags: [ansible](), [bastion](), [how-to](), [network](), [remote](), [security](), [servers](), [ssh](), [user-guide]() ### AI overview This tutorial explains bastion hosts as intermediary jump hosts for connecting to internal servers. It describes how they reduce exposure by keeping internal services off the internet and outlines setup using users, SSH ProxyCommand, an SSH configuration file, and Ansible playbooks. ### Source excerpt What is a bastion host Bastion host just works like a jumphost, through which, we can connect to other servers. It can be used as an entry point to connect to internal servers which increases security as well. The following diagram can be useful to understand how we can connect to our internal hosts through a bastion host. The benefit of using bastion hosts One advantage using a bastion is the ability to use services which do not need to be directly accessible from internet, as in do not require a floating IP for each individual server. Most admin traffic and some user traffic can be tunnelled through the bastion host. The second and more important advantage is that by not having a floating IP the attack surface is reduced. A common mistake is that users are running inherently insecure services like network files system, databases, remote desktops a.s.o. and by mistake makes these services available to the internet. By not giving floating IPs to instances that run internal service you are eliminating one attack vector and also implement good practice of using layered security https://simple.wikipedia.org/wiki/Defense_in_depth_(computing). A bastion host itself has quite a small attack vector, for example in our example linked below we would only allow for incoming ssh traffic. How to create a bastion host We can make one host as our bastion host. To do so, we need to set up users in bastion host so that through bastion host, same users can pass through and log in to other internal servers and do administrative tasks. After setting up users in the bastion host, we can use proxycommand option of ssh to connect to other servers behind the bastion host. We also need a predefined ssh.config file. We can use ansible to make a host into a bastion host. Bastion git repo has some playbooks and an example ssh.config file. How to connect a host through a bastion host After we have set up our users in bastion host we can connect to our internal servers through a bastion host by ## Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository DevFeed: [Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-04-29T09:06:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Git](), [ceph](), [pip](), [Haskell]() Tags: [aws](), [ceph](), [data](), [data-lake](), [git](), [install](), [object-storage](), [python](), [rados](), [radosgw](), [s3](), [storage]() ### AI overview This post explains why binary files should not be stored directly in Git repositories and explores using CSC's Pouta Cloud Object Storage instead. It documents attempts with git-annex and git-bigstore, including a patch to configure a non-default S3-compatible endpoint. ### Source excerpt 1. It's bad to store binary blobs in git. This is known [1] 2. A great place to store binary files is in object storage 3. Can we use CSC's Pouta Cloud Object Storage service for this? Why yes, yes you can! This is a post in a series about different use cases for object storage. Others include for example hosting a reveal.js presentation and NFS server backups More details about CSC's Pouta Cloud Object Storage Service, it is based on CEPH Rados Object Storage Gateway. What The general solution* to storing binaries in git is to not store them in git and instead create a link to some other place Attempt 1 Install an application that you like. In https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories I searched for "S3", found git-annex which had recent commits so it looked like it wasn't dead. Installation instructions: https://git-annex.branchable.com/install/ The version in EPEL/Ubuntu Xenial does not have S3 support and is from 2014. There's a recent one in Fedora's repos though. Unfortunately, compiling it requires cabal. I don't know Haskell. https://spack.readthedocs.io/en/latest/package_list.html does not have any Haskell, cabal or git-annex available. Next. Attempt 2 Next application in the blog post was https://github.com/lionheart/git-bigstore To install it was much more comfortable for me: pip install "git-bigstore<=2.0" First issue was that looking at https://github.com/lionheart/git-bigstore/blob/master/bigstore/backends/s3.py there was no way to tell it to use another endpoint than the default of AWS S3. A small patch takes care of that, add , endpoint_url='https://object.pouta.csc.fi:443' to the client() boto3 call in the backends/s3.py file from git-bigstore. For a more detailed example see this Pull Request. After spending some time going down some rabbit holes it seems that the Profile option that is supported in git-bigstore does not support endpoints. It can only be used to specify credentials and a few other parameters. Mino ## cPouta Cloud Object Storage Service has been updated to run CEPH Luminous! DevFeed: [cPouta Cloud Object Storage Service has been updated to run CEPH Luminous!]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-02-06T10:03:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [Amazon S3](), [Cloud](), [configuration](), [Command-line interface]() Tags: [amazon-s3](), [api](), [automatic](), [ceph](), [cli](), [configuration](), [cpouta](), [format](), [object-storage](), [openstack](), [radosgw](), [s3](), [storage](), [upgrade](), [xml]() ### AI overview The CSC Pouta Cloud Team describes an upgrade of its Ceph Object Storage Gateway from Jewel to Luminous. The update enables AWS4 signatures by default and adds support for several S3 features, including bucket lifecycle, website hosting, and request payment. The article also begins an example of configuring an S3 lifecycle policy with s3cmd and XML. ### Source excerpt We (the CSC Pouta Cloud Team) has finally upgraded CEPH Object Storage Gateway from Jewel to Luminous! There may come more blog posts about other improvements to other of our Openstack services (cinder and glance) which use CEPH. No need to use s3 --signature-v2 anymore! Before this upgrade, anyone using the S3 protocol needed to use the "AWS2" for most actions to work. Now that we run Luminous it is no longer necessary and the default AWS4 signature is working. Yay! In general things should be faster, shinier and also more API calls are supported. Comparing http://docs.ceph.com/docs/luminous/radosgw/s3/ and http://docs.ceph.com/docs/jewel/radosgw/s3/ these features* have gone from unsupported to supported: Bucket Lifecycle Supported Removing expired files is supported Bucket Website Supported Bucket Request Payment Supported Which got me a bit excited! Now to learn a bit about these and see if they work on our setup and if possible if we can make them work without breaking anything for existing objects & users: Bucket Lifecycle This is about expiration/automatic removal of objects at a certain time or after a certain time has passed. Let's start with S3 Some more details about the policies from Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html https://docs.aws.amazon.com/AmazonS3/latest/dev/set-lifecycle-cli.html Telefonica open cloud has some nice documentation with s3cmd examples: https://support.telefonicaopencloud.com/en-us/ugs3cmd/obs/en-us_topic_0051518501.html I have not found anything better on docs.ceph.com -- Step 1. Create a lifecycle policy with s3cmd and in the XML format. (there is a JSON style with awscli and "aws s3api get-bucket-lifecycle-configuration --bucket mybucketname" ): Call the file expiry.xml and make the content something like this: ExampleRule expire_these_ Enabled 1 ) Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-12-31T06:50:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [mount](), [Filesystems](), [Linux](), [Cloud]() Tags: [cinder](), [cloud](), [cpouta](), [filesystem](), [fstab](), [image](), [linux](), [maintenance](), [mount](), [openstack](), [pouta](), [server](), [storage](), [systemd](), [volume](), [volumes]() ### AI overview This article explains how to mount block-device volumes in Linux instances and why relying on standard /etc/fstab entries can prevent an instance from booting when a volume is missing or differs after an image-based deployment. It recommends mount options such as nofail and discusses noauto. ### Source excerpt Some background: As part of the CSC Pouta Cloud Openstack installation we provide the cinder service. This allows our customers to attach block devices to their instances. A common way to illustrate what this means is to say: ".. it's like attaching a USB drive to the instance. " - Some Openstack trainer After attaching the block devices one has to format it with a filesystem and mount it. The usual way to mount block devices (in most Linux based distributions I've seen, if you know other ways please comment below!) every time the server reboots is to put something like this into /etc/fstab: /dev/vdb1 /mnt ext4 defaults 0 0 All clear so far I hope. It mounts block device /dev/vdb to /mnt as filesystem ext4 with the defaults mount options and the last two fields (numbers) are about dumping and file system checks. (Sometimes it's wise to use UUID or LABEL and not /dev/vdb1 in case you have more than two volumes.) But what happens if disk with is not there anymore? Well, with the above fstab your instance will not boot into the operating system. You'll be stuck having to perhaps enter single user mode or attaching a volume and then rebooting and modifying fstab. Single user mode is probably OK sometimes (it's inconvenient and takes time). But let's say you took a snapshot of an instance in Openstack with the glance service. This creates an image of the root disk. If you then spawn a new instance with that image as a root disk it will still have that /dev/vdb1 in /etc/fstab. Hey ho, the new instance booted from the image will not boot. Here you could of course just create a new volume and attach it. But if you used UUID or LABEL in fstab then it wouldn't find those and also break! So what is the best(tm) way? By using some nice fstab mount options. For example nofail UUID=1c7fc2f-f9d9-4e01-8d0f-190f05416831 /mnt ext4 nofail,defaults 0 0 There's another mount option called "noauto". The difference is put quite nicely in this stackoverflow question which has a piece from ## cPouta Flavour Pikkujoulu 2018 Communiqué DevFeed: [cPouta Flavour Pikkujoulu 2018 Communiqué]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-11-26T14:44:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cloud-infrastructure](), [ceph](), [Hardware]() Tags: [capacity](), [ceph](), [cpouta](), [energy](), [hardware](), [hpc](), [standard](), [storage]() ### AI overview The communiqué announces changes to cPouta VM flavours: several older HPC flavours will no longer be available for new launches, while new hpc.4 and standard.xxlarge flavours will be added. The hpc.4 flavours use Ceph for VM root-disk backing storage, replacing older hardware without adding cPouta capacity. ### Source excerpt We are making some changes to the cPouta flavours offering tl;dr You won't be able to launch _new_ hpc-gen1.*core, hpc-gen2.2core, hpc-gen2.8core, hpc-gen2.16core flavours. We are adding hpc.4 flavours. We are adding a standard.xxlarge flavour. The new hpc.4 flavours The hpc.4 flavours are similar to the similarly named flavour in ePouta. The big difference between the hpc-genX is that hpc.4 will be using Ceph as default backing storage for VM root disks. We have wholesome experiences from Ceph in ePouta where all hpc-flavours are using Ceph as backing storage. One of the benefit is that if a hypervisor breaks, we can either migrate the customer instances away without customer noticing anything, or if the hypervisor has catastrophic failure the instance can be booted up on a new hypervisor without loss of data. Furthermore the sequential reads are much faster than on local spinning disks. This change will not bring additional capacity in cPouta, but replace old energy inefficient hardware in-kind with newer hardware. The new bigger standard flavour standard.xxlarge The standard flavours are the most popular flavours by a huge margin. An interesting thing that we have noticed is that if you combine the amount of hpc-gen2.8core and hpc-gen1.8core flavours that would be the second most popular flavour. We belivie that many will prefer to use standard.xxlarge instead of hpc flavours with similar amount of RAM. The new flavours hpc.4.5core - 5 cores 22GB RAM - replacement for hpc-gen1.1core, hpc-gen1.4core, alternative to hpc-gen2.2core, hpc.4.10core - 10 cores 44GB RAM - replacement for hpc-gen1.8core, alternative to hpc-gen2.8core hpc.4.20core - 20 cores 88GB RAM - replacement for hpc-gen1.16core, alternative to hpc-gen2.16core and hpc-gen2.24core standard.xxlarge - 8 oversubscribed cores, 32GB RAM - alternative to hpc-gen2.8core, hpc-gen1.8core, hpc4.5core and hpc.4.10core Why are the hpc-gen1 flavours disappearing? Existing instances of these flavour will continue to ## ePouta Flavour Halloween 2018 Communiqué DevFeed: [ePouta Flavour Halloween 2018 Communiqué]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-10-19T10:51:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [GPU](), [cpu](), [Nvidia]() Tags: [applications](), [cpu](), [deprecated](), [epouta](), [espoo](), [flavors](), [gpu](), [hpc](), [newton](), [numa](), [nvidia](), [openstack](), [performance](), [pike](), [pouta](), [standard](), [westmere]() ### AI overview The communiqué announces changes to ePouta flavors, including public availability of the gpu.2.1gpu flavor with an NVIDIA Tesla V100 and NUMA-aware CPU and memory behavior. It also announces the deprecation of hpc.*.westmere flavors and recommends replacement flavors for new instances. ### Source excerpt We are making some changes to our flavor offering . Here's a summary of the changes: We now have even more GPUs in ePouta. The gpu.2.1gpu flavor is now publicly available and each has one NVIDIA Tesla V100 via PCI pass through. This flavor was already added in July but could only be accessed after requesting it from servicedesk. Now we are making them publicly available. The gpu.2.1gpu flavor is our first NUMA aware flavor. This should speed up CPU & memory heavy applications. Our current version (Newton) of OpenStack is not able to guarantee PCI devices from the same NUMA node too, this is possible to come when we get to Pike or later. Deprecation of hpc.*.westmere flavors This does not affect running instances but in the future you might see changes in the performance and therefor we suggest that you resize your flavor to a newer flavor type. You won't be able to launch new instances with the deprecated flavors. Flavors that will be deprecated and suggested replacement: hpc.medium.westmere - standard.xlarge hpc.large.westmere - hpc.4.10core hpc.xlarge.westmere - hpc.4.10core or hpc.4.20core hpc.largemem.westmere - hpc.4.20core or hpc.3.28core More information about the different flavors can be found here: https://research.csc.fi/pouta-flavours No known Copyright Please acknowledge 'Sir George Grey Special Collections, Auckland Libraries, 1-W682' when re-using this image. ## Ubuntu 18.04 images published in cPouta and ePouta DevFeed: [Ubuntu 18.04 images published in cPouta and ePouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-10-17T07:30:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Ubuntu](), [Cloud](), [ssh](), [Command-line interface](), [releases]() Tags: [cli](), [cloud](), [cpouta](), [diskimage](), [ed25519](), [epouta](), [horizon](), [image](), [lts](), [openstack](), [release](), [ssh](), [ubuntu]() ### AI overview Ubuntu 18.04 LTS images are available in the cPouta and ePouta cloud environments. The article explains that the images use Canonical's unmodified image, the default SSH user is ubuntu, and users can upload other Ubuntu cloud images or configure users with cloud-init and the OpenStack CLI. ### Source excerpt Hello Ubuntu fan boys and girls! An image containing latest LTS Ubuntu release (18.04 Bionic Beaver) is now available in c and ePouta! This time we opted to use the image provided by Canonical without any modifications. This means the user used to ssh in is now "ubuntu" instead of "cloud-user". In any case, trying to ssh in as root will tell you which user to ssh in as In case you want to try even newer versions of Ubuntu, say the "in-between" releases like 18.10 Cosmic Cuttlefish - head over to http://cloud-images.ubuntu.com/ , select the release and fetch the "cloudimg-amd64.img" - for example "cosmic-server-cloudimg-amd64.img" - which is a QCOW2 image. This you can then upload as an image in your c/ePouta project. Our user guide has some pictures and instructions how to work with and upload new images: https://research.csc.fi/pouta-adding-images To add your ssh key under a new "cloud-user" (in case you don't want to use ubuntu user) you can do something like below. First create a file that has the cloud-init user data. Then use that file as input to the openstack CLI tool. $ cat cloud-init.cfg #cloud-config users: - default: - default_user: name: cloud-user gecos: Cloud User sudo: ["ALL=(ALL) NOPASSWD:ALL"] ssh_authorized_keys: - ssh-ed25519 AAAAMYKEYQ user@email.example.org And then the openstack CLI command : $ openstack server create ubuntu1804 --flavor standard.small --image Ubuntu-18.04 --key mykey --security-group default --security-group ssh-group --user-data cloud-init.cfg It is if course possible to use horizon to supply the user data as well. p.s. Yes, ed25519 keys works this way! The web interface openstack horizon still does not support them yet. ## ePouta adds new VM flavors and server hardware DevFeed: [ePouta adds new VM flavors and server hardware]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-07-03T09:23:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cloud-infrastructure](), [virtual machines](), [Hardware](), [cpu](), [Nvidia]() Tags: [availability](), [cpu](), [epouta](), [espoo](), [flavors](), [gpu](), [hardware](), [hpc](), [nvidia](), [openstack](), [pouta](), [virtual-machines]() ### AI overview CSC announces new VM flavors and server hardware for ePouta, including standard flavors, Skylake-based HPC flavors, a 735 GB big-memory flavor, and NVIDIA Tesla V100 GPU instances available by request. ### Source excerpt We are pleased to announce the immediate availability of new lovely VM flavors in ePouta backed by new server hardware! A summary of the changes: introduced standard-flavors to ePouta; these are the most popular flavors in cPouta. introduced a new generation of hpc-flavors which have Skylake CPUs. a new generation of big memory flavor with 700GB of RAM. GPUs in ePouta. The flavors have 1 NVIDIA Tesla V100. These are only available per request to CSC Service Desk. deprecating hpc.mini, hpc.small and all Haswell hpc-flavors except hpc.fullnode.haswell. This does not affect the io.haswell-flavors. As always the current description of flavors can be found on https://research.csc.fi/pouta-flavours New Flavors The new standard flavors are of the same configuration that already exists in cPouta. This flavor is really cheap but not suitable for heavy workloads since the CPU is oversubscribed. This means that many virtual machines have to share the same CPUs. These instances are very good for running IT-services, web-servers, and doing development with. There will be standard.*-flavors (for example standard.small) with between 1GB and 16GB of RAM. These are the cheapest flavors we have in ePouta. These will most-likely be your go-to flavor for non-heavy workloads. We are introducing hpc4 flavors, the next generation of HPC flavors. They are named hpc.4.80cores, hpc.4.40cores, hpc.4.20cores and hpc.4.10cores. These are very similar to the hpc.3.* flavors, with additional sizes and the newer Skylake CPU generation. The biggest instance has 50% more cores and memory (45, 90, 180 and 360 GB) than the previous generation. These flavors will probably be your go-to flavor for computational workloads. The new big memory flavor is called tb.4.735RAM and has 80 hyper-threaded Skylake cores, 735GB of memory and a whopping 2500GB of local SSD disk space. The SSD disks are configured in a RAID0 configuration which means that they are really fast but if the disk fails your data will be lo ## Object Storage Use Cases Part 2: Storing your reveal.js presentation in Object Storage DevFeed: [Object Storage Use Cases Part 2: Storing your reveal.js presentation in Object Storage]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-05-09T11:00:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Command-line interface](), [ceph](), [Cloud](), [Amazon Web Services]() Tags: [aws](), [ceph](), [cli](), [demo](), [installation](), [object-storage](), [openstack](), [presentation](), [python](), [radosgw](), [s3](), [storage]() ### AI overview A technical tutorial in the CSC Pouta Cloud Team series explains how to upload a reveal.js presentation to Pouta Object Storage using the awscli tool with Ceph RadosGW's S3-like implementation. It covers URL requirements, credentials, virtual-environment setup, installation, and configuration. ### Source excerpt This is part of a technical series of blog post where the CSC Pouta Cloud Team illustrates with words and sometimes figures how to do things with our object storage. The pevious installation in this series was the magnificent NFS + ZFS + Encryption blog post Topic for this post: we will experiment with Pouta Object Storage, the aws tool and reveal.js to upload a presentation at a nice https://mypresentation20180423.object.pouta.csc.fi/index.html URL. At the time of writing we have not enabled any automatic redirection to index.html with HAProxy if there is no filename specified, so "index.html" is needed or you will see the S3 XML details about the bucket (or AccessDenied). Also https:// at the beginning is needed as we do not yet redirect from TCP 80 to TCP 443. This blog post has two tracks, either you continue with the content on this blog post or you read the content in a reveal.js format on https://mypresentation20180423.object.pouta.csc.fi/index.html Step 1: Configure aws-cli We need something to write files to object storage. We will try the "industry standard" awscli tool to illustrate that it does work even though we are using Ceph's RadosGW which has an s3-like implementation. There are many clients one can use with S3. awscli comes with an s3 client inside which is working with our version for Ceph RadosGW. For some clients we have rudimentary documentation how to use them. You will need OpenStack ec2 credentials to use S3. You get those with this command: $ openstack ec2 credentials create Setting up a python virtual environment and install awscli inside it (so we don't mess with system libraries): $ virtualenv venv_aws $ source venv_aws/bin/activate $ pip install -U setuptools # this was needed on my CentOS7 laptop $ pip install awscli # don't confuse this with "aws" in pip Configuring it: $ aws configure AWS Access Key ID [None]: Enteryouraccesskeyidhere AWS Secret Access Key [None]: Enteryoursecretaccesskeyhere Default region name [None]: # press ENTE ## Object Storage Use Cases Part 1: NFS Server Backups to cPouta S3 DevFeed: [Object Storage Use Cases Part 1: NFS Server Backups to cPouta S3]() Original publisher: [Read original article]() Author: Jukka Nousiainen (noreply@blogger.com) Published: 2018-04-24T11:20:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Amazon S3](), [Server](), [Ansible](), [configuration-management](), [ceph](), [Availability](), [Ubuntu](), [API](), [cloud-infrastructure]() Tags: [ansible](), [backups](), [ceph](), [object-storage](), [storage](), [ubuntu]() ### AI overview This tutorial presents a backup design for NFS shares hosted on a cPouta virtual machine, with backups scheduled to cPouta object storage. It discusses encrypting data before it reaches shared storage, using ZFS, Ansible, S3, and Ubuntu 16.04, and considers failure domains and storage redundancy. ### Source excerpt After releasing object storage in cPouta, we've been asked quite a lot about various use cases. Thus we're starting a blog series to showcase some of them. Premise A popular design pattern for Pouta users is to host a data storage NFS server in the default project network and back it up to a host on their campus or home organization network. In similar fashion, in this blog post we will be enabling NFS shares from a cPouta Virtual Machine and scheduling backups to cPouta object storage. Since the demand to process biomedical and other types of sensitive data in cloud computing platforms is increasing, we also try to cater for these kind of scenarios by choosing a model where data is encrypted before it lands on any kind of shared storage. The reader is expected to be somewhat familiar with the following concepts: NFS which is used for file sharing within project network ZFS which is used as encrypting file storage Ansible which is used for configuration management S3 as the chosen API for uploading objects. The operating system used in the examples is Ubuntu 16.04. Failure domains One of the most important things to take into account when designing any kind of backup scheme are the failure domains of the underlying systems and services. In cPouta, users can store data in several locations, which map to several failure domains: standard persistent volumes on highly redundant Ceph block storage standard VM root disks on highly redundant Ceph block storage Highly redundant Ceph object storage RAID1 root disks of gpu VM flavors RAID0 root disks of hpc-gen1, hpc-gen2 and io flavors RAID0 ephemeral disks of io flavors Furthermore, all of these resources are currently in the single availability zone called nova. In our scheme, active NFS server data will reside on a dedicated persistent volume. Backups will go to object storage. As for redundancy, currently all backing Ceph pools are configured to replicate three copies of each block or object. Targeted data locations are ## Customer Invisible Maintenance of the CSC Object Storage Servers DevFeed: [Customer Invisible Maintenance of the CSC Object Storage Servers]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-03-14T07:36:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [BGP](), [networking](), [Load Balancing](), [Availability](), [round robin]() Tags: [bgp](), [ceph](), [cpouta](), [dns](), [exabgp](), [failover](), [ha](), [high-availability](), [load-balancing](), [maintenance](), [network](), [networking](), [object-storage](), [radosgw](), [round-robin](), [routing](), [server]() ### AI overview The CSC Pouta Cloud Team describes using ExaBGP on object storage servers to combine round-robin DNS load balancing with BGP routing for high availability and failover during maintenance. ### Source excerpt Hello! It's the CSC Pouta Cloud Team again! You may recognize us from other object storage films such as: http://pouta.blog.csc.fi/2018/02/admin-stories-implement-object-storage.html where we go through in excruciating detail how we configured CEPH RadosGW and why we configured it like we did http://pouta.blog.csc.fi/2018/01/objectively-speaking-in-cpouta.html where we announced object storage! In this episode we're going to talk a bit about how we implemented invisible(*) maintenance for our Object Storage customers. Configuration of ExaBGP with Object Storage CSC - IT Center for Science Ltd. has a close collaboration with the Finnish University and Research Network - FUNET, and because of that we are in a great position to do some fun network stuff. Actually, as long as your network has a BGP relationship with its internet provider you can also do what we have done. When we first launched Object Storage we just had a DNS entry pointing to two IP addresses. This gave us some load balancing and if we really wanted to we could take a node out of the DNS and do maintenance. But we wanted more! *ExaBGP enters the stage* In short we installed ExaBGP on our object storage linux servers, configured one dummy network interface per server and started to announce two IP addresses from each server. Then we pointed object.pouta.csc.fi DNS entry to these two IP addresses. The objective was to use round-robin DNS for load-balancing and BGP routing for High Availability. For failover this means that if one node goes down, the other IP is reachable on the other node and stuff will keep magically(*) working! There are some other BGP softwares out there, like BIRD or Quagga. Despite this, we never actually tried any other software since ExaBG has some great qualities such as: it came recommended by the network team at CSC who uses it for some services its low resource footprint and small number of software dependencies it is not a full routing suite - it is meant to be used for basi ## Admin Stories: Implement Object Storage in CSC's cPouta DevFeed: [Admin Stories: Implement Object Storage in CSC's cPouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-02-12T12:58:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [Amazon S3](), [Cloud](), [API]() Tags: [backend](), [ceph](), [cpouta](), [infrastructure](), [object](), [object-storage](), [openstack](), [rados](), [radosgw](), [rgw](), [s3](), [server](), [servers](), [storage](), [volume]() ### AI overview This article explains the decisions behind implementing object storage in CSC's cPouta cloud. It describes using Ceph RADOS Gateway with existing Ceph clusters and clarifies the relationship between S3 buckets, Swift containers, accounts, and OpenStack project IDs. ### Source excerpt In our previous user survey we found out that you were interested in "OpenStack admin stories". This will be our first attempt in doing that. User guide: https://research.csc.fi/pouta-user-guide While implementing object storage we made a few decisions. This blog post is about highlighting the decisions and their backing thought process. Some background information: cPouta uses OpenStack as the underlying cloud middleware, and Ceph for storing volumes and images. And going forward, objects, too. Terminology RADOS Gateway/RadosGW/RGW/Ceph RGW/Ceph radosgw/radosgw. We mean the same thing. It's a piece of software which exposes an API for storing and retrieving objects. Ceph - the storage system which RadosGW daemons access as a client. Buckets in S3 and Containers in Swift are synonymous. The Ceph radosgw-admin tool also operates on buckets. The latter are mostly the same, but can contain a bit more information i.e. OpenStack project ID. If we write about buckets, the reader can presume we are referring to S3 buckets and Swift containers unless otherwise noted. Account is something that RGW binds usage data into. In some of the radosgw-admin CLI calls, accounts are referred to as users or UIDs. These users and/or accounts get a mapping to OpenStack project IDs in our configuration. So be it account/user/project, we are always talking about the same thing. - Why Ceph RadosGW The other option we briefly looked at for an Object Storage server was the OpenStack Swift server. Yes, there is an API called Swift and a piece of sofware called Swift which serves the Swift API. Not to mention the programming language Swift, the ISO standard SWIFT or the Scottish potato variant Swift. In the end we used none of these for backend because we prefer to reuse existing infrastructure. We already have Ceph clusters used for Virtual Machine image and block (volume) storage in our clouds. Installing some Ceph RadosGW servers and pointing those to an existing Ceph cluster utilizes a lot o ## cPouta Releases Object Storage for All Customers DevFeed: [cPouta Releases Object Storage for All Customers]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2018-01-31T11:32:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Cloud](), [cloud-infrastructure](), [data]() Tags: [access-control](), [backups](), [cloud-storage](), [cpouta](), [datasets](), [images](), [object-storage](), [pipeline](), [release](), [storage](), [web]() ### AI overview cPouta has released object storage for all customers. The cloud storage service stores and retrieves data over HTTPS, can be accessed from inside or outside cPouta, and supports different access-control levels and use cases such as website assets, datasets, research pipelines, and backups. ### Source excerpt We're pleased to release one of the most sought after features in cPouta - object storage! Object storage is now usable by all cPouta customers. It's a new service with new usage models, so for details I suggest that you check out our user documentation. This cat lives in the cPouta object storage. What is Object Storage? Object storage is a cloud storage service, where you can store and retrieve data over HTTPS. This is not tied to any individual virtual machines, and it's accessible from anywhere, inside and outside cPouta. There is also a variety of ways to share the data with others. Why should I use it? Object storage is generally used for a different purpose that many other storage solutions. Some benefits include You do not need to set up a virtual machine to serve or receive the data. The data can be accessed from anywhere using the same URL. The data can have different levels of access control. What can it be Used for? The object storage has tons of different use cases. Web sites can host their assets there (images, videos), researchers can publish their datasets, you can use it as a data staging platform for your research pipeline, you can push your backups there, among a horde of other things. The Inevitable "But" As with all new services, the first public release is never perfect. In the coming months we will upgrade things on the backend, to improve the system after real-scale experience. This should mostly be invisible to the you, but you might see small glitches when things are changed. We'll also improve the documentation and use case descriptions based on feedback. What we won't do is harm your data. We would not be releasing the service, if we were worried that we have to remove, or move or change the things you store there. For any problems with using the service, please don't hesitate to contact servicedesk@csc.fi. ## Spectre and Meltdown in Poutas DevFeed: [Spectre and Meltdown in Poutas]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2018-01-09T13:36:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cpu](), [Vulnerabilities](), [Exploit](), [virtual machines](), [Linux](), [Ubuntu](), [x86](), [passwords]() Tags: [apt](), [cpu](), [exploit](), [installation](), [linux](), [packages](), [passwords](), [ubuntu](), [update](), [virtual-machines](), [vulnerabilities](), [x86]() ### AI overview This article explains the Spectre and Meltdown CPU vulnerabilities and their impact on cloud providers and customers. It recommends updating kernels and rebooting virtual machines, with specific instructions for Ubuntu and CentOS. Meltdown requires immediate mitigation, while Spectre is harder to fully mitigate. ### Source excerpt Three serious CPU vulnerabilities were revealed last week, and they require immediate action. Both we as providers and you as customers are affected. Meltdown should be mitigated against immediately. Update the kernels on your virtual machines and reboot your virtual machines. 2018-01-10 13:45: Ubuntu has released new kernel patches, which should work. We recommend upgrading the kernels on Ubuntu and rebooting the virtual machines. Why is it relevant to me? There are a lot of resources to learn about the vulnerabilities online, the official resource linked above is a good place to start. In short, the vulnerability lets other users on the same machine read memory they should not. Meltdown (variant 3) is the easiest to patch against and you need to patch it on the virtual machine. Spectre (variant 1) has a more limited impact, and the kernel patch mitigates it. Spectre (variant 2) is harder to exploit, but it needs patches from our side. At worst these vulnerabilities can cause leaks of secret keys, passwords and other private information to third parties. Please note that Spectre is hard to completely mitigate against. We expect more variants to be released in time. We are following the situation. What should I do as a customer? First of all, install the latest kernel and reboot your virtual machines. CentOS instructions Installation (note: since autoupdates are enabled, this you might already have packages installed and just need a reboot): sudo -i yum install kernel reboot Verification: dmesg |grep "x86/pti" If you see a line like [ 0.000000] x86/pti: Unmapping kernel while in userspace you have a kernel with the patches. Ubuntu instructions Installation (note: since autoupdates are enabled, this you might already have packages installed and just need a reboot): sudo -i apt update apt install linux-image-generic reboot Verification: dmesg |grep isolation If you see a line like [ 0.000000] Kernel/User page tables isolation: enabled you have a kernel with the patche ## New GPGPU flavors in cPouta DevFeed: [New GPGPU flavors in cPouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2017-09-22T09:38:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cloud-infrastructure](), [Hardware](), [Nvidia](), [PyTorch](), [Deep learning](), [Server]() Tags: [compute](), [cpouta](), [deep-learning](), [hardware](), [nvidia](), [servers](), [tensorflow]() ### AI overview CSC is launching GPGPU flavors on the cPouta cloud platform. The instances use servers with four NVIDIA P100 Pascal cards each, substantial memory, and two SSDs in RAID1. They support GPU-accelerated computing, machine and deep learning, and rendering, with limited initial availability and relatively high billing-unit consumption. ### Source excerpt Today we're launching GPGPU (General Purpose Computing on Graphics Processing Units) flavors on cPouta! The servers have 4 NVIDIA P100 Pascal cards - each! They are backed by tons of memory and two SSDs in RAID1. This ensures that you can get the most out of these power-packed cards. This hardware is being shared with the CSC Super Cluster Taito; we can move GPGPUs between these when we see the need to (hint: contact CSC if there's no free GPGPUs and you need more). We're starting off with only a few GPGPU servers in cPouta while we get a feel for the demand. The details about the new flavors are listed in our user guide at research.csc.fi. Please note that the billing units consumed by these instances is quite significant compared to other flavors. The GPGPUs are great for running things like: High Performance Compute applications leveraging GPUs Machine- and deep learning i.a. TensorFlow Rendering ## Tech preview: Get your own Kubernetes container orchestration platform in cPouta DevFeed: [Tech preview: Get your own Kubernetes container orchestration platform in cPouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2017-08-18T11:26:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Kubernetes](), [Orchestration](), [container](), [Docker]() Tags: [commands](), [container](), [cpouta](), [documentation](), [kubernetes](), [new-feature](), [orchestration](), [preview]() ### AI overview This technology preview explains how users with cPouta access can launch a Kubernetes cluster using Magnum. It introduces container orchestration concepts and points readers to documentation describing the feature and its limitations. ### Source excerpt Introduction You can now launch your own Kubernetes cluster in cPouta with just a few simple commands using a new tool called Magnum. If you already have access to cPouta, it should take about 10 minutes to get up and running. You can find documentation for this new feature in the container orchestration chapter of the Pouta user guide. This is a technology preview feature for now, so there are some limitations that you should be aware of. Those are listed at the end of the documentation page. Why use container orchestration? While Docker containers can be useful on their own, if all you have is a bunch of Linux machines with Docker installed you still have to do a lot of work yourself to make an application fault tolerant, to make it discoverable from the outside world, to make sure the containers stay healthy and to enable containers to communicate with each other. You also have to manage the Linux machines themselves by installing and configuring packages and doing other normal administrative tasks. Container orchestration engines do a lot of this work for you by providing common abstractions around replication, routing, storage and other aspects of running a containerized application. A container orchestration engine typically runs as a clustered application on multiple servers. The servers are separated into two categories: master nodes and slave nodes. The master nodes are responsible for receiving commands via an API and acting on these commands to start and manage containerized applications that run on the slave nodes. Fault tolerance can be achieved by running multiple copies of an application on multiple slave nodes and the COE provides abstractions that make this easy. If you are not already familiar with container orchestration platforms, I recommend looking through the excellent Awesome-Kubernetes list of curated links about the topic. There you will find many introductory articles on Kubernetes, links to various related projects and many other things. ## New I/O flavors in ePouta DevFeed: [New I/O flavors in ePouta]() Original publisher: [Read original article]() Author: Jukka Nousiainen (noreply@blogger.com) Published: 2017-07-07T12:43:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cloud-infrastructure](), [IO](), [Persistence](), [Apache Cassandra](), [datasets](), [Server]() Tags: [cassandra](), [clusters](), [cpouta](), [epouta](), [persistence](), [ssd](), [storage]() ### AI overview The I/O Virtual Machine flavors previously released in cPouta are now available in ePouta. They use SSD-backed striped RAID0 storage, providing high I/O performance without persistence or VM health guarantees if a node or disk fails. They are suited to clustered software such as Hadoop and Cassandra, with anti-affinity server groups recommended for node separation. ### Source excerpt About a year ago we released the I/O Virtual Machine flavors in cPouta. As of July, similar flavors are now also available in ePouta; you can find an exhaustive list here. The I/O flavors are provided with the same disclaimer as with their cPouta counterparts: both the root disk and the ephemeral disk for data storage are hosted on a striped (RAID0) array of SSD disks. That means you get a lot of I/O, but absolutely no guarantees for the persistence or health of VMs in case a node or a disk breaks. Thus these flavors are most natural to use with software which clusters nicely, such as Hadoop or Cassandra. To further ensure each VM in a cluster ends up on separate nodes, we recommend creating an anti-affinity server group. ## cPouta - Now with Haka! DevFeed: [cPouta - Now with Haka!]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2017-02-17T12:32:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Cloud](), [browser](), [API](), [Command-line interface]() Tags: [api](), [browser](), [cloud](), [command-line](), [cpouta](), [web-interface]() ### AI overview The cPouta cloud service now supports logging in with a Haka account through its web interface. Users still need a CSC account and access to a cPouta project, while Haka login is not yet available for API or command-line access. ### Source excerpt Many times we've been asked Can I log in to cPouta using my Haka account? Finally we can answer yes! "How do I log in again?" Not THAT kind of haka. ...but Things are never quite that simple. There are a few restrictions. The most important one is, that you have to have a CSC account and access to a cPouta project to do this. How do you get these? This page should get you started. https://research.csc.fi/accounts-and-projects We still have to keep track of research projects, their users, and their resource usage, which is why you still need a project. But once you have that, point your browser to https://pouta.csc.fi and log in to your heart's content. Please note that Haka logins only work with the web interface for now. We're looking into the option of enabling this for API and command line access in the future. Your CSC Cloud Team ## Storage updates, revisited DevFeed: [Storage updates, revisited]() Original publisher: [Read original article]() Author: Jukka Nousiainen (noreply@blogger.com) Published: 2016-12-13T13:19:00Z Content type: news Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [cloud-infrastructure](), [migration](), [ceph](), [systems]() Tags: [ceph](), [cpouta](), [migration](), [storage](), [systems](), [updates]() ### AI overview The article revisits cPouta storage changes, including the migration of most detached HPC storage volumes to standard Ceph-based volumes. It also announces a December 28-30, 2016 service break, during which affected virtual machines may be forcefully shut down, and explains that Ceph will become cPouta's only storage system. ### Source excerpt Run of the Mill Back in September we blogged about disabling the possibility to create older cPouta volume types. This was the first step on the path to a more streamlined cPouta storage backend. Continuing on this path, we have now migrated most detached HPC storage volumes to standard volumes, and contacted users about their migration options for HPC storage volumes still attached to VMs. Next up, there will be a service break for the whole HPC storage environment, followed by severing the link between cPouta and the HPC storage backend for good. As pictures are often better than a thousand words, we'd like to illustrate these changes for the pleasure of the astute observer. Here's a (very) simplified diagram of cPouta's connectivity to its storage systems: Figure 1. Current topology of cPouta storage backends. The HPC storage system on the left hosts cPouta volumes that are visible to the user as "hpc-storage" or as the dash sign "-". The Ceph-based storage system on the right hosts cPouta volumes that are visible to the user as "standard". Going forward this Ceph-based system will be the only storage system of cPouta. Let it down The next upcoming change potentially affecting cPouta users is the service break for HPC storage systems between December 28th-30th 2016. Components affected by this maintenance are highlighted in Figure 2. Figure 2. Scope of service break on Dec 28th-30th 2016. Maintenance work on the cPouta specific storage controller will occur on December 29th. On this date, any cPouta VMs which still have hpc-storage volumes attached will be forcefully shut down to allow hypervisors to gracefully terminate their iSCSI connections to the HPC storage backend. Apologies about this inconvenience, which is unavoidable with the current firmware/kernel versions. All things must pass Throughout 2016 there has been aggressive growth in the utilization of CSC clouds' volume storage. This has been largely possible thanks to the elasticity of the Ceph-based st