# radosgw Published articles for radosgw. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Migrating cPouta Object Storage to the Allas Ceph Service DevFeed: [Migrating cPouta Object Storage to the Allas Ceph Service]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2019-12-27T11:23:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [migration](), [Data Management](), [data](), [cloud-infrastructure]() Tags: [ceph](), [data-management](), [migration](), [object-storage](), [openstack](), [radosgw](), [rgw](), [storage]() ### AI overview This article describes CSC's migration from the cPouta object storage service to Allas, a standalone Ceph-based object storage service. The plan aimed to move user data without requiring user changes or breaking existing data links, using a new Ceph cluster and new data pools. ### Source excerpt We have released our new Allas object storage service. We firmly believe it will play a growing role for data management, at CSC and for the whole academic field in Finland. Alas, the road to Allas was not completely without pain. CSC also hosts the cPouta IaaS service. It provided its own object storage service. Our goal with Allas was to build on this, and transform the object storage portion to a standalone service. This would raise the profile of the service, and make it easier to scale, both human resource wise and where it comes to the platform size. So, in brief, our aim was: Replace cPouta object storage with Allas Make sure all the user data from cPouta object storage moves over. Make sure cPouta object storage users don't have to do any changes. Don't break any existing links to data in cPouta object storage. How did we intend to do this, you may ask? Migration plan What were our options? We had a large bunch of new hardware, and a high level plan. For those of you who know Ceph, you might have already thought of how you'd do this. Add more hardware to the same storage pool, slap on an additional domain name to access the data, and call it a day. Create additional data pools, make sure radosgw uses old and new pools for data, slap on an additional domain name, and call it a day. Create a new Ceph cluster with new pools, migrate the data over, add the old and new domain names, and call it a day. Number 3. is definitely the most work, but we went with that. The main reason for that was that if we want to achieve all the benefits of separating Allas into its own service, we actually need to be a bit separate. For example, the old object storage pools were in the same cluster as the block storage pools used by our OpenStack virtual machines. This not only locked us to the current version of Ceph running there, but created a lot on interdependencies between the services. To make it easier, I'll use the terms rgw-old (cPouta object storage) and rgw-new (Allas) t ## Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository DevFeed: [Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-04-29T09:06:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Git](), [ceph](), [pip](), [Haskell]() Tags: [aws](), [ceph](), [data](), [data-lake](), [git](), [install](), [object-storage](), [python](), [rados](), [radosgw](), [s3](), [storage]() ### AI overview This post explains why binary files should not be stored directly in Git repositories and explores using CSC's Pouta Cloud Object Storage instead. It documents attempts with git-annex and git-bigstore, including a patch to configure a non-default S3-compatible endpoint. ### Source excerpt 1. It's bad to store binary blobs in git. This is known [1] 2. A great place to store binary files is in object storage 3. Can we use CSC's Pouta Cloud Object Storage service for this? Why yes, yes you can! This is a post in a series about different use cases for object storage. Others include for example hosting a reveal.js presentation and NFS server backups More details about CSC's Pouta Cloud Object Storage Service, it is based on CEPH Rados Object Storage Gateway. What The general solution* to storing binaries in git is to not store them in git and instead create a link to some other place Attempt 1 Install an application that you like. In https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories I searched for "S3", found git-annex which had recent commits so it looked like it wasn't dead. Installation instructions: https://git-annex.branchable.com/install/ The version in EPEL/Ubuntu Xenial does not have S3 support and is from 2014. There's a recent one in Fedora's repos though. Unfortunately, compiling it requires cabal. I don't know Haskell. https://spack.readthedocs.io/en/latest/package_list.html does not have any Haskell, cabal or git-annex available. Next. Attempt 2 Next application in the blog post was https://github.com/lionheart/git-bigstore To install it was much more comfortable for me: pip install "git-bigstore<=2.0" First issue was that looking at https://github.com/lionheart/git-bigstore/blob/master/bigstore/backends/s3.py there was no way to tell it to use another endpoint than the default of AWS S3. A small patch takes care of that, add , endpoint_url='https://object.pouta.csc.fi:443' to the client() boto3 call in the backends/s3.py file from git-bigstore. For a more detailed example see this Pull Request. After spending some time going down some rabbit holes it seems that the Profile option that is supported in git-bigstore does not support endpoints. It can only be used to specify credentials and a few other parameters. Mino ## cPouta Cloud Object Storage Service has been updated to run CEPH Luminous! DevFeed: [cPouta Cloud Object Storage Service has been updated to run CEPH Luminous!]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-02-06T10:03:00Z Content type: release Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [Amazon S3](), [Cloud](), [configuration](), [Command-line interface]() Tags: [amazon-s3](), [api](), [automatic](), [ceph](), [cli](), [configuration](), [cpouta](), [format](), [object-storage](), [openstack](), [radosgw](), [s3](), [storage](), [upgrade](), [xml]() ### AI overview The CSC Pouta Cloud Team describes an upgrade of its Ceph Object Storage Gateway from Jewel to Luminous. The update enables AWS4 signatures by default and adds support for several S3 features, including bucket lifecycle, website hosting, and request payment. The article also begins an example of configuring an S3 lifecycle policy with s3cmd and XML. ### Source excerpt We (the CSC Pouta Cloud Team) has finally upgraded CEPH Object Storage Gateway from Jewel to Luminous! There may come more blog posts about other improvements to other of our Openstack services (cinder and glance) which use CEPH. No need to use s3 --signature-v2 anymore! Before this upgrade, anyone using the S3 protocol needed to use the "AWS2" for most actions to work. Now that we run Luminous it is no longer necessary and the default AWS4 signature is working. Yay! In general things should be faster, shinier and also more API calls are supported. Comparing http://docs.ceph.com/docs/luminous/radosgw/s3/ and http://docs.ceph.com/docs/jewel/radosgw/s3/ these features* have gone from unsupported to supported: Bucket Lifecycle Supported Removing expired files is supported Bucket Website Supported Bucket Request Payment Supported Which got me a bit excited! Now to learn a bit about these and see if they work on our setup and if possible if we can make them work without breaking anything for existing objects & users: Bucket Lifecycle This is about expiration/automatic removal of objects at a certain time or after a certain time has passed. Let's start with S3 Some more details about the policies from Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html https://docs.aws.amazon.com/AmazonS3/latest/dev/set-lifecycle-cli.html Telefonica open cloud has some nice documentation with s3cmd examples: https://support.telefonicaopencloud.com/en-us/ugs3cmd/obs/en-us_topic_0051518501.html I have not found anything better on docs.ceph.com -- Step 1. Create a lifecycle policy with s3cmd and in the XML format. (there is a JSON style with awscli and "aws s3api get-bucket-lifecycle-configuration --bucket mybucketname" ): Call the file expiry.xml and make the content something like this: ExampleRule expire_these_ Enabled 1 ) Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-05-09T11:00:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Command-line interface](), [ceph](), [Cloud](), [Amazon Web Services]() Tags: [aws](), [ceph](), [cli](), [demo](), [installation](), [object-storage](), [openstack](), [presentation](), [python](), [radosgw](), [s3](), [storage]() ### AI overview A technical tutorial in the CSC Pouta Cloud Team series explains how to upload a reveal.js presentation to Pouta Object Storage using the awscli tool with Ceph RadosGW's S3-like implementation. It covers URL requirements, credentials, virtual-environment setup, installation, and configuration. ### Source excerpt This is part of a technical series of blog post where the CSC Pouta Cloud Team illustrates with words and sometimes figures how to do things with our object storage. The pevious installation in this series was the magnificent NFS + ZFS + Encryption blog post Topic for this post: we will experiment with Pouta Object Storage, the aws tool and reveal.js to upload a presentation at a nice https://mypresentation20180423.object.pouta.csc.fi/index.html URL. At the time of writing we have not enabled any automatic redirection to index.html with HAProxy if there is no filename specified, so "index.html" is needed or you will see the S3 XML details about the bucket (or AccessDenied). Also https:// at the beginning is needed as we do not yet redirect from TCP 80 to TCP 443. This blog post has two tracks, either you continue with the content on this blog post or you read the content in a reveal.js format on https://mypresentation20180423.object.pouta.csc.fi/index.html Step 1: Configure aws-cli We need something to write files to object storage. We will try the "industry standard" awscli tool to illustrate that it does work even though we are using Ceph's RadosGW which has an s3-like implementation. There are many clients one can use with S3. awscli comes with an s3 client inside which is working with our version for Ceph RadosGW. For some clients we have rudimentary documentation how to use them. You will need OpenStack ec2 credentials to use S3. You get those with this command: $ openstack ec2 credentials create Setting up a python virtual environment and install awscli inside it (so we don't mess with system libraries): $ virtualenv venv_aws $ source venv_aws/bin/activate $ pip install -U setuptools # this was needed on my CentOS7 laptop $ pip install awscli # don't confuse this with "aws" in pip Configuring it: $ aws configure AWS Access Key ID [None]: Enteryouraccesskeyidhere AWS Secret Access Key [None]: Enteryoursecretaccesskeyhere Default region name [None]: # press ENTE ## Customer Invisible Maintenance of the CSC Object Storage Servers DevFeed: [Customer Invisible Maintenance of the CSC Object Storage Servers]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-03-14T07:36:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [BGP](), [networking](), [Load Balancing](), [Availability](), [round robin]() Tags: [bgp](), [ceph](), [cpouta](), [dns](), [exabgp](), [failover](), [ha](), [high-availability](), [load-balancing](), [maintenance](), [network](), [networking](), [object-storage](), [radosgw](), [round-robin](), [routing](), [server]() ### AI overview The CSC Pouta Cloud Team describes using ExaBGP on object storage servers to combine round-robin DNS load balancing with BGP routing for high availability and failover during maintenance. ### Source excerpt Hello! It's the CSC Pouta Cloud Team again! You may recognize us from other object storage films such as: http://pouta.blog.csc.fi/2018/02/admin-stories-implement-object-storage.html where we go through in excruciating detail how we configured CEPH RadosGW and why we configured it like we did http://pouta.blog.csc.fi/2018/01/objectively-speaking-in-cpouta.html where we announced object storage! In this episode we're going to talk a bit about how we implemented invisible(*) maintenance for our Object Storage customers. Configuration of ExaBGP with Object Storage CSC - IT Center for Science Ltd. has a close collaboration with the Finnish University and Research Network - FUNET, and because of that we are in a great position to do some fun network stuff. Actually, as long as your network has a BGP relationship with its internet provider you can also do what we have done. When we first launched Object Storage we just had a DNS entry pointing to two IP addresses. This gave us some load balancing and if we really wanted to we could take a node out of the DNS and do maintenance. But we wanted more! *ExaBGP enters the stage* In short we installed ExaBGP on our object storage linux servers, configured one dummy network interface per server and started to announce two IP addresses from each server. Then we pointed object.pouta.csc.fi DNS entry to these two IP addresses. The objective was to use round-robin DNS for load-balancing and BGP routing for High Availability. For failover this means that if one node goes down, the other IP is reachable on the other node and stuff will keep magically(*) working! There are some other BGP softwares out there, like BIRD or Quagga. Despite this, we never actually tried any other software since ExaBG has some great qualities such as: it came recommended by the network team at CSC who uses it for some services its low resource footprint and small number of software dependencies it is not a full routing suite - it is meant to be used for basi ## Admin Stories: Implement Object Storage in CSC's cPouta DevFeed: [Admin Stories: Implement Object Storage in CSC's cPouta]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2018-02-12T12:58:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [Amazon S3](), [Cloud](), [API]() Tags: [backend](), [ceph](), [cpouta](), [infrastructure](), [object](), [object-storage](), [openstack](), [rados](), [radosgw](), [rgw](), [s3](), [server](), [servers](), [storage](), [volume]() ### AI overview This article explains the decisions behind implementing object storage in CSC's cPouta cloud. It describes using Ceph RADOS Gateway with existing Ceph clusters and clarifies the relationship between S3 buckets, Swift containers, accounts, and OpenStack project IDs. ### Source excerpt In our previous user survey we found out that you were interested in "OpenStack admin stories". This will be our first attempt in doing that. User guide: https://research.csc.fi/pouta-user-guide While implementing object storage we made a few decisions. This blog post is about highlighting the decisions and their backing thought process. Some background information: cPouta uses OpenStack as the underlying cloud middleware, and Ceph for storing volumes and images. And going forward, objects, too. Terminology RADOS Gateway/RadosGW/RGW/Ceph RGW/Ceph radosgw/radosgw. We mean the same thing. It's a piece of software which exposes an API for storing and retrieving objects. Ceph - the storage system which RadosGW daemons access as a client. Buckets in S3 and Containers in Swift are synonymous. The Ceph radosgw-admin tool also operates on buckets. The latter are mostly the same, but can contain a bit more information i.e. OpenStack project ID. If we write about buckets, the reader can presume we are referring to S3 buckets and Swift containers unless otherwise noted. Account is something that RGW binds usage data into. In some of the radosgw-admin CLI calls, accounts are referred to as users or UIDs. These users and/or accounts get a mapping to OpenStack project IDs in our configuration. So be it account/user/project, we are always talking about the same thing. - Why Ceph RadosGW The other option we briefly looked at for an Object Storage server was the OpenStack Swift server. Yes, there is an API called Swift and a piece of sofware called Swift which serves the Swift API. Not to mention the programming language Swift, the ISO standard SWIFT or the Scottish potato variant Swift. In the end we used none of these for backend because we prefer to reuse existing infrastructure. We already have Ceph clusters used for Virtual Machine image and block (volume) storage in our clouds. Installing some Ceph RadosGW servers and pointing those to an existing Ceph cluster utilizes a lot o