# Allas November 2020 incident details

DevFeed: [Allas November 2020 incident details](<https://devfeed.tech/articles/allas-november-2020-incident-details-19774.md>)

Original publisher: [Read original article](<https://cloud.blog.csc.fi/2020/12/allas-november-2020-incident-details.html>)

Author: Kalle Happonen (noreply@blogger.com)

Published: 2020-12-22T12:31:00Z

Content type: article

Language: en

Sources: [CSC - IT Center For Science - Cloud Team](<https://devfeed.tech/sources/csc-it-center-for-science-cloud-team.md>)

Topics: [ceph](<https://devfeed.tech/topics/ceph.md>), [incident](<https://devfeed.tech/topics/incident.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [rocksdb](<https://devfeed.tech/topics/rocksdb.md>), [NVMe](<https://devfeed.tech/topics/nvme.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [Vapor](<https://devfeed.tech/topics/swift-vapor.md>)

Tags: [ceph](<https://devfeed.tech/tags/ceph.md>), [deep-dive](<https://devfeed.tech/tags/deep-dive.md>), [incident](<https://devfeed.tech/tags/incident.md>), [nvme](<https://devfeed.tech/tags/nvme.md>), [processes](<https://devfeed.tech/tags/processes.md>), [rocksdb](<https://devfeed.tech/tags/rocksdb.md>), [s3](<https://devfeed.tech/tags/s3.md>), [software](<https://devfeed.tech/tags/software.md>), [storage](<https://devfeed.tech/tags/storage.md>), [swift](<https://devfeed.tech/tags/swift.md>), [technical](<https://devfeed.tech/tags/technical.md>)

## AI overview

A technical deep dive into the November 2020 Allas downtime incident. It describes Allas as a Ceph cluster and explains its storage nodes, OSD processes, S3 and SWIFT access, replicated metadata pools, RocksDB metadata, placement groups, and erasure-coded object storage.

## Source excerpt

Allas downtime November 2020 - technical deep-dive We use Ceph (https://ceph.io/) to provide software defined storage in our Allas service. I'll try to speak some Ceph here. There may be a lot of unfamiliar terms for most readers here as we dive deep into a technology. I'll try to clarify where I can, but architectural knowledge of Ceph helps when reading this. Allas is a Ceph cluster with 48 storage nodes (OSD nodes). Each has one nvme, 24 spinning disks, and 192 GiB of memory. Each OSD node runs one OSD process per disk. The Allas storage is provided over SWIFT and S3 protocols through radosgw daemons. The radowsgw layer is provided by separate internet facing servers. In addition we have 5 nodes running monitor processes (they control the state of the cluster), which also run Ceph manager (handles more advanced monitoring views into the cluster state). The object store metadata pools are 3-way replicated pools (data is stored in 3 different copies) on the nvmes across the OSD nodes. This metadata contains e.g. owners of buckets and objects, permissions, versions, etc. Everything that has to do with S3/SWIFT. The nvme also hosts the rocksdb (ceph internal metadata, as opposed to object store metadata) for the OSDs. This metadata is internal Ceph clustster status. Data on placement groups, etc. I'll use the term placement group (or pg) quite a lot in the post as they are central for Ceph. Basically when you store data objects into Ceph, they are mapped into placement groups, which define where in the cluster the data is are stored. The object store data is stored on an 8+3 erasure coded (data is split into 8 chunks + 3 parity chunks) pool on the spinning disks. The total amount of placement groups for the data pool is 16 384. In the graph the amount ~18 000 is shown, as the graph contains other pools too. The beginning Side note: I love that we collect trending data for our cluster (collectd+graphite in our case). As we had the data stored from a long time, this al