# ceph Published articles for ceph. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## I Turned My Proxmox Server Into a NAS Without Installing TrueNAS DevFeed: [I Turned My Proxmox Server Into a NAS Without Installing TrueNAS]() Original publisher: [Read original article]() Author: Brandon Lee Published: 2026-09-10T12:46:18Z Content type: article Language: en Sources: [Virtualization Howto]() Topics: [Proxmox](), [Homelab](), [virtualization](), [hosting](), [Linux](), [Debian]() Tags: [ceph](), [debian](), [home-lab](), [home-server](), [hosting](), [linux](), [nas](), [proxmox](), [self-hosted](), [self-hosting](), [server](), [storage](), [virtualization]() ### AI overview The article presents ANAS, a project that adds native storage-management panels to the existing Proxmox VE web interface. It aims to let Proxmox manage storage using the underlying Debian/Linux capabilities without installing TrueNAS or introducing a separate management interface. ### Source excerpt Proxmox is really good with all types of storage and I have used it as a dedicated NAS appliance many times. I have used dedicated NAS devices. I have also... The post I Turned My Proxmox Server Into a NAS Without Installing TrueNAS appeared first on Virtualization Howto. ## Self-Hosted Platform Build Order: Dependencies from Bare Metal to Model Serving DevFeed: [Self-Hosted Platform Build Order: Dependencies from Bare Metal to Model Serving]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-09-04T07:00:00Z Content type: article Language: en Sources: [Philipp Lehmann]() Topics: [Self-hosted](), [Ansible](), [Kubernetes](), [ceph](), [observability](), [model-serving](), [Compose]() Tags: [ansible](), [architecture](), [article](), [ceph](), [compose](), [infrastructure-as-code](), [kubernetes](), [model-serving](), [observability](), [self-hosted]() ### AI overview This article explains the dependency order for building a self-hosted platform. It covers consistent bare-metal inventory, Ansible configuration convergence, networking, Kubernetes, Ceph storage, identity, observability, and model serving. ### Source excerpt Self-hosted platform build order: why mesh, cluster, Ceph storage, identity and observability must precede model serving, shown with Compose depends_on. ## Recover the Ceph Monitor Store Using OSDs DevFeed: [Recover the Ceph Monitor Store Using OSDs]() Original publisher: [Read original article]() Author: Eugen Block, crossposted by Anthony D'Atri Published: 2026-07-04T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [ceph](), [monitor](), [Script](), [Containers](), [log management]() Tags: [blog](), [blog-post](), [ceph](), [containers](), [en-article](), [en-blog-post](), [how-to](), [logging](), [logs](), [mon](), [monitor](), [recovery]() ### AI overview This blog post presents a general procedure for recovering a failed Ceph Monitor store by collecting osdmaps from existing OSDs, with additional considerations for cephadm-managed container deployments. It automates osdmap collection and basic logging while leaving the store rebuild and client authentication steps for careful manual inspection. The procedure applies only to non-encrypted OSDs, and the article emphasizes the importance of backing up dmcrypt/LUKS keys and reviewing host logs. ### Source excerpt Introduction ¶ A few weeks ago I helped a Ceph user to recover his broken cluster (see this thread). Basically, after his Monitors stopped working he re-deployed a new cluster with the same Ceph FSID and attached the existing OSDs to the re-deployed hosts. But it's not that easy to re-activate those OSDs because the new Monitors don't have the old osdmap, hence they don't know anything about the existing OSDs. So how did we fix that? This scenario can be considered a total Monitor store failure. There's a documented procedure in the upstream docs, unfortunately it's written for non-cephadm clusters and doesn't contain many details about subsequent steps. When Ceph daemons run within containers there are more things to consider. So I decided to write this blog post and add some more details to the procedure, targeting mainly clusters managed by cephadm. But the recovery procedure doesn't specifically require cephadm-specific commands, so it can be considered as a general guideline how to recover from a Monitor store loss, just with some extra details about cephadm deployments. This procedure only works for non-encrypted OSDs. If you are using dmcrypt (LUKS) OSDs, make sure you have a backup of the keys, otherwise your data is lost forever! But there is some development, a user in Slack pointed me to this PR which adds a backup mechanism for Monitors, probably available in the upcoming Umbrella release. Note that the PR also contains this statement: Monitor backups complement, but do not replace, the existing Monitor recovery procedures The procedure covered by said docs is already written in script form, so I used that as a template and extended it for cephadm usage and included some very basic logging. It collects the osdmaps from all OSDs and contains the necessary considerations regarding containers. I decided to automate only the osdmap collection, not all required steps of the store rebuild procedure (e. g. Monitor store rebuild, client auth, etc.) because the r ## Reclaim Ceph Capacity Through CephFS Transcoding DevFeed: [Reclaim Ceph Capacity Through CephFS Transcoding]() Original publisher: [Read original article]() Author: Anthony D'Atri Published: 2026-07-01T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Transcodings](), [Software](), [data]() Tags: [2026](), [4k](), [article](), [blog-post](), [ceph](), [cephfs](), [dram](), [efficiency](), [en-article](), [en-blog-post](), [enterprise](), [enterprise-storage](), [filesystem](), [fujitsu](), [git](), [hardware](), [performance](), [software](), [space](), [storage](), [tentacle]() ### AI overview This article addresses rising storage demands and hardware costs by discussing CephFS capacity efficiency. It describes replicated pools, Erasure Coding, and Fast EC in Ceph Tentacle, while the title identifies CephFS transcoding as the article's focus. ### Source excerpt Data expands to fill available storage (and beyond)! ¶ It used to be that enterprise storage meant 6RU rackmount Fujitsu 2351 Eagles, each holding a mind-boggling 380 MiB of data: enough for a whole company! Today that 380 MiB can't even hold a 4k pickleball video. Enterprises, educational instutitions, and really just about anyone these days demand storage capacities that start on the order of hundreds of tebibytes and rapidly grow to pebibytes. As this article is written in the spring of 2026, the memory market, which includes DRAM, SSDs, and legacy HDDs, has experienced a dramatic escalation of pricing. It is not uncommon to be quoted a price four times what the same hardware cost a year ago, and there are signs that it is going to get worse before it gets better. What's a poor ammonite to do?? Cephers find themselves between the Charybdis of quotes approaching Disaster Area's hypermathematics and the Scylla of hungry users armed with torches and git forks. Git forks, pitchforks. Get it? Sigh. Tough room. Anyway... Short of nuking the site from orbit, how do we make everyone happy, or at worst mildly discontented? Efficiency! CephFS ¶ CephFS is a popular, highly available and scalable software-defined POSIX-style distributed filesystem that can easily store tens of pebibytes of precious data. Or, alternately, cat videos. Ceph deployments often begin small, with replicated pools for perceived performance needs. As the cluster grows to more nodes and more data, it may become feasible and desirable to switch to Erasure Coding (EC) to make more efficient use of raw capacity. An EC pool thus can require substantially less raw storage for a given amount of user data, or store gobs more user data on a given amount of raw capacity This EC overhead table presents efficiency (space amplification) factors for a spectrum of EC profiles. Replicated pools usually maintain three copies of data, so for comparison they manifest an overhead factor of 3.0. EC 4+2 or 6+3 presents a ## Ceph Q2 2026 Newsletter DevFeed: [Ceph Q2 2026 Newsletter]() Original publisher: [Read original article]() Author: Anthony Middleton Published: 2026-06-25T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [ceph](), [releases]() Tags: [2026](), [blog-post](), [ceph](), [ceph-events](), [collaboration](), [communication](), [community](), [contribute](), [developers](), [documentation](), [ecosystem](), [en-article](), [en-blog-post](), [event](), [governance](), [growth](), [metrics](), [newsletter](), [operational](), [operations](), [releases](), [resources](), [support]() ### AI overview The Ceph Q2 2026 Newsletter covers governance coordination, community elections, documentation initiatives, contributor resources, regional Ceph Days events, project releases, and ways to contribute. It emphasizes collaboration among users, developers, operators, and organizations across the Ceph ecosystem. ### Source excerpt During the second quarter, the Ceph Foundation focused on strengthening collaboration across the project, expanding community programs, supporting documentation efforts, and investing in regional events. This edition highlights recent governance discussions, community elections, new contributor resources, Ceph Days events, and opportunities for community members to get involved. If you have an idea for a Ceph-related event, outreach effort, or project, we encourage you to submit a funding request and work with the Foundation to help bring it to life. In This Issue ¶ Board and CSC Alignment on Community Priorities 2026 CSC Election Recap Meet the New Ceph Technical Writer Ceph Days Continue to Grow Latest Ceph Releases Community Committees and New Ways to Contribute Board and CSC Alignment on Community Priorities ¶ The Ceph Governing Board and Community Steering Committee (CSC) met this quarter to discuss the long-term health of the project, governance, documentation, community growth, and Foundation-supported initiatives. The discussion reaffirmed the distinct roles of each organization within the Ceph ecosystem. The CSC continues to provide technical governance for the project, while the Ceph Foundation focuses on community growth, events, outreach, and funding. The Linux Foundation provides operational, legal, and financial support that enables both groups to operate effectively. A significant area of focus was improving coordination between technical priorities and Foundation-supported programs. Current documentation initiatives include improvements to the documentation backport process, development of a new OSD operations guide, clarification of package and container documentation, and the removal of outdated content. The Board and CSC also discussed the future of community events and the importance of establishing clear goals and success metrics for Foundation-funded programs. Both groups reaffirmed their commitment to supporting collaboration between users, de ## v20.2.2 Tentacle released DevFeed: [v20.2.2 Tentacle released]() Original publisher: [Read original article]() Author: Yuri Weinstein, Patrick Donnelly Published: 2026-06-16T00:00:00Z Content type: release Language: en Sources: [Ceph Blog]() Topics: [bug](), [Vulnerabilities](), [Memory Leaks](), [Encryption](), [configuration](), [IO]() Tags: [2026](), [aio](), [blog-post](), [bug](), [ceph](), [configuration](), [en-article](), [en-blog-post](), [encryption](), [io](), [memory-leaks](), [rados](), [release](), [rgw](), [storage](), [tentacle](), [ui](), [update](), [vulnerabilities]() ### AI overview Ceph v20.2.2 Tentacle is a minor release that adds Rocky 10 package-install support and recommends updating. It fixes issues across MDS, OSD, RGW, RADOS libraries, watch/notify operations, asynchronous utilities, and the dashboard, including security vulnerabilities, memory leaks, encrypted-object lifecycle problems, and UI and configuration improvements. ### Source excerpt This is the second minor release in the Tentacle series. We recommend that all users update to this release. Release Date ¶ June 16, 2026 Notable Changes ¶ Rocky 10 package-based installs are now supported starting with v20.2.2. Please see the supported platforms for current and planned support in Ceph. MDS (Metadata Server) ¶ Segmentation fault fixed due to incorrect queueing of request retries. OSD (Object Storage Daemon) ¶ PGLog Missed List: Fixed a bug to ensure the correct version is attached to the missing list when ignoring log entries. Data Integrity Asserts: Added assertions to explicitly catch potential corruption in the OSD missing list. RGW (RADOS Gateway) ¶ Lifecycle Management: Fixed lifecycle transition issues affecting encrypted multipart objects. REST & Query Handling: RESTArgs::get_string() now properly URL-decodes incoming query parameters. RADOS / librados / neorados ¶ Linger Operations: Rewrote safety checks to eliminate use-after-free vulnerabilities and LingerOp memory leaks when an unwatch operation returns ENOTCONN. Replaced the flawed is_valid_watch() check with a safe linger_by_cookie() lookup which safely manages LingerOp references using intrusive_ptr. Ensured librados linger callbacks hold a persistent reference to LingerOp to protect against races with simultaneous linger_cancel() requests. Configured librados::IoCtxImpl::aio_unwatch() to asynchronously deliver ENOTCONN to AioCompletion instead of returning the error directly. Watch/Notify: Fixed neorados notification queue bounds enforcement so that an overflow marker is only appended on the first message exceeding capacity, rather than duplicating it on every subsequent message. Prevented double-cleanup triggers in watch/notify operations when incoming errors arrive after maybe_cleanup() runs. Fixed a bug where notify would lose original error values by ensuring it no longer attempts to decode empty responses. Enhanced io_context shutdown procedures to clear handlers and route cleanl ## Ceph Days Seattle 2026 - Cloud Transition to Tape DevFeed: [Ceph Days Seattle 2026 - Cloud Transition to Tape]() Original publisher: [Read original article]() Author: John Shubeck Published: 2026-06-15T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Open Source](), [Amazon S3](), [ibm](), [Cloud](), [Software]() Tags: [2026](), [blog-post](), [ceph](), [cloud](), [conferences](), [developers](), [en-article](), [en-blog-post](), [ibm](), [network](), [object-storage](), [open-source](), [s3](), [seattle](), [storage]() ### AI overview This article recaps Ceph Days Seattle 2026, a community-driven event about open-source storage. It introduces Ceph as a software-defined platform for block, file, and object storage, and highlights a presentation on transitioning Ceph cloud workloads to S3-enabled tape. ### Source excerpt I'm John Shubeck, an information technology professional with over 44 years of industry experience spanning both the customer and technology provider experience. I'm currently serving as a Senior Storage Technical Specialist for IBM Object Storage platforms across all market segments in the Americas. On Thursday, May 28th, 30 members, myself included, of the Ceph faithful descended on the SURF business incubator in downtown Seattle for "Ceph Days Seattle 2026". Following the traditional Ceph Days agenda, the Seattle program consisted of a series of "Ceph Talks" designed to introduce new ideas and share lessons learned. Ceph Days are community-driven events designed to bring together Ceph users, developers, architects, administrators, and anyone interested in open-source storage. Held throughout the year in locations around the world, these one-day gatherings offer a unique opportunity to learn from real-world deployments, explore new technologies, connect with peers, and contribute to discussions that help shape the future of Ceph. Did you know? ¶ Ceph is a software-defined, hardware-independent storage solution that provides block, file, and object storage on a single unified platform. Ceph is championed by developers, administrators, users, IT leaders, and Fortune 500 enterprise customers. It is a living, vibrant, and active group that has adopted Ceph in its IT operations. The Ceph project and community are supported by a Foundation (https://ceph.io/en/foundation/) comprising organizations, stakeholders, and industry leaders who collaborate to coordinate investment, development, and community activities for Ceph. With the setting, plot, and characters stated above, we turn to today's topic. A key activity of the Ceph community is a series of "Ceph Days" conferences. A Ceph Day is a full-day meetup where participants learn, network, make new acquaintances, and share ideas on how to enhance the value of Ceph. Sometimes IBM participates by providing the venue, a gue ## For whom the door-bell tolls DevFeed: [For whom the door-bell tolls]() Original publisher: [Read original article]() Author: Kyle Bader Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Caching](), [Large Language Model](), [Inference Performance](), [Low-Latency Inference](), [GPU](), [cpu]() Tags: [2025](), [ai](), [architecture](), [blog-post](), [cache](), [caching](), [ceph](), [cpu](), [en-article](), [en-blog-post](), [gpu](), [gpu-initated](), [inference](), [llm](), [nvme](), [rados](), [rados-nkv](), [storage]() ### AI overview The article explores using Ceph RADOS through NVMe key-value commands to support GPU-initiated storage access for LLM KV caching. It connects content-addressable caching with the efficiency of GPU-initiated NVMe access and discusses a possible vendor-specific extension for executing classes against objects. ### Source excerpt For whom the door-bell tolls In a previous post we extolled the benefits of KV caching, a technique to save the KV states from the prefill step of LLM-based inference to reduce time to first token (TTFT) and skip redundant computation. I co-presented this with Tushar Gohad at Cephalocon. Since then I've been thinking a lot about how to improve the state of the art. Really move the needle. We've made strides in a lot of areas in Ceph, especially the work going into Fast EC -- if you don't know what I'm talking about you should check it out, it promises huge benefits to a large category of workloads. That's not what we're here for today, though. Last year there was one paper that stuck in my mind, which is fairly remarkable because I read on the order of 130. That paper was GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture. I struggled with this. It describes a system where a CPU loads a kernel into the GPU that allows the GPU to serve as an NVMe initiator. The struggle was rooted in the fact that block just felt like the wrong interface for KV caching. If you use block, then you need a lookup table that maps the hash of the sequences representing a cache block to a particular (device, offset, length) tuple. It begs for a content-addressable approach with no centralized lookup or coordination. On the other hand, what was described in the paper was just flat-out more electrically efficient. I couldn't have my cake and eat it too. The idea: RADOS, spoken as NVMe key-value ¶ In 2025 we also saw the first ratified version of the NVMe key-value command set specification. We already have a Ceph implementation of NVMe/TCP that leverages SPDK, and it seemed like we could add support fairly easily for the key-value command set. For those who don't know the internal mechanics of Ceph, the native API operates against RADOS objects. RADOS is richer than most object stores: it supports reads and writes to arbitrary offsets, deletes, key-value sto ## Managing Multiple Lua Scripts with Ceph Object Storage DevFeed: [Managing Multiple Lua Scripts with Ceph Object Storage]() Original publisher: [Read original article]() Author: Kirby Chin Published: 2026-06-10T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Lua](), [Script](), [Scripting](), [Security](), [Command-line interface](), [cloud-infrastructure](), [ransomware]() Tags: [availability](), [blog](), [blog-post](), [ceph](), [cli](), [data](), [developer](), [en-article](), [en-blog-post](), [examples](), [feature](), [go](), [lua](), [management](), [new-feature](), [object-storage](), [operations](), [rados](), [ransomware](), [rgw](), [s3](), [scripting](), [security](), [storage]() ### AI overview This article introduces a Ceph RADOS Gateway feature for managing multiple Lua scripts within the same request context and tenant. It demonstrates scripts for enforcing bucket object locks to mitigate ransomware risks and for optimizing storage through object auto-tiering. ### Source excerpt Since the Pacific release, Lua scripting in Ceph's RADOS Gateway (RGW) has provided users the ability to interpolate a single script to upload operations per request context and tenant. This way of working might be completely fine for a storage deployment with limited scripting customizations. However, script management becomes increasingly difficult as more than one team wants to get involved in managing the Lua script within the same context and tenant. For this reason, we've released a new feature in RGW allowing you to manage more than one Lua script at a time. In this blog, we'll go over a couple of examples to walk you through how this feature can help your team to reduce runtime errors and lower developer friction when managing Lua scripts in RGW. To begin, we will create a script to enforce a bucket security control and then add another script to demonstrate the new feature. Script 1: Bucket security ¶ Let's suppose we are part of a security team and want to mitigate the risk of ransomware threats happening on our storage platform. One possible way to achieve this is to implement a Write Once, Read Many (WORM) strategy to add an object lock onto any newly created bucket. By using an object lock, we can add a constraint to write objects to disk only once, ensuring that an infected client cannot delete or override objects at a later time. To set our WORM strategy, we can create a Lua script in the prerequest context that aborts when a create_bucket operation is made without the write-once (object lock) requirement. objectlock.lua -- enforcing object lock on bucket creation if Request.RGWOp == "create_bucket" and Request.HTTP.Metadata["x-amz-bucket-object-lock-enabled"] ~= "true" then RGWDebugLog("object lock is missing on bucket: " .. Request.Bucket.Name) Request.Response.Message = "Bucket must have object lock enabled" return RGW_ABORT_REQUEST end You can create a new objectlock.lua file with the contents above and run the CLI commands below to upload the scr ## From Laminated Pages to Petabytes: Why I'm Joining the Ceph Foundation DevFeed: [From Laminated Pages to Petabytes: Why I'm Joining the Ceph Foundation]() Original publisher: [Read original article]() Author: Emmanuel Ameh Published: 2026-05-06T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Usability](), [Software](), [systems](), [incident](), [Disk image](), [CRUD]() Tags: [blog-post](), [ceph](), [community](), [developer](), [documentation](), [en-article](), [en-blog-post](), [guide](), [how-to](), [incident](), [logging](), [production](), [software](), [storage](), [systems](), [technical](), [usability](), [validation](), [workflow]() ### AI overview A new Ceph Foundation technical writer explains a documentation philosophy centered on usability and practical outcomes. The article introduces test-driven documentation: validate commands and workflows in clean environments, log user friction, and organize information so urgent how-to guidance is easy to find while deeper explanations remain accessible. ### Source excerpt Hello, Ceph Community! I'm Emmanuel Ameh, and I'm thrilled to be joining as your new Technical Writer. My path into technical writing started during a university internship at a manufacturing plant, where I was handed a problem that, looking back, shaped how I think about documentation to this day. The first piece of documentation I wrote was a one-page troubleshooting guide. It was a simple list of common production-line issues and their fixes, taped to a workstation. But it worked. Operators stopped escalating the same problems, uptime held steady, and that single laminated page did the job of a dozen phone calls. I think about that page a lot. It wasn't elegant. It wasn't comprehensive. It didn't win any awards. But it sat exactly where the people who needed it could find it, and it told them exactly what to do. That, to me, is what good documentation looks like, whether it's protecting a production line or a petabyte-scale storage cluster. That experience taught me my core philosophy: good documentation isn't measured by how thorough it looks. It's measured by what it lets people do. A doc that keeps a cluster running is worth more than a hundred pages nobody reads. That belief is what brings me to the Ceph community today. My Approach: Test-Driven Documentation ¶ My background is in enterprise software and developer advocacy, where I've focused on making complex systems feel approachable. I treat usability as seriously as technical accuracy. To me, a "perfectly correct" doc that a user can't follow is a broken feature. I advocate for test-driven documentation. In practice, this means: Validation: Running every command and workflow in a clean environment to ensure it actually works. If a tutorial says "run this, and you'll see X," then somewhere, on a fresh install, I should be able to run it and see X. If I can't, neither can the reader. Friction Logging: Identifying the exact moment a newcomer gets stuck and smoothing that path. The most valuable feedback I ca ## Ceph Q1 2026 Newsletter DevFeed: [Ceph Q1 2026 Newsletter]() Original publisher: [Read original article]() Author: Anthony Middleton Published: 2026-03-31T00:00:00Z Content type: news Language: en Sources: [Ceph Blog]() Topics: [Developer experience](), [Slack](), [ci]() Tags: [2026](), [blog-post](), [ceph](), [ceph-events](), [ci](), [communication](), [community](), [complexity](), [contribute](), [contributors](), [developer-experience](), [ecosystem](), [en-article](), [en-blog-post](), [event](), [events](), [governance](), [guide](), [onboarding](), [slack]() ### AI overview The Ceph Q1 2026 Newsletter covers the Ceph Foundation's governance, event planning, financial updates, community funding opportunities, and communication initiatives. It also reports on efforts to improve developer and contributor experience by addressing operational complexity, CI complexity, review capacity, onboarding, and communication between the Foundation Board and the Ceph Steering Committee. ### Source excerpt During this quarter, the Ceph Foundation focused on strengthening its structure by establishing new governance charters, event strategies, and financial plans. To enhance transparency as our community evolves, this newsletter offers a look behind the scenes at these foundational details. Our aim is to update Ceph Community members on the latest developments within the Foundation and to clarify how they can contribute to our ongoing growth. There are several ways to get involved with the foundation. If you have a concept for a Ceph-related project, we encourage you to take the first step toward bringing your idea to the next level and submit a funding request whenever you are ready. Feedback and suggestions will be offered along your journey with Ceph. In this issue ¶ CSC and Ceph Foundation Board Meeting Ceph Foundation Charters Approved Ceph Governing Board Hiring a Technical Writer OVHcloud Spending Update Ceph Tech Talks Are Back (Monthly Schedule) Upcoming Ceph Days Events Ceph Community Slack Upgraded to Pro CSC and Ceph Foundation Board Meeting ¶ The Ceph Foundation Board recently hosted the Ceph Steering Committee (CSC) for a collaborative discussion on the current state of the Ceph project and how both sides can work together to support the Ceph community. These quarterly meetings are designed to foster communication between the two committees working to build Ceph for the benefit of its users and contributors. The Ceph Board's goal is to help provide greater context to the CSC as they make decisions and to support their missions, thereby bridging the communication gap. The meeting's agenda is available here. Key discussion areas ¶ Operational complexity Friction in contributing and getting reviews Fragmented communication Unclear strategy in some areas Unclear ownership across parts of the ecosystem Major takeaways ¶ The Board will continue to work on closing the gap between developer experience and real-world operator needs, with help from the CSC around f ## Assessing the performance of the CLAY Erasure Code Plugin DevFeed: [Assessing the performance of the CLAY Erasure Code Plugin]() Original publisher: [Read original article]() Author: Jake Squelch (IBM) Published: 2026-02-11T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [benchmarking](), [IO](), [Code]() Tags: [4k](), [benchmark](), [benchmarking](), [benchmarks](), [blog-post](), [ceph](), [en-article](), [en-blog-post](), [io](), [performance]() ### AI overview This article assesses the performance of the CLAY Erasure Code Plugin using CBT benchmarks. It reports that CLAY and JErasure have practically identical read performance for 4K random reads and 1024K sequential reads, while CLAY has substantially lower write performance, especially at higher bandwidths. The article attributes this to CLAY's more complex encoding process, which performs two encoding stages plus an additional memory copy. It also reports worse performance with an OSD down, including increased latency for degraded reads, but the supplied text ends before the discussion is complete. ### Source excerpt CBT Performance Benchmarking - Part 4. What can we say about CLAY? Outline of the Blog Series ¶ Part 1 - How to start a Ceph cluster for a performance benchmark with CBT Part 2 - Defining YAML contents Part 3 - How to start a CBT performance benchmark Part 4 - Assessing the performance of the CLAY erasure code plugin Contents: Client IO results for CLAY Client IO with an OSD down What is CLAY good at? Problems with using CLAY How does CLAY read data from the drive? CLAY is broken in tentacle Summary Client IO results for CLAY ¶ As a refresher lets quickly look back on the client IO results of CLAY compared to JErasure: If we look back to Step 3 in Part 3 of the blog (Generating a comparison report), we saw that reads had practically identical curves between CLAY & JErasure for both 4K random reads and 1024K sequential reads. However, when we compared writes we saw that the performance hit to CLAY was substantially larger, particularly for higher bandwidths. The 1024k Sequential Writes diagram represents this: Click to see Part 3 diagrams So why was this? This is because of CLAY's encoding process, it is significantly more complex. While JErasure performs a single encoding pass, CLAY uses three phases: 50% of data is encoded using PRT (Product Recovery Transform), 50% of the data is copied to form an intermediate set of buffers All the intermediate data is encoded using RS (Reed-Solomon) to form a second set of intermediate buffers 50% of the result is encoded using PFT (Parity Fractional Transform), 50% of the data is copied to form the output buffers Essentially, CLAY performs 2x the encoding plus an additional memcpy (memory copy) compared to JErasure's 1x encoding. This overhead therefore directly translates to lower write throughput for CLAY, as shown by the diagrams above. The performance impact increases for larger IO sizes because more data is being encoded. Referenced the following: 'Clay Codes: Moulding MDS Codes to Yield an MSR Code' above for information ## RGW Bucket Resharding Without Pausing DevFeed: [RGW Bucket Resharding Without Pausing]() Original publisher: [Read original article]() Author: Daniel Alexander Parkes, Anthony D'Atri Published: 2026-02-01T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [systems](), [Scalability](), [Architecture & Design](), [cloud-infrastructure]() Tags: [2026](), [article](), [blog-post](), [ceph](), [concurrent](), [deep-dive](), [en-article](), [en-blog-post](), [io](), [operational](), [operations](), [performance](), [production](), [rgw](), [s3](), [scalability](), [scale](), [storage](), [systems]() ### AI overview The article explains how Ceph Tentacle introduces near-zero-impact RGW bucket resharding. Its two-phase architecture moves intensive work into a non-blocking background phase, avoiding the prolonged write unavailability and 504 errors seen in Ceph Squid and earlier versions. ### Source excerpt Introduction: The Foundation of Scalable Object Storage ¶ In the modern data landscape, object storage has evolved from a simple file repository into the foundational layer for AI/ML pipelines, data lakehouses, real-time analytics, and massive-scale archival systems. At the heart of this evolution is a deceptively simple question: How do you efficiently locate and access billions of objects stored in a single bucket? The answer lies in one of Ceph's most critical performance mechanisms: bucket index sharding. This architectural pattern divides a bucket's index into multiple parallel structures, enabling concurrent operations across thousands of objects while maintaining the consistency and reliability that enterprise workloads demand. But there's always been a catch. As workloads grow and evolve, buckets need to be resharded. Historically, when the buckets to be resharded had a vast number of objects, this operation came with a painful trade-off: blocking client writes from seconds to minutes, with a chance of causing application disruptions, 504 Gateway errors, and operational headaches. With Ceph Tentacle, we're eliminating this trade-off. The new near-zero impact bucket resharding architecture transforms what was once a maintenance window event into a seamless background operation that your applications will never notice. Note: As of 2026/02/05, the functionality described in this article is expected in an upcoming Tentacle update. Executive Summary ¶ The Challenge: In Ceph Squid, resharding a 20-million-object bucket blocked writes for 4+ minutes, returning 504 errors. Even larger buckets (500M objects) required 94 minutes of complete write unavailability. The Solution: Ceph Tentacle's two-phase architecture moves the heavy lifting to a non-blocking background phase, eliminating the impact on clients IO. The Results: (note: in this graphic 8.1 refers to Squid and 9.0 to Tentacle) In this deep dive, we'll explore: Why bucket sharding is essential for modern workl ## Mastering IAM in Ceph: Multi-Tenancy, Access Control, and Why ACLs Must Die DevFeed: [Mastering IAM in Ceph: Multi-Tenancy, Access Control, and Why ACLs Must Die]() Original publisher: [Read original article]() Author: Daniel Alexander Parkes, Anthony D'Atri Published: 2026-01-24T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [IAM](), [Authorization](), [Security](), [Multi-tenancy](), [Amazon S3](), [Amazon Web Services](), [Architecture & Design]() Tags: [access-control](), [aws](), [blog-post](), [ceph](), [defense-in-depth](), [en-article](), [en-blog-post](), [iam](), [identity](), [multi-tenancy](), [operations](), [production](), [rgw](), [s3](), [security]() ### AI overview This article explains why ACLs in Ceph Object Gateway should be disabled in favor of centralized IAM policies. It discusses S3 permission failures, multi-tenant access problems, AWS's ACL-disabled defaults, and defense-in-depth measures such as Block Public Access and explicit denies for ACL operations. ### Source excerpt Introduction ¶ Introduction: When Security Theater Becomes a Real Disaster ¶ In March 2017, a misconfigured S3 bucket at Verizon exposed the personal information of 14 million customers. The root cause wasn't a sophisticated attack; it was a simple oversight in access permissions. The bucket was set to be publicly accessible due to S3 permission misconfiguration, and no one noticed because ACLs were managed separately from the company's centralized IAM policies. The security team had implemented careful, identity-based access controls, but a resource-level ACL silently bypassed them by granting access to "All Users." This scenario repeats constantly across the industry: ACLs creating invisible access paths that security teams don't know exist, buckets accidentally exposed to the public internet, and contractors uploading data that the bucket owner cannot reliably read or administer, while still consuming capacity. Between 2017 and 2019, major companies exposed hundreds of millions of records via misconfigured S3 permissions (ACLs and/or bucket policies): Verizon (2017): 14 million customers - An AWS S3 bucket configured for public access exposed names, addresses, account PINs Facebook (2019): 540 million records - Third-party apps stored user data in publicly accessible S3 buckets Instagram (2019): 49 million records - Marketing firm left influencer database unprotected in AWS S3 The AWS response was clear: since April 2023, all new S3 buckets default to "ACLs disabled" (BucketOwnerEnforced) and Block Public Access enabled. AWS strongly recommends disabling ACLs on existing buckets and migrating to a pure policy-based model with IAM Accounts architecture. If you're running the Ceph Object Gateway (RGW), you have access to the same IAM Accounts model introduced in Ceph Squid 19.2.0. This post explains why ACLs must be disabled immediately and how to implement modern, secure access control with IAM policies. Do This First (Quick Security Wins) Before reading further, ## Breaking the Static Key Habit: Modernizing Ceph RGW S3 Security with STS DevFeed: [Breaking the Static Key Habit: Modernizing Ceph RGW S3 Security with STS]() Original publisher: [Read original article]() Author: Daniel Alexander Parkes, Anthony D'Atri Published: 2025-12-18T00:00:00Z Content type: tutorial Language: en Sources: [Ceph Blog]() Topics: [Security](), [Amazon S3](), [Credential theft](), [configuration](), [GitHub](), [CI/CD](), [App]() Tags: [amazon-s3](), [app](), [aws](), [blog-post](), [ceph](), [ci-cd](), [code](), [configuration](), [credential-theft](), [en-article](), [en-blog-post](), [github](), [rgw](), [s3](), [security]() ### AI overview This tutorial explains how to replace long-lived S3 credentials in Ceph Object Gateway (RGW) applications with temporary credentials issued through Security Token Service (STS). It uses the Uber breach to illustrate how stolen static keys can enable persistent unauthorized access, and discusses credentials stored in configuration files, scripts, and CI/CD variables. ### Source excerpt Introduction: The USD 148 Million Lesson ¶ In late 2016, Uber learned that intruders had accessed a trove of personal data stored in an Amazon S3 bucket. The entry point was painfully mundane: attackers accessed Uber's source code on GitHub using stolen credentials, found an AWS credential, and used it to access Uber's data. That single, long-lived credential exposed data on roughly 57 million users and 600,000 drivers. The breach was bad; the duration risk was worse. Static access keys do not expire. Once leaked, they remain active until someone notices, locates every instance in use, and rotates them. That makes credential theft uniquely dangerous in cloud and S3-style storage, because an attacker can repeatedly return, automate access, and quietly expand their footprint. Uber ultimately agreed to a $148 million multistate settlement related to how the incident was handled and disclosed. The exact dollar figure is not the main lesson, though. The lesson is this: a single static key can turn a small mistake into a durable breach. If you are running the Ceph Object Gateway (RGW), you face the same dynamic: S3 credentials in an application configuration file config.yaml, embedded in scripts, or stored in CI/CD variables. Each one is a long-lived credential that, once copied, can be used from anywhere the S3 endpoint is reachable. This post shows you how to eliminate static credentials using Security Token Service (STS) with temporary credentials that expire automatically. By the end, you'll understand how to implement the same security model that prevented these breaches from being even worse, and how to adapt it for Ceph RGW. The Static Credential Problem ¶ Let's take a look at some examples of how most applications access S3 storage today: # app-config.yaml (application config file) s3: endpoint: https://s3.example.com access_key: AKIA1234567890ABCDEF secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY bucket: production-data Or with the credentials embedded direc ## RocksDB Compression in Ceph: Space Savings with No Performance Cost DevFeed: [RocksDB Compression in Ceph: Space Savings with No Performance Cost]() Original publisher: [Read original article]() Author: Daniel Alexander Parkes, Anthony D'Atri Published: 2025-12-17T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [Compression](), [Database](), [Hardware](), [ibm]() Tags: [blog-post](), [ceph](), [compression](), [configuration](), [cost](), [devices](), [en-article](), [en-blog-post](), [ibm](), [mon](), [nvme](), [osd](), [performance](), [rados](), [reef](), [rocksdb](), [space](), [storage]() ### AI overview This article reports Ceph performance tests showing that enabling RocksDB compression can substantially reduce metadata database space, especially for smaller objects, without harming throughput or resource consumption. The tests used IBM Storage Ceph 7.1 with BlueStore OSDs, HDD object storage, and NVMe devices for the RocksDB WAL and database. ### Source excerpt Introduction ¶ In the world of data storage, engineers and architects constantly face a fundamental dilemma: the trade-off between performance and efficiency. It's a balancing act. When you want to save space, you typically enable features like compression, but the common assumption is that this will cost you performance, a CPU cycle tax that slows throughput. But what if you could significantly reduce your metadata storage footprint without slowing things down? This search for an answer to this question started with research work from Mark Nelson, who published a blog post on ceph.io that covers RocksDB tuning in depth, exploring RocksDB compression with positive results. These promising results sparked a conversation on the upstream GitHub about enabling compression by default; a link to the PR is available here. To build on the previous investigation, the Ceph performance team ran tests on a robust hardware configuration running IBM Storage Ceph 7.1 (Reef). The cluster used the BlueStore OSDs for an erasure-coded (EC 4+2) pool, with a hybrid OSD storage setup: HDDs for object data and fast NVMe drives for the BlueStore WAL+DB. To understand the test, it's helpful to know what the WAL+DB is. In modern Ceph, the BlueStore storage engine manages all data on the OSDs (physical devices). To do this, it must maintain a vast catalog of internal metadata: think of it as a high-speed index that quickly locates every piece of data. RocksDB, a high-performance key-value database, manages this critical index. In our hybrid cluster, the RocksDB database runs on the fast NVMe deviceses, while the actual object data resides on the slower HDDs. Because this metadata can grow very large, RocksDB's efficiency, how much space it consumes on those expensive NVMe drives, is a critical factor in the cluster's overall cost and performance. Our test, therefore, focuses on a simple, high-stakes question: Can we compress this metadata to save space without paying a performance penalty? Ex ## Migrating from VMware to Proxmox VE at 3,000-Core Scale DevFeed: [Migrating from VMware to Proxmox VE at 3,000-Core Scale]() Original publisher: [Read original article]() Author: Shivam Anand Published: 2025-08-23T08:25:07Z Content type: article Language: en Sources: [Shivam Anand - DevOps & Cloud Engineering Blog]() Topics: [Proxmox](), [virtualization](), [migration](), [Automation](), [Command-line interface](), [ceph](), [NVMe]() Tags: [automation](), [ceph](), [cli](), [kvm](), [migration](), [nvme](), [proxmox](), [virtualization-cloud-infrastructure](), [vmware]() ### AI overview The article describes a three-month migration of 3,000 CPU cores, 8 TB of RAM, and production and archival storage from VMware to Proxmox VE. It explains the use of NFS, Ceph, and a custom CLI orchestrator to manage the resulting infrastructure. ### Source excerpt Discover how I migrated 3,000 cores, 8TB RAM from VMware to Proxmox VE, built a custom CLI orchestrator, and leveraged Ceph storage for seamless management. ## Integrating Confluent Schema Registry with Apache Spark applications DevFeed: [Integrating Confluent Schema Registry with Apache Spark applications]() Original publisher: [Read original article]() Author: Dominik Liebler Published: 2022-01-24T08:04:19Z Content type: tutorial Language: en Sources: [YAZIO Engineering - Medium]() Topics: [Kafka](), [Apache Spark](), [Streaming](), [data lake](), [parquet](), [Kotlin](), [Kubernetes](), [ceph](), [JSON Schema]() Tags: [apache-spark](), [backpressure](), [ceph](), [confluent](), [data-engineering](), [data-lake](), [data-pipeline](), [json](), [kafka](), [kotlin](), [kubernetes](), [parquet](), [payload](), [pipelines](), [schema](), [schemaregistry](), [serialization](), [spark](), [streaming]() ### AI overview This engineering article explains YAZIO's data pipeline from mobile and web applications through Kafka and Spark Structured Streaming into a Ceph-based data lake. It discusses why schemas matter and describes replacing JSON with Apache Avro and Confluent Schema Registry to reduce message size while keeping schema information externally stored and cached. ### Source excerpt At YAZIO, we believe in making decisions backed by data to help people live healthier lives through better nutrition. For each new and existing feature we want to evaluate how well it performs and how our users interact with it. In order to do so, we need a lot of data and we need to handle backpressure in our systems. To cope with that we use a Kafka cluster managed by Strimzi operators running in Kubernetes. The data itself is being ingested from our mobile and web apps via HTTP or TCP endpoints serialized into JSON and stored in Kafka by a small application written in Kotlin/JVM. Overview of our data pipeline architecture At the other end of the pipeline, different Spark Structured Streaming applications (also written in Kotlin) dump this information into our data lake residing in a Ceph bucket. They read data from Kafka, deserialize it, transform some of the fields and write Parquet files into the data lake using a new schema. Why schemas? Schemas play an important role in data pipelines because they give meaning and context to data. In a world without schemas we would still do random interpretations about the context and meaning of data every now and then when using it. As you might have guessed already this would lead to a lot of bugs and misunderstandings. Photo by EJ Strat https://unsplash.com/photos/VjWi56AWQ9k Similar to a legal contract that binds you to certain limits, a schema binds the data to certain limits and meaning which narrow down the need of interpretation. Choice of serialization formats At the time of writing, Confluent Schema Registry supports these three serialization formats: Apache Avro Protocol Buffers (protobuf) JSON Schema From those choices, only two really provide more than just validation of the data that is ingested and transmitted through our data pipelines. Avro and Protobuf also allow us to shrink the sizes of our topics because only the payload is contained in a message, while the repeating schema will not be stored. In the cas ## How to safely upgrade to Proxmox 7 DevFeed: [How to safely upgrade to Proxmox 7]() Original publisher: [Read original article]() Author: Techno Tim Published: 2021-07-10T19:00:00Z Content type: tutorial Language: en Sources: [Techno Tim]() Topics: [Proxmox](), [Homelab](), [Debian](), [ceph](), [Network]() Tags: [ceph](), [debian](), [hardware](), [how-to](), [network](), [proxmox](), [update](), [video]() ### AI overview A video tutorial covering Proxmox 7's new features and the process for upgrading a Proxmox server safely. It discusses upgrade prompts, package and repository preparation, Ceph considerations, and network configuration changes for LACP or LAGG. ### Source excerpt As you may know, proxmox is my current choice for a hypervisor. Proxmox 7 is here and comes with a host of new features! In this video we're cover all of the new features in Proxmox 7 as well as how to upgrade your Proxmox server safely. We'll also cover all of the "scary" prompts you get while upgrading as well as some of the ways to make sure your upgrade is successful. So, if you're thinkin... ## Allas November 2020 incident details DevFeed: [Allas November 2020 incident details]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2020-12-22T12:31:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [incident](), [Amazon S3](), [rocksdb](), [NVMe](), [Processes](), [Vapor]() Tags: [ceph](), [deep-dive](), [incident](), [nvme](), [processes](), [rocksdb](), [s3](), [software](), [storage](), [swift](), [technical]() ### AI overview A technical deep dive into the November 2020 Allas downtime incident. It describes Allas as a Ceph cluster and explains its storage nodes, OSD processes, S3 and SWIFT access, replicated metadata pools, RocksDB metadata, placement groups, and erasure-coded object storage. ### Source excerpt Allas downtime November 2020 - technical deep-dive We use Ceph (https://ceph.io/) to provide software defined storage in our Allas service. I'll try to speak some Ceph here. There may be a lot of unfamiliar terms for most readers here as we dive deep into a technology. I'll try to clarify where I can, but architectural knowledge of Ceph helps when reading this. Allas is a Ceph cluster with 48 storage nodes (OSD nodes). Each has one nvme, 24 spinning disks, and 192 GiB of memory. Each OSD node runs one OSD process per disk. The Allas storage is provided over SWIFT and S3 protocols through radosgw daemons. The radowsgw layer is provided by separate internet facing servers. In addition we have 5 nodes running monitor processes (they control the state of the cluster), which also run Ceph manager (handles more advanced monitoring views into the cluster state). The object store metadata pools are 3-way replicated pools (data is stored in 3 different copies) on the nvmes across the OSD nodes. This metadata contains e.g. owners of buckets and objects, permissions, versions, etc. Everything that has to do with S3/SWIFT. The nvme also hosts the rocksdb (ceph internal metadata, as opposed to object store metadata) for the OSDs. This metadata is internal Ceph clustster status. Data on placement groups, etc. I'll use the term placement group (or pg) quite a lot in the post as they are central for Ceph. Basically when you store data objects into Ceph, they are mapped into placement groups, which define where in the cluster the data is are stored. The object store data is stored on an 8+3 erasure coded (data is split into 8 chunks + 3 parity chunks) pool on the spinning disks. The total amount of placement groups for the data pool is 16 384. In the graph the amount ~18 000 is shown, as the graph contains other pools too. The beginning Side note: I love that we collect trending data for our cluster (collectd+graphite in our case). As we had the data stored from a long time, this al ## Preparing a Proxmox Server for Production Use DevFeed: [Preparing a Proxmox Server for Production Use]() Original publisher: [Read original article]() Author: Techno Tim Published: 2020-11-28T14:00:00Z Content type: tutorial Language: en Sources: [Techno Tim]() Topics: [Proxmox](), [networking](), [ceph](), [Linux]() Tags: [ceph](), [clustering](), [hardware](), [iommu](), [linux](), [networking](), [proxmox](), [ssh](), [storage](), [tutorials]() ### AI overview A tutorial covering the author's preparation steps for newly installed Proxmox servers, including updates, storage, networking and VLANs, ISO uploads, clustering, PCI passthrough, cloned Linux machine cleanup, and alerts. ### Source excerpt After setting up my Proxmox servers, there are a few things I do before I use them for their intended purpose.This ranges from updates, to storage, to networking and VLANS, to uploading ISOs, to clustering, and more.Join me as we pick up where the rest of the proxmox tutorials stop, and that's everything you need to do to make these production ready (and maybe a bonus item too). 📺 Watch Vide... ## Migrating cPouta Object Storage to the Allas Ceph Service DevFeed: [Migrating cPouta Object Storage to the Allas Ceph Service]() Original publisher: [Read original article]() Author: Kalle Happonen (noreply@blogger.com) Published: 2019-12-27T11:23:00Z Content type: article Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [ceph](), [migration](), [Data Management](), [data](), [cloud-infrastructure]() Tags: [ceph](), [data-management](), [migration](), [object-storage](), [openstack](), [radosgw](), [rgw](), [storage]() ### AI overview This article describes CSC's migration from the cPouta object storage service to Allas, a standalone Ceph-based object storage service. The plan aimed to move user data without requiring user changes or breaking existing data links, using a new Ceph cluster and new data pools. ### Source excerpt We have released our new Allas object storage service. We firmly believe it will play a growing role for data management, at CSC and for the whole academic field in Finland. Alas, the road to Allas was not completely without pain. CSC also hosts the cPouta IaaS service. It provided its own object storage service. Our goal with Allas was to build on this, and transform the object storage portion to a standalone service. This would raise the profile of the service, and make it easier to scale, both human resource wise and where it comes to the platform size. So, in brief, our aim was: Replace cPouta object storage with Allas Make sure all the user data from cPouta object storage moves over. Make sure cPouta object storage users don't have to do any changes. Don't break any existing links to data in cPouta object storage. How did we intend to do this, you may ask? Migration plan What were our options? We had a large bunch of new hardware, and a high level plan. For those of you who know Ceph, you might have already thought of how you'd do this. Add more hardware to the same storage pool, slap on an additional domain name to access the data, and call it a day. Create additional data pools, make sure radosgw uses old and new pools for data, slap on an additional domain name, and call it a day. Create a new Ceph cluster with new pools, migrate the data over, add the old and new domain names, and call it a day. Number 3. is definitely the most work, but we went with that. The main reason for that was that if we want to achieve all the benefits of separating Allas into its own service, we actually need to be a bit separate. For example, the old object storage pools were in the same cluster as the block storage pools used by our OpenStack virtual machines. This not only locked us to the current version of Ceph running there, but created a lot on interdependencies between the services. To make it easier, I'll use the terms rgw-old (cPouta object storage) and rgw-new (Allas) t ## Linux I/O 栈浅析 DevFeed: [Linux I/O 栈浅析]() Original publisher: [Read original article]() Author: Joway Published: 2019-08-11T00:00:00Z Content type: tutorial Language: zh Sources: [Random Thoughts]() Topics: [IO](), [Linux](), [Cache](), [file](), [Ext4](), [XFS]() Tags: [block](), [cache](), [ceph](), [ext4](), [file](), [filesystem](), [hdd](), [hdfs](), [io](), [linux](), [mysql](), [oracle](), [ssd](), [tech](), [xfs]() ### AI overview This article provides a high-level overview of Linux file I/O, tracing the path from applications through VFS, page cache, filesystems, FUSE, block devices, and I/O scheduling. It also explains hard and symbolic links, direct I/O, filesystem partitioning, and access to raw devices. ### Source excerpt 在 Linux 中，所有外部资源都以文件形式作为一个抽象视图，并提供一套统一的接口给应用程序调用。本文将以宏观视角试图阐述 Linux 中关于文件 IO 的整个调用脉络。 VFS 在 Linux 中，所有 IO 都必须先经由 VFS 层进行转发。通过 VFS 将包括磁盘、网络 Socket、打印机、管道等资源全部封装成统一的接口。 ## Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository DevFeed: [Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-04-29T09:06:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Git](), [ceph](), [pip](), [Haskell]() Tags: [aws](), [ceph](), [data](), [data-lake](), [git](), [install](), [object-storage](), [python](), [rados](), [radosgw](), [s3](), [storage]() ### AI overview This post explains why binary files should not be stored directly in Git repositories and explores using CSC's Pouta Cloud Object Storage instead. It documents attempts with git-annex and git-bigstore, including a patch to configure a non-default S3-compatible endpoint. ### Source excerpt 1. It's bad to store binary blobs in git. This is known [1] 2. A great place to store binary files is in object storage 3. Can we use CSC's Pouta Cloud Object Storage service for this? Why yes, yes you can! This is a post in a series about different use cases for object storage. Others include for example hosting a reveal.js presentation and NFS server backups More details about CSC's Pouta Cloud Object Storage Service, it is based on CEPH Rados Object Storage Gateway. What The general solution* to storing binaries in git is to not store them in git and instead create a link to some other place Attempt 1 Install an application that you like. In https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories I searched for "S3", found git-annex which had recent commits so it looked like it wasn't dead. Installation instructions: https://git-annex.branchable.com/install/ The version in EPEL/Ubuntu Xenial does not have S3 support and is from 2014. There's a recent one in Fedora's repos though. Unfortunately, compiling it requires cabal. I don't know Haskell. https://spack.readthedocs.io/en/latest/package_list.html does not have any Haskell, cabal or git-annex available. Next. Attempt 2 Next application in the blog post was https://github.com/lionheart/git-bigstore To install it was much more comfortable for me: pip install "git-bigstore<=2.0" First issue was that looking at https://github.com/lionheart/git-bigstore/blob/master/bigstore/backends/s3.py there was no way to tell it to use another endpoint than the default of AWS S3. A small patch takes care of that, add , endpoint_url='https://object.pouta.csc.fi:443' to the client() boto3 call in the backends/s3.py file from git-bigstore. For a more detailed example see this Pull Request. After spending some time going down some rabbit holes it seems that the Profile option that is supported in git-bigstore does not support endpoints. It can only be used to specify credentials and a few other parameters. Mino [Next page]()