Page MenuHomeSoftware Heritage

Using EOS to store objects
Closed, ResolvedPublic

Description

https://eos-web.web.cern.ch/eos-web/

"EOS is based on various open source components. The native protocol and framework used for implementation is XRootD. Meta-data is persisted in RocksDB databases using a proprietary KV store called QuarkDB. Data is stored natively in XFS filesystems on hard disks or SSDs or on virtualised back-end storage (e.g. RADOS block devices) or distributed filesystems like Lustre or CephFS. "

Event Timeline

dachary changed the task status from Open to Work in Progress.Feb 15 2021, 9:36 PM
dachary triaged this task as Normal priority.
dachary created this task.
dachary created this object in space S1 Public.

https://xrootd.slac.stanford.edu/papers/Scalla-Intro.pdf

The Scalla software suite provides two fundamental building blocks: an xrootd server for low latency high bandwidth data access and an olbd server for building scalable xrootd clusters. This paper describes the architecture, how low latency is achieved, and the scaling opportunities the software allows. Actual performance measurements are presented and discussed. Scalla offers a readily deployable framework in which to construct large fault-tolerant high performance data access configurations using commodity hardware with a minimum amount of administrative overhead.

At first glance EOS is an entire system that adresses all the needs of the researchers at CERN. It includes an object storage with data and metadata separated, which is what the Software Heritage is likely to look like as well. However, this part is not standalone. Although it is a great source of inspiration:

  • the choice of Rocksdb and how it is replicated and synchronized with a consensus algorithm for metadata (maybe motivated by the size of the metadata it contains ?)
  • the choice of RBD to store data (not the only choice but one of the options)