Page MenuHomeSoftware Heritage
Files F9341040

2017-09-27-kernel-recipes.org
No OneTemporary
Actions

2017-09-27-kernel-recipes.org
View Options

#+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %10BEAMER_act(Act) %4BEAMER_col(Col) %10BEAMER_extra(Extra) %8BEAMER_opt(Opt)
#+TITLE: Software Heritage: Our Software Commons, Forever.
#+SUBTITLE: challenges in storing the biggest VCS graph in history
#+BEAMER_HEADER: \date[Kernel Recipes]{27 September 2017\\Kernel Recipes --- Paris, France}
#+AUTHOR: Nicolas Dandrimont
#+DATE: 27 September 2017
#+INCLUDE: "../../common/modules/prelude-toc.org" :minlevel 1
#+INCLUDE: "../../common/modules/169.org"
#+BEAMER_HEADER: \institute{Inria, Software Heritage}
#+LATEX_HEADER: \usepackage{bbding}
#+LATEX_HEADER: \DeclareUnicodeCharacter{66D}{\FiveStar}
#+BEAMER_HEADER: \pgfdeclareimage[height=90mm,width=160mm]{bgd}{swh-world-169.png}
#+BEAMER_HEADER: \setbeamertemplate{background}{\pgfuseimage{bgd}}
#+LaTeX_CLASS_OPTIONS: [aspectratio=169,xcolor=table]
* The Software Commons
#+INCLUDE: "../../common/modules/source-code-different-short.org::#softwareisdifferent" :minlevel 2
** Our Software Commons
#+INCLUDE: "../../common/modules/foss-commons.org::#commonsdef" :only-contents t
#+BEAMER: \pause
*** Source code is /a precious part/ of our commons
\hfill are we taking care of it?
# #+INCLUDE: "../../common/modules/swh-motivations-foss.org::#main" :only-contents t :minlevel 2
#+INCLUDE: "../../common/modules/swh-motivations-foss.org::#fragile" :minlevel 2
* Software Heritage
#+INCLUDE: "../../common/modules/swh-overview-sourcecode.org::#mission" :minlevel 2
** Our principles
#+latex: \begin{center}
#+ATTR_LATEX: :width .9\linewidth
file:SWH-as-foundation-slim.png
#+latex: \end{center}
#+BEAMER: \pause
*** Open approach :B_block:BMCOL:
:PROPERTIES:
:BEAMER_col: 0.4
:BEAMER_env: block
:END:
- 100% FOSS
- transparency
*** In for the long haul :B_block:BMCOL:
:PROPERTIES:
:BEAMER_col: 0.4
:BEAMER_env: block
:END:
- replication
- non profit
* Architecture
#+INCLUDE: "../../common/modules/status-extended.org::#archivinggoals" :minlevel 2
#+INCLUDE: "../../common/modules/status-extended.org::#architecture" :only-contents t
#+INCLUDE: "../../common/modules/status-extended.org::#merkletree" :minlevel 2
#+INCLUDE: "../../common/modules/status-extended.org::#giantdag" :only-contents t
#+INCLUDE: "../../common/modules/status-extended.org::#archive" :minlevel 2
* Gory details
** Technology: how do you store the SWH DAG?
*** Problem statement
- How would you store and query a graph with 10 billion nodes and 60 billion edges?
- How would you store the contents of more than 3 billion files, 300TB of raw data?
- on a limited budget (100 000 € of hardware overall)
#+BEAMER: \pause
*** Our hardware stack
- two hypervisors with 512GB RAM, 20TB SSD each, sharing access to a storage array (60 x 6TB spinning rust)
- one backup server with 48GB RAM and another storage array
*** Our software stack
- A RDBMS (PostgreSQL, what else?), for storage of the graph nodes and edges
- filesystems for storing the actual file contents
** Technology: archive storage components
*** Metadata storage
- Python module *swh.storage*
- thin Python API over a pile of PostgreSQL functions
- motivation: keeping relational integrity for the DAG at the lowest layer
*** Content ("object") storage
- Python module *swh.objstorage*
- very thin object storage abstraction layer (PUT, APPEND and GET) over classic storage technologies
- separate layer for asynchronous replication and integrity management (*swh.archiver*)
- motivation: stay as technology neutral as possible for future mirrors
** Object storage
*** Current primary deployment
- Storage on 16 sharded XFS filesystems; key = /sha1/ (content), value = /gzip/ (content)
- if sha1 = *abcdef01234...*, file path = / srv / storage / *a* / *ab* / *cd* / *ef* / *abcdef01234...*
- 3 directory levels deep, each level 256-wide = 16 777 216 directories (1 048 576 per partition)
*** It sucks.
Slow spinning storage + little RAM (48GB) + 16 million directory inodes = (very, very) bad performance
*** Outlook
- Moving towards a scale-out system: we need better performance and scalability.
- Can't really ask partial mirrors to commit to 100k EUR of hardware to spin up a copy of the archive.
** Better object storage on plain Linux filesystems
*** fix our wrong assumptions about filesystems
Less directory depth, more breadth: make better use of "recent" filesystem
features such as dir\textunderscore{}index. Yes, directory lookups use
hashtables on ext4 now!
*** optimize dentry storage / cache
shorter filenames (e.g. use base64/ascii85 instead of hex digits) = smaller
directory entries, which means more entries in the dir\textunderscore{}index
hashtable.
*** does it work?
probably! not enough spare storage (or time) to test our enhancements :(
* Community
** You can help!
#+BEAMER: \vspace{-1mm}
*** Coding
- \url{www.softwareheritage.org/community/developers/}
- \url{forge.softwareheritage.org} --- *our own code*
#+BEAMER: \vspace{-3mm}
*** Current development priorities
| ٭٭٭ | listers for unsupported forges, distros, pkg. managers |
| ٭٭٭ | loaders for unsupported VCS, source package formats |
| ٭٭ | Web UI: eye candy wrapper around the Web API |
| ٭ | content indexing and search |
#+BEAMER: \vspace{-2mm}
… /all/ contributions equally welcome!
#+BEAMER: \pause \vspace{-1mm}
#+INCLUDE: "../../common/modules/endorsement.org::#endorsement" :minlevel 2
#+INCLUDE: "../../common/modules/swh-sponsors.org::#sponsors" :minlevel 2
* Conclusion
#+INCLUDE: "../../common/modules/swh-backmatter.org::#conclusion" :minlevel 2
* FAQ :B_appendix:
:PROPERTIES:
:BEAMER_env: appendix
:END:
#+INCLUDE: "../../common/modules/swh-backmatter.org::#faq-sha1" :minlevel 2

File Metadata

Mime Type
text/x-tex
Expires
Fri, Jul 4, 11:33 AM (3 w, 3 d ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3245494

Event Timeline

About · Developer information · Legal