diff --git a/talks-public/2018-02-04-deposit-vault-walkthrough/deposit-vault-walkthrough.org b/talks-public/2018-02-04-deposit-vault-walkthrough/deposit-vault-walkthrough.org index f71da72..0ea8077 100644 --- a/talks-public/2018-02-04-deposit-vault-walkthrough/deposit-vault-walkthrough.org +++ b/talks-public/2018-02-04-deposit-vault-walkthrough/deposit-vault-walkthrough.org @@ -1,257 +1,308 @@ #+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %10BEAMER_act(Act) %4BEAMER_col(Col) %10BEAMER_extra(Extra) %8BEAMER_opt(Opt) #+LATEX_HEADER_EXTRA: \usepackage{listings} +# the latex headers below are only for the tikz diagram +#+LATEX_HEADER_EXTRA: \usepackage{tikz} +#+LATEX_HEADER_EXTRA: \usetikzlibrary{arrows,shapes} +#+LATEX_HEADER_EXTRA: \definecolor{swh-orange}{RGB}{254,205,27} +#+LATEX_HEADER_EXTRA: \definecolor{swh-red}{RGB}{226,0,38} +#+LATEX_HEADER_EXTRA: \definecolor{swh-green}{RGB}{77,181,174} + #+INCLUDE: "../../common/modules/prelude.org" :minlevel 1 #+TITLE: Source Code Deposit Walkthrough # does not allow short title, so we override it for beamer as follows : #+BEAMER_HEADER: \title[Software Heritage]{Software Heritage\\Source Code Deposit Walkthrough} #+BEAMER_HEADER: \author{Morane Gruenepter} #+BEAMER_HEADER: \date[04/01/2018, Deposit walkthrough]{04 January 2018\\Deposit walkthrough\\Paris, France} #+AUTHOR: Morane Gruenpeter #+DATE: 04 January 2018 #+EMAIL: morane@gmail.com #+DESCRIPTION: Software Heritage: Source Code Deposit Walkthroug #+KEYWORDS: software heritage preservation knowledge deposit technology sword #+BEAMER_HEADER: \institute[Software Heritage]{Metadata specialist\\Software Heritage\\\href{mailto:morane@softwareheritage.org}{\tt morane@softwareheritage.org}} * Source code deposit: the new SWH push feature ** Source code deposit: the new SWH push feature :PROPERTIES: :CUSTOM_ID: walkthrough :END: **** First version of our software deposit prototype \\ *\url{https://deposit.softwareheritage.org/}* **** Features - pushing *deposits* to the Software Heritage archive - software source code + metadata - full *transparency* of the loading and downloading processes - download the deposit by cooking the bundle in the *vault* **** SWORD-compliant - *SWORD v2* protocol for single and multipart deposits - deposit /MUST/, /SHOULD/ and /MAY/ contain certain metadata attributes * Deposit walkthrough ** Deposit walkthrough *** Prepare source code for deposit #+BEAMER: \tiny #+BEGIN_SRC $ tar cvf #+END_SRC #+BEAMER: \pause *** Create metadata file with /MUST/ metadata #+BEAMER: \tiny - *the url* representing the location of the source /MUST/ be provided - *the external-identifier* /MUST/ be provided - *the name/title* of the software deposit /MUST/ be provided - *the author/s*- one or more authors /MUST/ be provided *** #+BEAMER: \tiny #+BEGIN_SRC Je suis GPL ext-id forge.softwareheritage.org/source/jesuisgpl/ Yes, this is another implementation of "Hello, world!” when you run it. GPL https://www.gnu.org/licenses/gpl.html Reuben Thomas and Sami Kerola Maintainers #+END_SRC ** Deposit walkthrough *** Pushing a single deposit with metadata #+BEAMER: \tiny #+BEGIN_SRC $ swh-deposit --username 'name' --password 'pass' je-suis-gpl.tgz #+END_SRC #+BEAMER: \pause *** Response #+BEAMER: \tiny #+BEGIN_SRC 11 Jan. 4, 2018, 2:51 p.m. je-suis-gpl.tar.gz ready-for-checks ... #+END_SRC ** Deposit walkthrough *** multipart deposit #+BEAMER: \footnotesize A multipart deposit with *partial* status can be: - completely replaced - updated with new content or metadata #+BEAMER: \pause *** Update by replacing archive and metadata #+BEAMER: \tiny #+BEGIN_SRC $ swh-deposit --username 'name' --password 'secret' \ --deposit-id 11 updated-je-suis-gpl.tar.gz #+END_SRC #+BEAMER: \pause *** Complete multipart deposit #+BEAMER: \tiny #+BEGIN_SRC $ swh-deposit --username 'name' --password 'secret' --partial \ foo.tar.gz $ swh-deposit --username 'name' --password 'secret' --partial \ --deposit-id 42 add-foo.tar.gz $ swh-deposit --username 'name' --password 'secret' \ --deposit-id 42 last-foo.tar.gz #+END_SRC ** Deposit walkthrough *** What's your status? #+BEAMER: \footnotesize - - *partial* : multipart deposit is still ongoing - - *deposited*: deposit completed - - *rejected*: deposit failed the checks - - *verified*: content and metadata verified - - *done*: loading completed successfully - - *failed*: the deposit loading has failed + +\tikzstyle{fail} = [draw, thin, fill=swh-red!80!swh-orange, minimum height=1.5em] +\tikzstyle{processing} = [draw, thin, fill=swh-orange!100, minimum height=1.5em] +\tikzstyle{success} = [draw, thin, fill=swh-green!80!swh-orange, minimum height=1.5em] + +\begin{figure} +\begin{tikzpicture}[node distance=1.8cm, auto,>=latex', thick] + % We need to set at bounding box first. Otherwise the diagram + % will change position for each frame. + %\path[use as bounding box] (-1,0) rectangle (10,-2); + \path[->]<1-> node[processing] (partial) {partial} + node[success, right of=partial] (deposited) {deposited} + (partial) edge node {} (deposited) + node[processing, right of=deposited] (checking) {} + (deposited) edge node {} (checking) + node[success, right of=checking] (verified) {verified} + node[fail, below of=checking] (rejected) {rejected} + (checking) edge node {} (verified) + edge node[swap] {} (rejected) + node[processing, right of=verified] (loading) {} + node[success, right of=loading] (done) {done} + node[fail, below of=loading] (failed) {failed} + (verified) edge node {} (loading) + (loading) edge node {} (failed) + (loading) edge node {} (done); +\end{tikzpicture} +\end{figure} + #+BEAMER: \pause *** Checking the deposit's state #+BEAMER: \tiny #+BEGIN_SRC $ swh-deposit --login 'name' --pass 'secret' --deposit-id '11' \ --collection 'col' --status #+END_SRC #+BEAMER: \pause *** #+BEAMER: \tiny #+BEGIN_SRC HTTP/1.0 200 OK Date: Thu, 04 Jan 2018 15:20:12 GMT... 11 done The deposit has been successfully loaded into the Software Heritage archive swh:1:rev:sha1:608757ea... #+END_SRC ** *** Deposit completed and loaded The deposit was succesfuly pushed and integrated into the archive.\\ You can browse it using the deposit's /swh-id/: - *archive.softwareheritage.org/browse/*\\ +#+BEAMER: \pause + *** Download Now we want to download the content with the #+BEAMER: \huge *Vault* ** Vault walkthrough *** Software identifier to request download #+BEAMER: \footnotesize The swh-id *swh:1:rev:608757ea9bd8494d729732cc9a414948c160bd3c* is composed of: - the context *swh:1:rev* - and the object identifier a sha1 hash \\ *608757ea9bd8494d729732cc9a414948c160bd3c* We will use the object identifier to create a bundle to download *** Requesting download with swh-id #+BEAMER: \tiny #+BEGIN_SRC sh $ curl -X POST /api/1/vault/revision/608757ea.../gitfast #+END_SRC +*** Email notification +#+BEAMER: \footnotesize +optionally, an email POST parameter containing an e-mail to notify when the bundle cooking has ended. + ** Vault walkthrough *** Checking progress #+BEAMER: \tiny #+BEGIN_SRC sh $ curl /api/1/vault/revision/608757ea.../gitfast #+END_SRC # can we cook objects that aren't revisions? *** Response #+BEAMER: \tiny #+BEGIN_SRC json { 'fetch_url': '/api/1/vault/revision/608757ea.../gitfast/raw/', 'progress_message': None, 'status': 'pending', 'id': 4, 'obj_id': '608757ea9bd8494d729732cc9a414948c160bd3c', 'obj_type': 'revision_gitfast' } #+END_SRC *** What's your status? #+BEAMER: \small - - *new*: bundle requested for the object - - *pending*: bundle in preparation - - *failed* : the bundle cooking has failed - - *done*: bundle ready for download +\tikzstyle{fail} = [draw, thin, fill=swh-red!80!swh-orange, minimum height=1.5em] +\tikzstyle{processing} = [draw, thin, fill=swh-orange!100, minimum height=1.5em] +\tikzstyle{success} = [draw, thin, fill=swh-green!80!swh-orange, minimum height=1.5em] + +\begin{figure} +\begin{tikzpicture}[node distance=1.8cm, auto,>=latex', thick] + % We need to set at bounding box first. Otherwise the diagram + % will change position for each frame. + %\path[use as bounding box] (-1,0) rectangle (10,-2); + \path[->]<1-> node[processing] (new) {new} + node[processing, right of=new] (pending) {pending} + node[processing, right of=pending] (loading) {} + node[success, right of=loading] (done) {done} + node[fail, below of=loading] (failed) {failed} + (new) edge node {} (pending) + (pending) edge node {} (loading) + (loading) edge node {} (failed) + (loading) edge node {} (done); +\end{tikzpicture} +\end{figure} ** Vault walkthrough *** Checking progress #+BEAMER: \tiny #+BEGIN_SRC sh $ curl /api/1/vault/revision/608757ea.../gitfast #+END_SRC *** Response #+BEAMER: \tiny #+BEGIN_SRC json { 'fetch_url': '/api/1/vault/revision/608757ea.../gitfast/raw/', 'progress_message': None, 'status': 'done', 'id': 4, 'obj_id': '608757ea9bd8494d729732cc9a414948c160bd3c', 'obj_type': 'revision_gitfast' } #+END_SRC *** Download available when status is marked /done/ #+BEAMER: \tiny #+BEGIN_SRC sh $ curl /api/1/vault/revision/608757ea.../gitfast/raw/ \ -O path/to/revision.gitfast.gz $ git init $ zcat path/to/revision.gitfast.gz | git fast-import $ git checkout HEAD #+END_SRC