Page MenuHomeSoftware Heritage
Differential D6448

Deploy a read-only objstorage on moma
ClosedPublic
Actions

Authored by vsellier on Oct 8 2021, 5:14 PM.

Details

Reviewers
olasd
Group Reviewers
System administrators
Maniphest Tasks
T3621: Create a production read-only objstorage
Commits
rSPSITE4f19b14f25c0: Deploy a read-only objstorage on moma
Summary
  • use the already existing multiplexed configuration to rely on
    • banco
    • saam
  • Add support of basic authentication if needed on varnish vhosts
  • declare basic auth users for staging and production objstorage

    Related to T3621
Test Plan

example of the basic auth section in the varnish configuration:

if ( 1 == 1  # noop expression to be syntactically correct with the following &&
        && ! req.http.Authorization ~ "Basic c3doLXByb2Q6c3doOjpkZXBsb3k6Om9ianN0b3JhZ2U6OnJldmVyc2VfcHJveHk6OmJhc2ljX2F1dGg6OnN3aC1wcm9k"
        && ! req.http.Authorization ~ "Basic ZW5lYS1wcm9kOnN3aDo6ZGVwbG95OjpvYmpzdG9yYWdlOjpyZXZlcnNlX3Byb3h5OjpiYXNpY19hdXRoOjplbmVhLXByb2Q="
) {
    return(synth(401, "Restricted"));
}

octocatalog-diffs:

  • rp0.staging
diff origin/production/rp0.internal.staging.swh.network current/rp0.internal.staging.swh.network
*******************************************
  File[/etc/varnish/includes/50_vhost_objstorage.staging.swh.network.vcl] =>
   parameters =>
     content =>
      @@ -15,4 +15,10 @@
                   return(synth(850, "Moved permanently"));
               } else {
      +            if ( 1 == 1  # noop expression to be syntaxivly correct with the following &&
      +                    && ! req.http.Authorization ~ "Basic c3doLXN0Zzpzd2g6OmRlcGxveTo6b2Jqc3RvcmFnZTo6cmV2ZXJzZV9wcm94eTo6YmFzaWNfYXV0aDo6c3doLXN0Zw=="
      +                    && ! req.http.Authorization ~ "Basic ZW5lYS1zdGc6c3doOjpkZXBsb3k6Om9ianN0b3JhZ2U6OnJldmVyc2VfcHJveHk6OmJhc2ljX2F1dGg6OmVuZWEtc3Rn"
      +            ) {
      +                return(synth(401, "Restricted"));
      +            }
                   set req.http.X-Forwarded-Proto = "https";
                   set req.backend_hint = objstorage;
*******************************************
  Profile::Varnish::Vcl_include[vhost_objstorage.staging.swh.network] =>
   parameters =>
     content =>
      @@ -15,4 +15,10 @@
                   return(synth(850, "Moved permanently"));
               } else {
      +            if ( 1 == 1  # noop expression to be syntaxivly correct with the following &&
      +                    && ! req.http.Authorization ~ "Basic c3doLXN0Zzpzd2g6OmRlcGxveTo6b2Jqc3RvcmFnZTo6cmV2ZXJzZV9wcm94eTo6YmFzaWNfYXV0aDo6c3doLXN0Zw=="
      +                    && ! req.http.Authorization ~ "Basic ZW5lYS1zdGc6c3doOjpkZXBsb3k6Om9ianN0b3JhZ2U6OnJldmVyc2VfcHJveHk6OmJhc2ljX2F1dGg6OmVuZWEtc3Rn"
      +            ) {
      +                return(synth(401, "Restricted"));
      +            }
                   set req.http.X-Forwarded-Proto = "https";
                   set req.backend_hint = objstorage;
*******************************************
  Profile::Varnish::Vhost[deposit.staging.swh.network] =>
   parameters =>
     basic_auth =>
      + false
*******************************************
  Profile::Varnish::Vhost[objstorage.staging.swh.network] =>
   parameters =>
     basic_auth =>
      + true
     basic_auth_strings =>
      + ["c3doLXN0Zzpzd2g6OmRlcGxveTo6b2Jqc3RvcmFnZTo6cmV2ZXJzZV9wcm94eTo6YmFzaWNfYXV0aDo6c3doLXN0Zw==", "ZW5lYS1zdGc6c3doOjpkZXBsb3k6Om9ianN0b3JhZ2U6OnJldmVyc2VfcHJveHk6OmJhc2ljX2F1dGg6OmVuZWEtc3Rn"]
*******************************************
  Profile::Varnish::Vhost[webapp.staging.swh.network] =>
   parameters =>
     basic_auth =>
      + false
*******************************************
  Varnish::Vcl[/etc/varnish/includes/50_vhost_objstorage.staging.swh.network.vcl] =>
   parameters =>
     content =>
      @@ -15,4 +15,10 @@
                   return(synth(850, "Moved permanently"));
               } else {
      +            if ( 1 == 1  # noop expression to be syntaxivly correct with the following &&
      +                    && ! req.http.Authorization ~ "Basic c3doLXN0Zzpzd2g6OmRlcGxveTo6b2Jqc3RvcmFnZTo6cmV2ZXJzZV9wcm94eTo6YmFzaWNfYXV0aDo6c3doLXN0Zw=="
      +                    && ! req.http.Authorization ~ "Basic ZW5lYS1zdGc6c3doOjpkZXBsb3k6Om9ianN0b3JhZ2U6OnJldmVyc2VfcHJveHk6OmJhc2ljX2F1dGg6OmVuZWEtc3Rn"
      +            ) {
      +                return(synth(401, "Restricted"));
      +            }
                   set req.http.X-Forwarded-Proto = "https";
                   set req.backend_hint = objstorage;
*******************************************
*** End octocatalog-diff on rp0.internal.staging.swh.network
  • moma:
diff origin/production/moma.softwareheritage.org current/moma.softwareheritage.org
*******************************************
+ Concat::Fragment[/etc/varnish/includes.vcl:objstorage] =>
   parameters =>
     "content": "include \"includes/01_objstorage.vcl\";",
     "order": "01",
     "target": "/etc/varnish/includes.vcl"
*******************************************
+ Concat::Fragment[/etc/varnish/includes.vcl:vhost_objstorage.softwareheritage.org] =>
   parameters =>
     "content": "include \"includes/50_vhost_objstorage.softwareheritage.org.vcl\...
     "order": "50",
     "target": "/etc/varnish/includes.vcl"
*******************************************
+ Concat::Fragment[hitch::domain objstorage_production] =>
   parameters =>
     "content": "pem-file = \"/etc/hitch/objstorage_production.pem\"\n",
     "notify": "Class[Hitch::Service]",
     "order": "10",
     "target": "/etc/hitch/hitch.conf"
*******************************************
+ Concat::Fragment[nginx-swh-objstorage-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    proxy_pass            http://swh-objstorag...
     "order": 500,
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ Concat::Fragment[nginx-swh-objstorage-default-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    index     index.html index.htm index.php;\...
     "order": 500,
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ Concat::Fragment[nginx-swh-objstorage-default-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ Concat::Fragment[nginx-swh-objstorage-default-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen moma.internal.softwarehe...
     "order": "001",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ Concat::Fragment[nginx-swh-objstorage-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ Concat::Fragment[nginx-swh-objstorage-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen moma.internal.softwarehe...
     "order": "001",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ Concat::Fragment[objstorage_production cacert] =>
   parameters =>
     "notify": "Class[Hitch::Service]",
     "order": "03",
     "source": "/etc/ssl/certs/letsencrypt/objstorage_production/chain.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat::Fragment[objstorage_production cert] =>
   parameters =>
     "notify": "Class[Hitch::Service]",
     "order": "02",
     "source": "/etc/ssl/certs/letsencrypt/objstorage_production/cert.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat::Fragment[objstorage_production dhparams] =>
   parameters =>
     "notify": "Class[Hitch::Service]",
     "order": "04",
     "source": "/etc/hitch/dhparams.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat::Fragment[objstorage_production key] =>
   parameters =>
     "notify": "Class[Hitch::Service]",
     "order": "01",
     "source": "/etc/ssl/certs/letsencrypt/objstorage_production/privkey.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat::Fragment[swh-objstorage-gunicorn_upstream_footer] =>
   parameters =>
     "content": "}\n",
     "order": "90",
     "target": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ Concat::Fragment[swh-objstorage-gunicorn_upstream_header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nupstream swh-objstorage-gunicorn {\n",
     "order": "10",
     "target": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ Concat::Fragment[swh-objstorage-gunicorn_upstream_member_gunicorn-objstorage] =>
   parameters =>
     "content": "  server unix:/run/gunicorn/swh-objstorage/gunicorn.sock;\n",
     "order": 40,
     "target": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ Concat[/etc/hitch/objstorage_production.pem] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "_hitch",
     "mode": "0640",
     "notify": "Class[Hitch::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/hitch/objstorage_production.pem",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat[/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat[/etc/nginx/sites-available/nginx-swh-objstorage-default.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat[/etc/nginx/sites-available/nginx-swh-objstorage.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/nginx/sites-available/nginx-swh-objstorage.conf",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat_file[/etc/hitch/objstorage_production.pem] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "_hitch",
     "mode": "0640",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_hitch_objstorage_production.pem"
*******************************************
+ Concat_file[/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_nginx_conf.d_swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ Concat_file[/etc/nginx/sites-available/nginx-swh-objstorage-default.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage-default.conf"
*******************************************
+ Concat_file[/etc/nginx/sites-available/nginx-swh-objstorage.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage.conf"
*******************************************
+ Concat_fragment[/etc/varnish/includes.vcl:objstorage] =>
   parameters =>
     "content": "include \"includes/01_objstorage.vcl\";",
     "order": "01",
     "tag": "_etc_varnish_includes.vcl",
     "target": "/etc/varnish/includes.vcl"
*******************************************
+ Concat_fragment[/etc/varnish/includes.vcl:vhost_objstorage.softwareheritage.org] =>
   parameters =>
     "content": "include \"includes/50_vhost_objstorage.softwareheritage.org.vcl\...
     "order": "50",
     "tag": "_etc_varnish_includes.vcl",
     "target": "/etc/varnish/includes.vcl"
*******************************************
+ Concat_fragment[hitch::domain objstorage_production] =>
   parameters =>
     "content": "pem-file = \"/etc/hitch/objstorage_production.pem\"\n",
     "order": "10",
     "tag": "_etc_hitch_hitch.conf",
     "target": "/etc/hitch/hitch.conf"
*******************************************
+ Concat_fragment[nginx-swh-objstorage-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    proxy_pass            http://swh-objstorag...
     "order": 500,
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ Concat_fragment[nginx-swh-objstorage-default-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    index     index.html index.htm index.php;\...
     "order": 500,
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage-default.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ Concat_fragment[nginx-swh-objstorage-default-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage-default.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ Concat_fragment[nginx-swh-objstorage-default-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen moma.internal.softwarehe...
     "order": "001",
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage-default.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ Concat_fragment[nginx-swh-objstorage-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ Concat_fragment[nginx-swh-objstorage-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen moma.internal.softwarehe...
     "order": "001",
     "tag": "_etc_nginx_sites-available_nginx-swh-objstorage.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ Concat_fragment[objstorage_production cacert] =>
   parameters =>
     "order": "03",
     "source": "/etc/ssl/certs/letsencrypt/objstorage_production/chain.pem",
     "tag": "_etc_hitch_objstorage_production.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat_fragment[objstorage_production cert] =>
   parameters =>
     "order": "02",
     "source": "/etc/ssl/certs/letsencrypt/objstorage_production/cert.pem",
     "tag": "_etc_hitch_objstorage_production.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat_fragment[objstorage_production dhparams] =>
   parameters =>
     "order": "04",
     "source": "/etc/hitch/dhparams.pem",
     "tag": "_etc_hitch_objstorage_production.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat_fragment[objstorage_production key] =>
   parameters =>
     "order": "01",
     "source": "/etc/ssl/certs/letsencrypt/objstorage_production/privkey.pem",
     "tag": "_etc_hitch_objstorage_production.pem",
     "target": "/etc/hitch/objstorage_production.pem"
*******************************************
+ Concat_fragment[swh-objstorage-gunicorn_upstream_footer] =>
   parameters =>
     "content": "}\n",
     "order": "90",
     "tag": "_etc_nginx_conf.d_swh-objstorage-gunicorn-upstream.conf",
     "target": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ Concat_fragment[swh-objstorage-gunicorn_upstream_header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nupstream swh-objstorage-gunicorn {\n",
     "order": "10",
     "tag": "_etc_nginx_conf.d_swh-objstorage-gunicorn-upstream.conf",
     "target": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ Concat_fragment[swh-objstorage-gunicorn_upstream_member_gunicorn-objstorage] =>
   parameters =>
     "content": "  server unix:/run/gunicorn/swh-objstorage/gunicorn.sock;\n",
     "order": 40,
     "tag": "_etc_nginx_conf.d_swh-objstorage-gunicorn-upstream.conf",
     "target": "/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf"
*******************************************
+ File[/etc/gunicorn/instances/swh-objstorage.cfg] =>
   parameters =>
     "content": "# Gunicorn instance configuration.\n# Managed by puppet (class g...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0644",
     "notify": "Service[gunicorn-swh-objstorage]",
     "owner": "swhstorage"
*******************************************
+ File[/etc/nginx/sites-enabled/nginx-swh-objstorage-default.conf] =>
   parameters =>
     "ensure": "link",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "owner": "root",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage-default.conf"
*******************************************
+ File[/etc/nginx/sites-enabled/nginx-swh-objstorage.conf] =>
   parameters =>
     "ensure": "link",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "owner": "root",
     "target": "/etc/nginx/sites-available/nginx-swh-objstorage.conf"
*******************************************
+ File[/etc/softwareheritage/objstorage/server.yml] =>
   parameters =>
     "content": "---\nobjstorage:\n  cls: multiplexer\n  args:\n    objstorages:\...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0640",
     "notify": "Service[gunicorn-swh-objstorage]",
     "owner": "root"
*******************************************
+ File[/etc/softwareheritage/objstorage] =>
   parameters =>
     "ensure": "directory",
     "group": "swhstorage",
     "mode": "0750",
     "owner": "root"
*******************************************
+ File[/etc/ssl/certs/letsencrypt/objstorage_production/cert.pem] =>
   parameters =>
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "source": "puppet:///le_certs/objstorage_production/cert.pem"
*******************************************
+ File[/etc/ssl/certs/letsencrypt/objstorage_production/chain.pem] =>
   parameters =>
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "source": "puppet:///le_certs/objstorage_production/chain.pem"
*******************************************
+ File[/etc/ssl/certs/letsencrypt/objstorage_production/fullchain.pem] =>
   parameters =>
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "source": "puppet:///le_certs/objstorage_production/fullchain.pem"
*******************************************
+ File[/etc/ssl/certs/letsencrypt/objstorage_production/privkey.pem] =>
   parameters =>
     "ensure": "present",
     "group": "root",
     "mode": "0600",
     "owner": "root",
     "source": "puppet:///le_certs/objstorage_production/privkey.pem"
*******************************************
+ File[/etc/ssl/certs/letsencrypt/objstorage_production] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "mode": "0755",
     "owner": "root"
*******************************************
+ File[/etc/systemd/system/gunicorn-swh-objstorage.service] =>
   parameters =>
     "content": "# File managed by puppet (class gunicorn::instance swh-objstorag...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/tmpfiles.d/gunicorn-swh-objstorage.conf] =>
   parameters =>
     "ensure": "absent",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Tmpfiles]",
     "owner": "root"
*******************************************
+ File[/etc/varnish/includes/01_objstorage.vcl] =>
   parameters =>
     "content": "# backend_default.vcl\n#\n# Default backend definition.\n#\n# Fi...
     "group": "root",
     "mode": "0644",
     "notify": "Exec[vcl_reload]",
     "owner": "root"
*******************************************
+ File[/etc/varnish/includes/50_vhost_objstorage.softwareheritage.org.vcl] =>
   parameters =>
     "content": "# vhost_objstorage.softwareheritage.org.vcl\n#\n# Settings for t...
     "group": "root",
     "mode": "0644",
     "notify": "Exec[vcl_reload]",
     "owner": "root"
*******************************************
+ Gunicorn::Instance[swh-objstorage] =>
   parameters =>
     "config_base_module": "swh.core.api.gunicorn_config",
     "config_mode": "0644",
     "ensure": "enabled",
     "environment": {
       "SWH_CONFIG_FILENAME": "/etc/softwareheritage/objstorage/server.yml",
       "SWH_LOG_TARGET": "journal",
       "SWH_SENTRY_DSN": "https://swh::deploy::objstorage::sentry_token@sentry.so...
       "SWH_SENTRY_ENVIRONMENT": "production",
       "SWH_MAIN_PACKAGE": "swh.objstorage"
     },
     "executable": "swh.objstorage.api.server:make_app_from_configfile()",
     "group": "swhstorage",
     "log_only_errors": true,
     "settings": {
       "bind": "unix:/run/gunicorn/swh-objstorage/gunicorn.sock",
       "workers": 4,
       "worker_class": "aiohttp.worker.GunicornWebWorker",
       "timeout": 3600,
       "graceful_timeout": 3600,
       "keepalive": 5,
       "max_requests": 0,
       "max_requests_jitter": 0,
       "statsd_host": "127.0.0.1:8125",
       "statsd_prefix": "swh-objstorage"
     },
     "user": "swhstorage"
*******************************************
+ Hitch::Domain[objstorage_production] =>
   parameters =>
     "cacert_source": "/etc/ssl/certs/letsencrypt/objstorage_production/chain.pem...
     "cert_source": "/etc/ssl/certs/letsencrypt/objstorage_production/cert.pem",
     "default": false,
     "ensure": "present",
     "key_source": "/etc/ssl/certs/letsencrypt/objstorage_production/privkey.pem"...
*******************************************
+ Nginx::Resource::Location[nginx-swh-objstorage-default-default] =>
   parameters =>
     "add_header": {
     },
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "flv": false,
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "internal": false,
     "location": "/",
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "mp4": false,
     "notify": "Class[Nginx::Service]",
     "priority": 500,
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_ignore_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "90s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "rewrite_rules": [
    
     ],
     "server": "nginx-swh-objstorage-default",
     "ssl": false,
     "ssl_only": false,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Location[nginx-swh-objstorage-default] =>
   parameters =>
     "add_header": {
     },
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "flv": false,
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "internal": false,
     "location": "/",
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "mp4": false,
     "notify": "Class[Nginx::Service]",
     "priority": 500,
     "proxy": "http://swh-objstorage-gunicorn",
     "proxy_buffering": "off",
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_ignore_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "3600s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "rewrite_rules": [
    
     ],
     "server": "nginx-swh-objstorage",
     "ssl": false,
     "ssl_only": false,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Server[nginx-swh-objstorage-default] =>
   parameters =>
     "add_header": {
     },
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "format_log": "combined",
     "geo_mappings": {
     },
     "group": "root",
     "http2": "off",
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "ipv6_enable": false,
     "ipv6_listen_ip": "::",
     "ipv6_listen_options": "default ipv6only=on",
     "ipv6_listen_port": 80,
     "listen_ip": "moma.internal.softwareheritage.org",
     "listen_options": "default_server",
     "listen_port": 5003,
     "listen_unix_socket": "/var/run/nginx.sock",
     "listen_unix_socket_enable": false,
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "locations": {
     },
     "locations_defaults": {
     },
     "maintenance": true,
     "maintenance_value": "return 444",
     "mode": "0644",
     "owner": "root",
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "90s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "resolver": [
    
     ],
     "rewrite_non_www_to_www": false,
     "rewrite_rules": [
    
     ],
     "rewrite_www_to_non_www": false,
     "server_name": [
       "nginx-swh-objstorage-default"
     ],
     "spdy": "off",
     "ssl": false,
     "ssl_cache": "shared:SSL:10m",
     "ssl_ciphers": "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:EC...
     "ssl_listen_option": true,
     "ssl_port": 443,
     "ssl_prefer_server_ciphers": "on",
     "ssl_protocols": "TLSv1 TLSv1.1 TLSv1.2",
     "ssl_redirect": false,
     "ssl_session_timeout": "5m",
     "ssl_stapling": false,
     "ssl_stapling_verify": false,
     "ssl_verify_client": "on",
     "string_mappings": {
     },
     "use_default_location": true,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Server[nginx-swh-objstorage] =>
   parameters =>
     "add_header": {
     },
     "client_max_body_size": "4G",
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "format_log": "combined if=$error_status",
     "geo_mappings": {
     },
     "group": "root",
     "http2": "off",
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "ipv6_enable": false,
     "ipv6_listen_ip": "::",
     "ipv6_listen_options": "default ipv6only=on",
     "ipv6_listen_port": 80,
     "listen_ip": "moma.internal.softwareheritage.org",
     "listen_options": "deferred",
     "listen_port": 5003,
     "listen_unix_socket": "/var/run/nginx.sock",
     "listen_unix_socket_enable": false,
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "locations": {
     },
     "locations_defaults": {
     },
     "maintenance": false,
     "maintenance_value": "return 503",
     "mode": "0644",
     "owner": "root",
     "proxy": "http://swh-objstorage-gunicorn",
     "proxy_buffering": "off",
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "3600s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "resolver": [
    
     ],
     "rewrite_non_www_to_www": false,
     "rewrite_rules": [
    
     ],
     "rewrite_www_to_non_www": false,
     "server_name": [
       "objstorage.softwarehritage.org objstorage.internal.softwareheritage.org",...
       "moma.internal.softwareheritage.org",
       "moma",
       "127.0.0.1",
       "localhost",
       "::1"
     ],
     "spdy": "off",
     "ssl": false,
     "ssl_cache": "shared:SSL:10m",
     "ssl_ciphers": "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:EC...
     "ssl_listen_option": true,
     "ssl_port": 443,
     "ssl_prefer_server_ciphers": "on",
     "ssl_protocols": "TLSv1 TLSv1.1 TLSv1.2",
     "ssl_redirect": false,
     "ssl_session_timeout": "5m",
     "ssl_stapling": false,
     "ssl_stapling_verify": false,
     "ssl_verify_client": "on",
     "string_mappings": {
     },
     "use_default_location": true,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Upstream::Member[gunicorn-objstorage] =>
   parameters =>
     "backup": false,
     "context": "http",
     "ensure": "present",
     "port": 80,
     "resolve": false,
     "server": "unix:/run/gunicorn/swh-objstorage/gunicorn.sock",
     "upstream": "swh-objstorage-gunicorn"
*******************************************
+ Nginx::Resource::Upstream[swh-objstorage-gunicorn] =>
   parameters =>
     "cfg_append": {
     },
     "cfg_prepend": {
     },
     "context": "http",
     "ensure": "present",
     "ip_hash": false,
     "least_conn": false,
     "member_defaults": {
     },
     "members": {
       "gunicorn-objstorage": {
         "server": "unix:/run/gunicorn/swh-objstorage/gunicorn.sock"
       }
     },
     "ntlm": false
*******************************************
+ Package[python3-swh.objstorage.cloud] =>
   parameters =>
     "ensure": "installed"
*******************************************
+ Package[python3-swh.objstorage] =>
   parameters =>
     "ensure": "present",
     "notify": [
       "Service[gunicorn-swh-objstorage]"
     ]
*******************************************
+ Profile::Hitch::Ssl_cert[objstorage_production] =>
   parameters =>
     "ssl_cert_name": "objstorage_production"
*******************************************
+ Profile::Letsencrypt::Certificate[objstorage_production] =>
   parameters =>
     "basename": "objstorage_production",
     "privkey_group": "root",
     "privkey_mode": "0600",
     "privkey_owner": "root"
*******************************************
+ Profile::Swh::Deploy::Rpc_server[objstorage] =>
   parameters =>
     "config_key": "objstorage",
     "executable": "swh.objstorage.api.server:make_app_from_configfile()",
     "gunicorn_config_base_module": "swh.core.api.gunicorn_config",
     "http_check_string": "SWH Objstorage API server",
     "instance_name": "objstorage",
     "worker": "async"
*******************************************
+ Profile::Varnish::Vcl_include[objstorage] =>
   parameters =>
     "basename": "objstorage",
     "content": "# backend_default.vcl\n#\n# Default backend definition.\n#\n# Fi...
     "order": "01"
*******************************************
+ Profile::Varnish::Vcl_include[vhost_objstorage.softwareheritage.org] =>
   parameters =>
     "basename": "vhost_objstorage.softwareheritage.org",
     "content": "# vhost_objstorage.softwareheritage.org.vcl\n#\n# Settings for t...
     "order": "50"
*******************************************
  Profile::Varnish::Vhost[archive.softwareheritage.org] =>
   parameters =>
     basic_auth =>
      + false
*******************************************
  Profile::Varnish::Vhost[deposit.softwareheritage.org] =>
   parameters =>
     basic_auth =>
      + false
*******************************************
+ Profile::Varnish::Vhost[objstorage.softwareheritage.org] =>
   parameters =>
     "aliases": [
       "objstorage.internal.softwareheritage.org"
     ],
     "backend_http_host": "moma.internal.softwareheritage.org",
     "backend_http_port": "5003",
     "backend_name": "objstorage",
     "basic_auth": true,
     "basic_auth_strings": [
       "c3doLXByb2Q6c3doOjpkZXBsb3k6Om9ianN0b3JhZ2U6OnJldmVyc2VfcHJveHk6OmJhc2ljX...
       "ZW5lYS1wcm9kOnN3aDo6ZGVwbG95OjpvYmpzdG9yYWdlOjpyZXZlcnNlX3Byb3h5OjpiYXNpY...
     ],
     "hsts_max_age": 15768000,
     "order": "50",
     "servername": "objstorage.softwareheritage.org",
     "websocket_support": false
*******************************************
+ Service[gunicorn-swh-objstorage] =>
   parameters =>
     "enable": true,
     "ensure": "running",
     "restart": "/bin/systemctl reload gunicorn-swh-objstorage.service"
*******************************************
+ Systemd::Tmpfile[gunicorn-swh-objstorage.conf] =>
   parameters =>
     "ensure": "absent",
     "filename": "gunicorn-swh-objstorage.conf",
     "path": "/etc/tmpfiles.d"
*******************************************
+ Systemd::Unit_file[gunicorn-swh-objstorage.service] =>
   parameters =>
     "content": "# File managed by puppet (class gunicorn::instance swh-objstorag...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[gunicorn-swh-objstorage]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Varnish::Vcl[/etc/varnish/includes/01_objstorage.vcl] =>
   parameters =>
     "content": "# backend_default.vcl\n#\n# Default backend definition.\n#\n# Fi...
     "file": "/etc/varnish/includes/01_objstorage.vcl"
*******************************************
+ Varnish::Vcl[/etc/varnish/includes/50_vhost_objstorage.softwareheritage.org.vcl] =>
   parameters =>
     "content": "# vhost_objstorage.softwareheritage.org.vcl\n#\n# Settings for t...
     "file": "/etc/varnish/includes/50_vhost_objstorage.softwareheritage.org.vcl"...
*******************************************
*** End octocatalog-diff on moma.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier requested review of this revision.Oct 8 2021, 5:14 PM
vsellier created this revision.
Harbormaster completed remote builds in B24339: Diff 23423.Oct 8 2021, 5:14 PM
olasd mentioned this in T3621: Create a production read-only objstorage.Oct 8 2021, 5:32 PM
vsellier planned changes to this revision.Oct 18 2021, 3:39 PM
vsellier updated this revision to Diff 23773.Oct 22 2021, 1:37 PM

rebase

Harbormaster completed remote builds in B24658: Diff 23773.Oct 22 2021, 1:37 PM
vsellier updated this revision to Diff 23778.Oct 22 2021, 3:55 PM

Add basic authentication support and activate it in staging too
(main description will be updated accordingly)

Harbormaster completed remote builds in B24663: Diff 23778.Oct 22 2021, 3:55 PM
vsellier edited the summary of this revision. (Show Details)Oct 22 2021, 3:57 PM
vsellier edited the test plan for this revision. (Show Details)
vsellier edited the test plan for this revision. (Show Details)Oct 22 2021, 4:01 PM
vsellier updated this revision to Diff 23779.Oct 22 2021, 4:02 PM

fix a typo

Harbormaster completed remote builds in B24664: Diff 23779.Oct 22 2021, 4:02 PM
olasd accepted this revision.Nov 2 2021, 11:44 AM
olasd added a subscriber: olasd.

Nice, thanks!

This revision is now accepted and ready to land.Nov 2 2021, 11:44 AM
vsellier updated this revision to Diff 23960.Nov 2 2021, 5:27 PM

rebase

Harbormaster completed remote builds in B24837: Diff 23960.Nov 2 2021, 5:27 PM
Closed by commit rSPSITE4f19b14f25c0: Deploy a read-only objstorage on moma (authored by vsellier). · Explain WhyNov 2 2021, 5:30 PM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rSPSITE4f19b14f25c0: Deploy a read-only objstorage on moma.

Revision Contents
Changeset List

PathSize
data/
common/
10 lines
deployments/
staging/
4 lines
hostname/
25 lines
subnets/
2 lines
site-modules/
profile/
manifests/
swh/
deploy/
30 lines
varnish/
2 lines
templates/
varnish/
9 lines
role/
manifests/
2 lines

Diff 23961

View Options

data/common/common.yaml

Loading...
View Options

data/deployments/staging/common.yaml

Loading...
View Options

data/hostname/moma.softwareheritage.org.yaml

Loading...
View Options

data/subnets/vagrant.yaml

Loading...
View Options

site-modules/profile/manifests/swh/deploy/reverse_proxy.pp

Loading...
View Options

site-modules/profile/manifests/varnish/vhost.pp

Loading...
View Options

site-modules/profile/templates/varnish/vhost.vcl.erb

Loading...
View Options

site-modules/role/manifests/swh_rp_webapps.pp

Loading...
About · Developer information · Legal