Page MenuHomeSoftware Heritage
Differential D4963

webapp1: use the same deployment pattern than moma
ClosedPublic
Actions

Authored by vsellier on Jan 28 2021, 3:10 PM.

Details

Reviewers
ardumont
Group Reviewers
System administrators
Commits
rSPSITEb82b0d93c2ec: webapp1: use the same deployment pattern than moma
Summary

It will allow to update the storage locally without interrupting
the service on saam

Test Plan
diff origin/production/webapp1.internal.softwareheritage.org current/webapp1.internal.softwareheritage.org
*******************************************
+ Concat::Fragment[nginx-swh-storage-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    proxy_pass            http://swh-storage-g...
     "order": 500,
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ Concat::Fragment[nginx-swh-storage-default-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    index     index.html index.htm index.php;\...
     "order": 500,
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ Concat::Fragment[nginx-swh-storage-default-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ Concat::Fragment[nginx-swh-storage-default-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen 127.0.0.1:5002 default_s...
     "order": "001",
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ Concat::Fragment[nginx-swh-storage-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ Concat::Fragment[nginx-swh-storage-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen 127.0.0.1:5002 deferred;...
     "order": "001",
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ Concat::Fragment[swh-storage-gunicorn_upstream_footer] =>
   parameters =>
     "content": "}\n",
     "order": "90",
     "target": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf"
*******************************************
+ Concat::Fragment[swh-storage-gunicorn_upstream_header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nupstream swh-storage-gunicorn {\n",
     "order": "10",
     "target": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf"
*******************************************
+ Concat::Fragment[swh-storage-gunicorn_upstream_member_gunicorn-storage] =>
   parameters =>
     "content": "  server unix:/run/gunicorn/swh-storage/gunicorn.sock;\n",
     "order": 40,
     "target": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf"
*******************************************
+ Concat[/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat[/etc/nginx/sites-available/nginx-swh-storage-default.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/nginx/sites-available/nginx-swh-storage-default.conf",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat[/etc/nginx/sites-available/nginx-swh-storage.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure": "present",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "order": "alpha",
     "owner": "root",
     "path": "/etc/nginx/sites-available/nginx-swh-storage.conf",
     "replace": true,
     "show_diff": true,
     "warn": false
*******************************************
+ Concat_file[/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_nginx_conf.d_swh-storage-gunicorn-upstream.conf"
*******************************************
+ Concat_file[/etc/nginx/sites-available/nginx-swh-storage-default.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_nginx_sites-available_nginx-swh-storage-default.conf"
*******************************************
+ Concat_file[/etc/nginx/sites-available/nginx-swh-storage.conf] =>
   parameters =>
     "backup": "puppet",
     "ensure_newline": false,
     "force": false,
     "format": "plain",
     "group": "root",
     "mode": "0644",
     "order": "alpha",
     "owner": "root",
     "replace": true,
     "show_diff": true,
     "tag": "_etc_nginx_sites-available_nginx-swh-storage.conf"
*******************************************
+ Concat_fragment[nginx-swh-storage-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    proxy_pass            http://swh-storage-g...
     "order": 500,
     "tag": "_etc_nginx_sites-available_nginx-swh-storage.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ Concat_fragment[nginx-swh-storage-default-500-6666cd76f96956469e7be39d750cc7d9] =>
   parameters =>
     "content": "\n  location / {\n    index     index.html index.htm index.php;\...
     "order": 500,
     "tag": "_etc_nginx_sites-available_nginx-swh-storage-default.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ Concat_fragment[nginx-swh-storage-default-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "tag": "_etc_nginx_sites-available_nginx-swh-storage-default.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ Concat_fragment[nginx-swh-storage-default-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen 127.0.0.1:5002 default_s...
     "order": "001",
     "tag": "_etc_nginx_sites-available_nginx-swh-storage-default.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ Concat_fragment[nginx-swh-storage-footer] =>
   parameters =>
     "content": "}\n",
     "order": "699",
     "tag": "_etc_nginx_sites-available_nginx-swh-storage.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ Concat_fragment[nginx-swh-storage-header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nserver {\n  listen 127.0.0.1:5002 deferred;...
     "order": "001",
     "tag": "_etc_nginx_sites-available_nginx-swh-storage.conf",
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ Concat_fragment[swh-storage-gunicorn_upstream_footer] =>
   parameters =>
     "content": "}\n",
     "order": "90",
     "tag": "_etc_nginx_conf.d_swh-storage-gunicorn-upstream.conf",
     "target": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf"
*******************************************
+ Concat_fragment[swh-storage-gunicorn_upstream_header] =>
   parameters =>
     "content": "# MANAGED BY PUPPET\nupstream swh-storage-gunicorn {\n",
     "order": "10",
     "tag": "_etc_nginx_conf.d_swh-storage-gunicorn-upstream.conf",
     "target": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf"
*******************************************
+ Concat_fragment[swh-storage-gunicorn_upstream_member_gunicorn-storage] =>
   parameters =>
     "content": "  server unix:/run/gunicorn/swh-storage/gunicorn.sock;\n",
     "order": 40,
     "tag": "_etc_nginx_conf.d_swh-storage-gunicorn-upstream.conf",
     "target": "/etc/nginx/conf.d/swh-storage-gunicorn-upstream.conf"
*******************************************
+ File[/etc/gunicorn/instances/swh-storage.cfg] =>
   parameters =>
     "content": "# Gunicorn instance configuration.\n# Managed by puppet (class g...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0644",
     "notify": "Service[gunicorn-swh-storage]",
     "owner": "swhstorage"
*******************************************
+ File[/etc/nginx/sites-enabled/nginx-swh-storage-default.conf] =>
   parameters =>
     "ensure": "link",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "owner": "root",
     "target": "/etc/nginx/sites-available/nginx-swh-storage-default.conf"
*******************************************
+ File[/etc/nginx/sites-enabled/nginx-swh-storage.conf] =>
   parameters =>
     "ensure": "link",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Nginx::Service]",
     "owner": "root",
     "target": "/etc/nginx/sites-available/nginx-swh-storage.conf"
*******************************************
+ File[/etc/softwareheritage/storage/storage.yml] =>
   parameters =>
     "content": "---\nstorage:\n  cls: local\n  args:\n    db: host=db.internal.s...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0640",
     "notify": "Service[gunicorn-swh-storage]",
     "owner": "root"
*******************************************
+ File[/etc/softwareheritage/storage] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "mode": "0755",
     "owner": "root"
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -6,5 +6,5 @@
         cls: remote
         args:
      -    url: http://saam.internal.softwareheritage.org:5002/
      +    url: http://localhost:5002/
       vault:
         cls: remote
*******************************************
+ File[/etc/systemd/system/gunicorn-swh-storage.service] =>
   parameters =>
     "content": "# File managed by puppet (class gunicorn::instance swh-storage),...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/tmpfiles.d/gunicorn-swh-storage.conf] =>
   parameters =>
     "ensure": "absent",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Tmpfiles]",
     "owner": "root"
*******************************************
+ Gunicorn::Instance[swh-storage] =>
   parameters =>
     "config_base_module": "swh.core.api.gunicorn_config",
     "config_mode": "0644",
     "ensure": "enabled",
     "environment": {
       "SWH_CONFIG_FILENAME": "/etc/softwareheritage/storage/storage.yml",
       "SWH_LOG_TARGET": "journal",
       "SWH_SENTRY_DSN": "https://swh::deploy::storage::sentry_token@sentry.softw...
       "SWH_SENTRY_ENVIRONMENT": "production",
       "SWH_MAIN_PACKAGE": "swh.storage"
     },
     "executable": "swh.storage.api.server:make_app_from_configfile()",
     "group": "swhstorage",
     "log_only_errors": true,
     "settings": {
       "bind": "unix:/run/gunicorn/swh-storage/gunicorn.sock",
       "workers": 16,
       "worker_class": "sync",
       "timeout": 3600,
       "graceful_timeout": 3600,
       "keepalive": 5,
       "max_requests": 10000,
       "max_requests_jitter": 1000,
       "statsd_host": "127.0.0.1:8125",
       "statsd_prefix": "swh-storage"
     },
     "user": "swhstorage"
*******************************************
+ Nginx::Resource::Location[nginx-swh-storage-default-default] =>
   parameters =>
     "add_header": {
     },
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "flv": false,
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "internal": false,
     "location": "/",
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "mp4": false,
     "notify": "Class[Nginx::Service]",
     "priority": 500,
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_ignore_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "90s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "rewrite_rules": [
    
     ],
     "server": "nginx-swh-storage-default",
     "ssl": false,
     "ssl_only": false,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Location[nginx-swh-storage-default] =>
   parameters =>
     "add_header": {
     },
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "flv": false,
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "internal": false,
     "location": "/",
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "mp4": false,
     "notify": "Class[Nginx::Service]",
     "priority": 500,
     "proxy": "http://swh-storage-gunicorn",
     "proxy_buffering": "on",
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_ignore_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "3600s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "rewrite_rules": [
    
     ],
     "server": "nginx-swh-storage",
     "ssl": false,
     "ssl_only": false,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Server[nginx-swh-storage-default] =>
   parameters =>
     "add_header": {
     },
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "format_log": "combined",
     "geo_mappings": {
     },
     "group": "root",
     "http2": "off",
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "ipv6_enable": false,
     "ipv6_listen_ip": "::",
     "ipv6_listen_options": "default ipv6only=on",
     "ipv6_listen_port": 80,
     "listen_ip": "127.0.0.1",
     "listen_options": "default_server",
     "listen_port": 5002,
     "listen_unix_socket": "/var/run/nginx.sock",
     "listen_unix_socket_enable": false,
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "locations": {
     },
     "locations_defaults": {
     },
     "maintenance": true,
     "maintenance_value": "return 444",
     "mode": "0644",
     "owner": "root",
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "90s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "resolver": [
    
     ],
     "rewrite_non_www_to_www": false,
     "rewrite_rules": [
    
     ],
     "rewrite_www_to_non_www": false,
     "server_name": [
       "nginx-swh-storage-default"
     ],
     "spdy": "off",
     "ssl": false,
     "ssl_cache": "shared:SSL:10m",
     "ssl_ciphers": "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:EC...
     "ssl_listen_option": true,
     "ssl_port": 443,
     "ssl_prefer_server_ciphers": "on",
     "ssl_protocols": "TLSv1 TLSv1.1 TLSv1.2",
     "ssl_redirect": false,
     "ssl_session_timeout": "5m",
     "ssl_stapling": false,
     "ssl_stapling_verify": false,
     "ssl_verify_client": "on",
     "string_mappings": {
     },
     "use_default_location": true,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Server[nginx-swh-storage] =>
   parameters =>
     "add_header": {
     },
     "client_max_body_size": "4G",
     "ensure": "present",
     "fastcgi_params": "/etc/nginx/fastcgi.conf",
     "format_log": "combined if=$error_status",
     "geo_mappings": {
     },
     "group": "root",
     "http2": "off",
     "index_files": [
       "index.html",
       "index.htm",
       "index.php"
     ],
     "ipv6_enable": false,
     "ipv6_listen_ip": "::",
     "ipv6_listen_options": "default ipv6only=on",
     "ipv6_listen_port": 80,
     "listen_ip": "127.0.0.1",
     "listen_options": "deferred",
     "listen_port": 5002,
     "listen_unix_socket": "/var/run/nginx.sock",
     "listen_unix_socket_enable": false,
     "location_allow": [
    
     ],
     "location_deny": [
    
     ],
     "locations": {
     },
     "locations_defaults": {
     },
     "maintenance": false,
     "maintenance_value": "return 503",
     "mode": "0644",
     "owner": "root",
     "proxy": "http://swh-storage-gunicorn",
     "proxy_buffering": "on",
     "proxy_connect_timeout": "90s",
     "proxy_hide_header": [
    
     ],
     "proxy_pass_header": [
    
     ],
     "proxy_read_timeout": "3600s",
     "proxy_send_timeout": "90s",
     "proxy_set_header": [
       "Host $host",
       "X-Real-IP $remote_addr",
       "X-Forwarded-For $proxy_add_x_forwarded_for",
       "Proxy \"\""
     ],
     "resolver": [
    
     ],
     "rewrite_non_www_to_www": false,
     "rewrite_rules": [
    
     ],
     "rewrite_www_to_non_www": false,
     "server_name": [
       "webapp1.internal.softwareheritage.org",
       "webapp1",
       "127.0.0.1",
       "localhost",
       "::1"
     ],
     "spdy": "off",
     "ssl": false,
     "ssl_cache": "shared:SSL:10m",
     "ssl_ciphers": "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:EC...
     "ssl_listen_option": true,
     "ssl_port": 443,
     "ssl_prefer_server_ciphers": "on",
     "ssl_protocols": "TLSv1 TLSv1.1 TLSv1.2",
     "ssl_redirect": false,
     "ssl_session_timeout": "5m",
     "ssl_stapling": false,
     "ssl_stapling_verify": false,
     "ssl_verify_client": "on",
     "string_mappings": {
     },
     "use_default_location": true,
     "uwsgi_params": "/etc/nginx/uwsgi_params"
*******************************************
+ Nginx::Resource::Upstream::Member[gunicorn-storage] =>
   parameters =>
     "backup": false,
     "context": "http",
     "ensure": "present",
     "port": 80,
     "resolve": false,
     "server": "unix:/run/gunicorn/swh-storage/gunicorn.sock",
     "upstream": "swh-storage-gunicorn"
*******************************************
+ Nginx::Resource::Upstream[swh-storage-gunicorn] =>
   parameters =>
     "cfg_append": {
     },
     "cfg_prepend": {
     },
     "context": "http",
     "ensure": "present",
     "ip_hash": false,
     "least_conn": false,
     "member_defaults": {
     },
     "members": {
       "gunicorn-storage": {
         "server": "unix:/run/gunicorn/swh-storage/gunicorn.sock"
       }
     },
     "ntlm": false
*******************************************
+ Package[python3-swh.storage] =>
   parameters =>
     "ensure": "present",
     "notify": [
       "Profile::Swh::Deploy::Rpc_server[storage]"
     ]
*******************************************
+ Profile::Swh::Deploy::Rpc_server[storage] =>
   parameters =>
     "config_key": "storage",
     "executable": "swh.storage.api.server:make_app_from_configfile()",
     "gunicorn_config_base_module": "swh.core.api.gunicorn_config",
     "http_check_string": "<title>Software Heritage storage server</title>",
     "instance_name": "storage",
     "worker": "sync"
*******************************************
+ Service[gunicorn-swh-storage] =>
   parameters =>
     "enable": true,
     "ensure": "running",
     "restart": "/bin/systemctl reload gunicorn-swh-storage.service"
*******************************************
+ Systemd::Tmpfile[gunicorn-swh-storage.conf] =>
   parameters =>
     "ensure": "absent",
     "path": "/etc/tmpfiles.d"
*******************************************
+ Systemd::Unit_file[gunicorn-swh-storage.service] =>
   parameters =>
     "content": "# File managed by puppet (class gunicorn::instance swh-storage),...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[gunicorn-swh-storage]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
*** End octocatalog-diff on webapp1.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier requested review of this revision.Jan 28 2021, 3:10 PM
vsellier created this revision.
Harbormaster completed remote builds in B18853: Diff 17712.Jan 28 2021, 3:10 PM
ardumont accepted this revision.Jan 28 2021, 3:13 PM
ardumont added a subscriber: ardumont.

lgtm

That installs a similar pattern as the other production node (moma regarding webapp/storage deployment)

This revision is now accepted and ready to land.Jan 28 2021, 3:13 PM
Closed by commit rSPSITEb82b0d93c2ec: webapp1: use the same deployment pattern than moma (authored by vsellier). · Explain WhyJan 28 2021, 3:18 PM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rSPSITEb82b0d93c2ec: webapp1: use the same deployment pattern than moma.

Revision Contents
Changeset List

Diff 17713

View Options

data/hostname/webapp1.internal.softwareheritage.org.yaml

Loading...
View Options

manifests/site.pp

Loading...
View Options

site-modules/role/manifests/swh_rp_webapp_with_swh_search.pp

Loading...
View Options

site-modules/role/manifests/swh_rp_webapp_with_swh_search_and_storage.pp

Loading...
About · Developer information · Legal