Page MenuHomeSoftware Heritage

docker: Install save code now refresh status cron
AbandonedPublic

Authored by ardumont on Apr 26 2021, 6:18 PM.

Details

Summary

Without the cron, as currently, the fresh new build with web 0.0.302, new save code now
requests are not updated anymore through the ui. The following makes sure the status are
updated periodically.

This is the equivalent of what's been deployed in production [2]

[2] D5616

Related to T3280

Test Plan

docker-compose up, do some save code now requests and amazingly check the status gets
updated when browsing through pagination:

swh-doco exec swh-web-cron /bin/bash
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-web-cron /bin/bash
root@8b20f39d055c:/# tail -f /var/log/refresh-statuses.log
+ SWH_CONFIG_FILENAME=/web.yml
+ /srv/softwareheritage/venv/bin/django-admin refresh_savecodenow_statuses
/srv/softwareheritage/venv/lib/python3.7/site-packages/jose/backends/cryptography_backend.py:18: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes, int_to_bytes
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (1): swh-scheduler:5008
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._make_request:461 - http://swh-scheduler:5008 "POST /task/get HTTP/1.1" 200 497
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._make_request:461 - http://swh-scheduler:5008 "POST /task_run/get HTTP/1.1" 200 281
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (1): swh-storage:5002
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._make_request:461 - http://swh-storage:5002 "POST /origin/get HTTP/1.1" 200 3
Successfully updated 2 save request(s).

Diff Detail

Repository
rDENV Development environment
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 21053
Build 32674: arc lint + arc unit

Event Timeline

docker/docker-compose.yml
242

adding the cron as mountpoint here installs the file but no cron is running so no.

Finally made the damn thing work...

swh-doco exec swh-web-cron /bin/bash
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-web-cron /bin/bash
root@8b20f39d055c:/# tail -f /var/log/refresh-statuses.log
+ SWH_CONFIG_FILENAME=/web.yml
+ /srv/softwareheritage/venv/bin/django-admin refresh_savecodenow_statuses
/srv/softwareheritage/venv/lib/python3.7/site-packages/jose/backends/cryptography_backend.py:18: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes, int_to_bytes
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (1): swh-scheduler:5008
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._make_request:461 - http://swh-scheduler:5008 "POST /task/get HTTP/1.1" 200 497
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._make_request:461 - http://swh-scheduler:5008 "POST /task_run/get HTTP/1.1" 200 281
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (1): swh-storage:5002
[27/Apr/2021 14:58:04] [DEBUG] urllib3.connectionpool._make_request:461 - http://swh-storage:5002 "POST /origin/get HTTP/1.1" 200 3
Successfully updated 2 save request(s).
ardumont edited the test plan for this revision. (Show Details)
ardumont added inline comments.
docker/Dockerfile.cron
10

to avoid waiting on the first cron run to actually create it and be able to 'tail' it immediatly after connecting to the container.

docker/services/swh-web/refresh-savecodenow-statuses.cron
1

^ because it's hell to debug it otherwise...

ardumont retitled this revision from Install save code now refresh status cron to docker: Install save code now refresh status cron.Apr 27 2021, 5:05 PM

I think this would be simpler to deploy the cron in the same docker-compose service as the webapp.
The cron command could be set directly in the swh-web entrypoint.

You will surely need to modify cron.allow file for that.

docker/docker-compose.yml
248

Adding environment variable in the container does not transmit the environment variables required for the cron to actually work so not an option either...

docker/services/swh-web/refresh-savecodenow-statuses
6

thus the duplication here.

Note also that this is installed through the Dockerfile.cron during the image build.

I think this would be simpler to deploy the cron in the same docker-compose service as
the webapp. The cron command could be set directly in the swh-web entrypoint.

I tried that and it did not work [1] The cron did not run at all.

[1] D5615#inline-38988

You will surely need to modify cron.allow file for that.

do you mean adding the swh user to it ^
(i did not try although that'd mean modifying the main Dockerfile, ain't it?)

I think this would be simpler to deploy the cron in the same docker-compose service as the webapp.

> The cron command could be set directly in the swh-web entrypoint.

I tried that and it did not work [1] The cron did not run at all.

[1] D5615#inline-38988

You will surely need to modify cron.allow file for that.

do you mean adding the swh user to it ^
(i did not try although that'd mean modifying the main Dockerfile, ain't it?)

Yes, this way you should be able to use crontab in the swh-web entrypoint. There is also crontab -u that could be used.

Yes, this way you should be able to use crontab in the swh-web entrypoint. There
is also crontab -u that could be used.

I'll try now that i'm sure it can work in some way ;)

Yes, this way you should be able to use crontab in the swh-web entrypoint. There
is also crontab -u that could be used.

I'll try now that i'm sure it can work in some way ;)

So, i can't find a way to make your suggestion work properly.
See D5639 with the latest tryouts (changes out of this diff)...

Instead of adding a new docker-compose service, how about adding a background bash process that will call the refresh_savecodenow_statuses django command periodically ?

This can be setup in the swh-web service entrypoint the following way:

diff --git a/docker/services/swh-web/entrypoint.sh b/docker/services/swh-web/entrypoint.sh
index 2d0ef86..9fb843f 100755
--- a/docker/services/swh-web/entrypoint.sh
+++ b/docker/services/swh-web/entrypoint.sh
@@ -46,6 +46,16 @@ case "$1" in
         echo "Creating Django admin user"
         echo "$create_admin_script" | python3 -m swh.web.manage shell
 
+        (
+            while true
+            do
+                (date && django-admin refresh_savecodenow_statuses \
+                    --settings=${DJANGO_SETTINGS_MODULE} 2>&1) >> /tmp/refresh-statuses.log
+                sleep 5
+            done
+        ) &
+        disown
+
         echo "starting the swh-web server"
         exec gunicorn --bind 0.0.0.0:5004 \
              --threads 2 \

yep, i'll adapt tomorrow, thx.

Superseded by D5641

(only closed to keep reference alive)