Page MenuHomeSoftware Heritage

Memcached seems to be automatically cleared pretty often on moma
Closed, MigratedEdits Locked

Description

On moma, the server where the production webapp is hosted, there is an automatic run of puppet on a regular basis.

When puppet runs on moma, we can see that the memcached configuration is updated each time which will trigger
a refresh of the memcached service and thus clear the current webapp cache.

*** swh-deploy: starting test run on moma.internal.softwareheritage.org...
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Applying configuration version '1638972115'
Notice: /Stage[main]/Memcached/File[/etc/memcached.conf]/content: 
--- /etc/memcached.conf 2021-12-08 13:35:21.435550583 +0000
+++ /tmp/puppet-file20211208-3773096-k2mx4      2021-12-08 14:02:09.969214287 +0000
@@ -10,7 +10,7 @@
  logfile /var/log/memcached.log
  
  # Use <num> MB memory max to use for object storage.
--m 1235
+-m 1176
  
  
  

Notice: /Stage[main]/Memcached/File[/etc/memcached.conf]/content: current_value '{md5}6cd8b2c619c53eae5f6bff188e35ccf2', should be '{md5}83b44ab40734309b38aa6d55d877f106' (noop)
Info: /Stage[main]/Memcached/File[/etc/memcached.conf]: Scheduling refresh of Service[memcached]
Notice: /Stage[main]/Memcached/Service[memcached]: Would have triggered 'refresh' from 1 event
Notice: Class[Memcached]: Would have triggered 'refresh' from 2 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 8.81 seconds
Connection to moma.internal.softwareheritage.org closed.

This is problematic for the authentication feature of the webapp as OpenID Connect profiles of users
are stored in the django cache backed by memcached.

As a consequence, when the memcached is cleared users are automatically logout and they must click on the
login link again to authenticate again, which is not really convenient.

We should either stop touching the memcached config when running puppet or disable the automatic puppet run
on moma to fix that issue.

Event Timeline

anlambert created this task.

... or disable the automatic puppet run on moma to fix that issue.

That would create discrepancy between nodes which is not a good thing. Plus, that won't
solve the current problem. That is, when we'll run puppet ourselves, that will create
the problem described nonetheless.

We should either stop touching the memcached config when running puppet ...

I'd say that the proper fix. I'll have a look.

Thanks for raising concerns about this.

The fix is deployed. This should no longer be a bother from now on [1].

Note:
"Of course", one change occurred from 1224 to 1221 very very recently since i opened and landed and deployed the diff...
So people may have been disconnected again about now.

...
--m 1221
+-m 1224
...
Notice: Applied catalog in 9.01 seconds
ardumont changed the task status from Open to Work in Progress.Dec 8 2021, 4:44 PM
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.