In terms of RAM, we currently have a 1/20 ratio to the storage for the postgresql storage. If we want to keep the same ratio, that's 1.5TB of RAM for the cache. We also need at least 32+8GB/server for Cassandra itself, which is negligible. So that's 1.5TB of RAM total, which is more reasonable; so assuming 64GB sticks (because cheaper), that's 24 sticks, so we only need two servers to hold that much RAM.

Summary of a discussion on 2021-01-05, on using "HDD+fully loaded in RAM" vs "SSD":

Is this task still valid ?

Closing at it's not relevant anymore

Package swh.storage 0.23.1 built both for stable and unstable.

Now stable build is stuck on storage tests, the one using a journal.

Status, unstable build is now ok.

For the previous tryout to work, we need to exclude all unwanted jres.

Tryout in progress

Another suggestion which sounds more standard, debian build wise:

13:05 <+olasd> I think you can just add a `Build-Conflicts: openjdk-17-jre-headless`
13:06 <+olasd> which should make the sbuild dependency resolver avoid it altogether
13:10 <+ardumont> ack, i'll try

Another solution (to prevent hard-coding JAVA_HOME) is to invert the dependency order
currently defined in debian/rules.

Forcing JAVA_HOME to a jdk11

What changed from the last unstable build ok [1] to the new failing one [2], the jdk
versions pulled for the build changed.

D5063 is applied, the main webapp is now using swh-search by default.

The main webapp search can be switch from the sql search to the swh-search as all the tests performed on staging and https://webapp1.internal.softwareheritage.org are ok

storage-ok and ko computed out of the py3 installed: line in those output above:

$ diff storage-ok.txt storage-ko.txt
12c12
< confluent-kafka==1.5.0
---
> confluent-kafka==1.6.0
21c21
< idna==2.10
---
> idna==3.1

storage-ok:

py3 installed: aiohttp==3.7.3
aiohttp-utils==3.1.1
apipkg==1.5
async-timeout==3.0.1
attrs==20.3.0
attrs-strict==0.2.0
blinker==1.4
cassandra-driver==3.24.0
certifi==2020.12.5
chardet==3.0.4
click==7.1.2
confluent-kafka==1.5.0
coverage==5.4
decorator==4.4.2
Deprecated==1.2.11
execnet==1.8.0
Flask==1.1.2
geomet==0.2.1.post1
gunicorn==20.0.4
hypothesis==5.49.0
idna==2.10
importlib-metadata==3.4.0
iniconfig==1.1.1
iso8601==0.1.13
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==1.1.1
mirakuru==2.3.0
msgpack==1.0.2
multidict==5.1.0
mypy==0.800
mypy-extensions==0.4.3
packaging==20.9
pluggy==0.13.1
port-for==0.4
psutil==5.8.0
psycopg2==2.8.6
py==1.10.0
pyparsing==2.4.7
pytest==6.2.2
pytest-cov==2.11.1
pytest-forked==1.3.0
pytest-mock==3.5.1
pytest-postgresql==2.5.3
pytest-xdist==2.2.0
python-dateutil==2.8.1
python-mimeparse==1.6.0
pytz==2021.1
PyYAML==5.4.1
requests==2.25.1
sentry-sdk==0.19.5
six==1.15.0
sortedcontainers==2.3.0
sqlalchemy-stubs==0.4
swh.core==0.11.0
swh.journal==0.7.0
swh.model==0.12.0
swh.objstorage==0.2.2
swh.storage @ file:///home/tony/work/inria/repo/swh/swh-environment/swh-storage/.tox/.tmp/package/1/swh.storage-0.22.1.dev7%2Bg89cf1e51.zip
tenacity==6.3.1
toml==0.10.2
typed-ast==1.4.2
typing-extensions==3.7.4.3
urllib3==1.26.3
Werkzeug==1.0.1
wrapt==1.12.1
yarl==1.6.3
zipp==3.4.0

diff so far: confluent-kafka is 1.5.0 vs 1.6.0 where it's stuck

This is a duplicate of T75, the history of which would probably be useful to take into account (I suspect it can be closed).

Migration ran to the end:

### range: [151800000, 151900000]
Timing is on.
UPDATE 0
Time: 3085.767 ms (00:03.086)
### range: [151900000, 152000000]
Timing is on.
UPDATE 0
Time: 6.366 ms

tl; dr: ETA: 12 hours [3]

So the "improved query keeps on taking more time from 15s initially to cranking up
around 2 min now (after running from yesterday up to now).

Actually, this is probably not needed. (I may close this task later)

Improved version (moving the join part within the update part of the query):

Run ok in 20 min for the staging db [1]

Script resulting from the prior analysis:

The naive need is:

explain update origin_visit_status as ovs
set type=ov.type
from origin_visit ov
where ov.visit=ovs.visit and ov.origin=ovs.origin
and ovs.type is null;