In T3202#65029, @anlambert wrote:

So we have a winner here.

So I have played with intro.js and it is clearly the best guided tour library that I have tested so far:

• AGPL license
• nice display and smooth animations
• JSON configurable
• no Bootstrap dependency (CSS framework agnostic)
• well maintained

Turned out Bootstrap Tour is not compatible with Bootstrap 4.x but hopefully it exists a fork Bootstrap Tourist that supports it.

So I did some first tests using chardin.js but the result is not really conclusive.

In T1226#64927, @anlambert wrote:

Is this feature still needed?

I think so, some origins can be long to load into the archive (huge svn repo for instance),
having a mail notification would be of interest here.

If yes, is it easy to implement it now?

Not at the moment, we need to resolve T3286 first.

Is this feature still needed?

A lot has changed since this was opened:

This is now implemented and deployed to production: a new checkbox to filter user own requests has been added to the Save code now Web UI.

The new rate limit policy has been implemented and deployed to production, see X-RateLimit-Limit header value below.

In T3312#64763, @anlambert wrote:

If we need to tune rate limit for specific type of users, this could be easily added in the new throttling
code I am currently working on.

Thanks! Please make sure we can also have specific rate limits for specific classees of authenticated users, on top of the default new rate limit for all registered users.

In T3312#64760, @anlambert wrote:

@zack, @rdicosmo yes this is totally feasible by adding a dedicated Django REST Framework throttling handler for authenticated users.

Let's work on that then.

@zack, @rdicosmo yes this is totally feasible by adding a dedicated Django REST Framework throttling handler for authenticated users.

@anlambert ; ping me when this is done, so we can answer some pending requests :-)

Fix has been deployed to production, closing this.

Closing this as resolved now the search feature is using elasticsearch in production.

In T3298#64431, @anlambert wrote:

So for SWHID v1, the resolver should turn the core part into lowercase , am I right ?

I'm not a fan of changing the spec of SWHID version 1 to make them case insensitive, as it seems to be a significant change (in particular for the code that checks for the syntactic correctness of IDs).

In T3298#64426, @zack wrote:

This is going to be an interesting challenge/trade-off for SWHIDv2. Because I was considering there to use more compact encodings than hex, in order to shorten the SWHID length, like base58, but those are case-sensitive in order to be more dense.

So, as a counter argument above the "SHOULD" idea, we need to be careful about promoting a practice now that might change when switching from SWHIDv1 to SWHIDv2.

Ah, this is an interesting practical problem.
I'm not a fan of changing the spec of SWHID version 1 to make them case insensitive, as it seems to be a significant change (in particular for the code that checks for the syntactic correctness of IDs).
But we can totally add a "SHOULD" section to the resolvers part of the spec recommending (but not mandating) that resolvers treat core SWHIDs as case insensitive. (Of course all the contextual parts cannot be considered case insensitive.)

> I also recall now that vincent added a graph [1] recently enough.

This to try and compare a bit the counter approaches together.

So that's still using the old plumbing at least for that part.

[1] https://grafana.softwareheritage.org/goto/BlkwHorMz

In T2912#64208, @ardumont wrote:

What about the old counter pipeline? Has it been decommissioned already?

I don't think so as I do not recall seeing diffs about clean up.

In any case, it's not part of what's currently deployed (so no risk for
data mangling if that's part the concern).

After discussion with @anlambert and @rdicosmo, we agreed on the following as a first
iteration of the ui for the new bundle type.

Hii @moranegg @zack I'm working on this task and I noticed that we are using pybadges for implementing swh-badges which isn't the case with shields.io. Based on what I understand, these badges have to be re-implemented as shields.io services will use swh apis to generate the dynamic parts of the badges.

What about the old counter pipeline? Has it been decommissioned already?

In T2912#64174, @ardumont wrote:

Last bits deployed on archive.s.o (including the author counters).

Last bits deployed on archive.s.o (including the author counters).

Remains one or two concerns about this prior to actually act on it.

In T3087#63887, @rdicosmo wrote:

In T3087#63791, @douardda wrote:

So what about exports of the archive available on git-annex?

Hear hear, it's kept up now:

ardumont@counters1:~% date;redis-cli pfcount person
Sat 24 Apr 2021 05:31:18 PM UTC
(integer) 42190221

If I understand well, url+time+length+filename+version are used in an heuristic to
avoid (down)loading over and over again something that is already ingested

In T3213#64118, @ardumont wrote:

I recall it's part of creating a primary key (of sort) composed of all the properties mentioned
above (when the artifact does not provide some hashes already).
This to bypass fetching all other again things already fetched.

(submitted too early)

In T3213#64001, @ardumont wrote:

Currently users only provide an url in the save code now, the loader expects a bit more
[1] (recall it's the lister which actually provide those).

The loader expects to be provided with a list of artifacts (could be only 1 in our
case). Still, such artifacts are described through the following:

• artifact url
• time
• length (could be derived from the url when discussing with the server but not all server provides it...)
• version (could be derived with heuristic from the url as well but that's regexp-hell-ish and prone to error)
• filename (could be derived from the url without too much risk i think...)

I gather the save code now ui could be enriched (and displayed according to chosen visit
type) but that becomes more involved for people in general.

Another road would be to make some of those properties optional...

Thoughts?

[1]

 "url": "https://ftp.gnu.org/old-gnu/emacs/",
 "artifacts": [{"url": "https://ftp.gnu.org/old-gnu/emacs/elib-1.0.tar.gz",
                "time": "1995-12-12T08:00:00+00:00",
                "length": 58335,
                "version": "1.0",
                "filename": "elib-1.0.tar.gz",
                },
                ...
               ]
...

and the authors are now displayed on staging and production (webapp1)

The lag for the production can be followed here: https://grafana.softwareheritage.org/goto/Di2H3z9Gk
(staging has already recovered)

the swh-counters is deployed in production too:

• upgrade swh-counters package and restart swh-counters backend and journal

root@counters1:~# apt dist-upgrade
...
Setting up python3-swh.counters (0.7.0-1~swh1~bpo10+1) ...
root@counters1:~# systemctl stop swh-counters-journal-client.service 
root@counters1:~# systemctl restart gunicorn-swh-counters.service 
root@counters1:~# systemctl start swh-counters-journal-client.service 
root@counters1:~# redis-cli pfcount person
(integer) 7

The count of the person already starts

• stopping the journal-client to be able to reset the releases and revisions offsets

root@counters1:~# systemctl stop swh-counters-journal-client.service

• reset the offsets

vsellier@kafka1 ~ % /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets.csv
# revision reset
vsellier@kafka1 ~ % 
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.revision
# release reset
vsellier@kafka1 ~ %  /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.release 
# checks
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets-backfill.csv 
vsellier@kafka1 ~ % diff ~/counters_journal_client_offsets.csv ~/counters_journal_client_offsets-backfill.csv | less 
1c1
< "swh.journal.objects.revision",25,8275180
---
> "swh.journal.objects.revision",25,0
8c8
< "swh.journal.objects.release",128,78484
---
> "swh.journal.objects.release",128,0
16c16
...

• journal client restarted

root@counters1:~# systemctl start swh-counters-journal-client.service

• the person counters is growing fastly

root@counters1:~# date;redis-cli pfcount person
Fri 23 Apr 2021 10:55:54 AM UTC
(integer) 72358
root@counters1:~# date;redis-cli pfcount person
Fri 23 Apr 2021 10:55:57 AM UTC
(integer) 80618

Also [1] to follow through the journal client consumption (it has data now ;)

I think you can close D5573 which is obsolete now with the latest change.

• version 0.7.0 release with the last improvement (D5576) of vlorentz (thanks)
• deployment done in staging
• the person counting has started on the live messages:

root@counters0:~# redis-cli
127.0.0.1:6379> pfcount person
(integer) 7

• now let reset the consumer offsets for the release and revision topics to backfill the person counter:

# offsets backup
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets.csv
# revision reset
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.revision
# release reset
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.release

I stand by what i said regarding the scheduling logic, it's as simple as I described
earlier... But...