Yes, we must filter this stuff out (we discussed this issue with @zack some time ago, and you may see Torvalds' opinion too https://www.zdnet.com/article/linux-boosts-microsoft-ntfs-support-as-linus-torvalds-complains-about-github-merges/ )

The agreement has been signed, so we can move forward and publish the blog post.

Cool! LGTM

In the future we will also need to answer a more complex use-case where the iframe resembles more as an embedded copy of the archive.

Do you have an example of a complex usecase to get a better idea of your needs ?

Thanks for all this great work...
A few questions/remarks:

• where is the Permalinks tab? I do not see it in the images
• for the url, what about "/embed/" instead of "/iframe/"? "embed" seems to be the canonical term used to designate this kind of things (see YouTube, etc.)
• instead of "go to the archive", why not "View in the Software Heritage Archive"?
• it's great that the iframe can use the width and height attibutes! How do you plan to handle strange values (e.g.: width=10px height=200%)?

Thanks for looking into this.
What about sending logs to a separate dedicated logging machine instead of storing them locally?

In T3127#67581, @anlambert wrote:

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Indeed there is something weird here as we have more than one million gitlab.com origins in database.

softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%';
  count  
---------
 1023499
(1 row)

Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?
And we know we had some 1.5m origins for Google code, why only 700k shown here?

Nice to see this moving forward!

Great, it seems we are getting there :-)

In T3365#66091, @anlambert wrote:

@ardumont, @rdicosmo, I just figured out that data we are missing (Content-Length, Last-Modified) from tarballs archived by the Internet archive are in fact available in x-archive-orig-* HTTP response headers,

Thanks @ardumont for investigating this. The fact that the IA does not provide the LastModified information may make sense for their specific case (it is possible that they do not have kept the LastModified info from the original location).

In T3213#65579, @anlambert wrote:

The feature has been implemented and looks ready for production use.

I just tested it using the Web API and the docker environment for a real world example: the Kermit Software Source Code Archive.

That will be helpful in general (to answer questions likes: which endpoint is over/underused for specific use cases) and also in view of seeing who over/underuses rate limits (e.g., to identify the need of having more generous rate limits for specific use cases).

In T3202#65230, @anlambert wrote:

If we want non staff users to give a try to the tour before official release, we could take advantage of authentication here and activate the guided tour only for users with a dedicated permission.

In T3202#65224, @anlambert wrote:

Is the Help page linked from some other place? (i.e.: are we risking 404s if we dump it?)

I mean dumping the link not the page but I could move it in the footer to still reach the page.

In T3202#65222, @anlambert wrote:

After some brainstorming on the subject, I was thinking to launch the guided tour through the Help link in the left sidebar and thus dump the Help page.

In T3202#65029, @anlambert wrote:

So we have a winner here.

In T1226#64927, @anlambert wrote:

Is this feature still needed?

I think so, some origins can be long to load into the archive (huge svn repo for instance),
having a mail notification would be of interest here.

If yes, is it easy to implement it now?

Not at the moment, we need to resolve T3286 first.

A lot has changed since this was opened:

In T3312#64763, @anlambert wrote:

If we need to tune rate limit for specific type of users, this could be easily added in the new throttling
code I am currently working on.

In T3312#64760, @anlambert wrote:

@zack, @rdicosmo yes this is totally feasible by adding a dedicated Django REST Framework throttling handler for authenticated users.

Let's work on that then.

@anlambert ; ping me when this is done, so we can answer some pending requests :-)

In T3298#64431, @anlambert wrote:

So for SWHID v1, the resolver should turn the core part into lowercase , am I right ?

In T3298#64426, @zack wrote:

This is going to be an interesting challenge/trade-off for SWHIDv2. Because I was considering there to use more compact encodings than hex, in order to shorten the SWHID length, like base58, but those are case-sensitive in order to be more dense.

So, as a counter argument above the "SHOULD" idea, we need to be careful about promoting a practice now that might change when switching from SWHIDv1 to SWHIDv2.

> I also recall now that vincent added a graph [1] recently enough.

This to try and compare a bit the counter approaches together.

So that's still using the old plumbing at least for that part.

[1] https://grafana.softwareheritage.org/goto/BlkwHorMz

In T2912#64174, @ardumont wrote:

Last bits deployed on archive.s.o (including the author counters).

In T3213#64118, @ardumont wrote:

I recall it's part of creating a primary key (of sort) composed of all the properties mentioned
above (when the artifact does not provide some hashes already).
This to bypass fetching all other again things already fetched.

In T3213#64001, @ardumont wrote:

Currently users only provide an url in the save code now, the loader expects a bit more
[1] (recall it's the lister which actually provide those).

The loader expects to be provided with a list of artifacts (could be only 1 in our
case). Still, such artifacts are described through the following:

• artifact url
• time
• length (could be derived from the url when discussing with the server but not all server provides it...)
• version (could be derived with heuristic from the url as well but that's regexp-hell-ish and prone to error)
• filename (could be derived from the url without too much risk i think...)

I gather the save code now ui could be enriched (and displayed according to chosen visit
type) but that becomes more involved for people in general.

Another road would be to make some of those properties optional...

Thoughts?

[1]

 "url": "https://ftp.gnu.org/old-gnu/emacs/",
 "artifacts": [{"url": "https://ftp.gnu.org/old-gnu/emacs/elib-1.0.tar.gz",
                "time": "1995-12-12T08:00:00+00:00",
                "length": 58335,
                "version": "1.0",
                "filename": "elib-1.0.tar.gz",
                },
                ...
               ]
...

Thanks @ardumont ... so it appears that adapting the logic is easy... may you do it?
@anlambert may you look into the needed modification of the UI, to enable the new type of save code now payloads for selected authenticated users?

In T3087#63791, @douardda wrote:

So what about exports of the archive available on git-annex?

Thanks, this is quite useful indeed.

Thanks for looking into this. If I look at https://grafana.softwareheritage.org/d/WXRVVc_Mz/save-code-now?viewPanel=4&orgId=1&from=1617954242247&to=1617975842247&var-environment=production&var-instance=moma.internal.softwareheritage.org&var-status=All&var-load_task_status=All&var-visit_type=All it seems there are also some 255 requests "not yet scheduled". Maybe it's the same issue?

Thanks, it is indeed an urgent matter, as various journals depend on this!

Well, it seems we have been hit by this again, in a different form:

Cool!

Thanks to all of you for this dicussion and proposals.

Great. In addition to the content of the free form field, the standard answer should contain proper boilerplate reminding what is expected in a Save Code Now request, along the lines of what is written in the "Help" tab of https://archive.softwareheritage.org/save/

On a related note, it may be useful to regularly report requests that did not complete (either as success or failure) in a reasonable amount of time after being scheduled.

In T2912#63245, @vsellier wrote:

This kind of journal client will be necessary in any case if we want to extend the usage of the counters for other perimeters (metadata count, origin per forge, ...)

In T3084#63278, @ardumont wrote:

Pushed, packaged, deployed.

scheduler runner continues happily to schedule existing tasks and some new task with priority

Apr 15 13:12:51 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 2084 tasks load-git
Apr 15 13:12:54 saatchi swh[234257]: INFO:swh.scheduler.cli.admin.runner:Scheduled 4128 tasks
Apr 15 13:14:06 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-pypi
Apr 15 13:14:06 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git (priority)
...

That task got done almost immediately...
So there you go ;)

In T2912#63242, @vsellier wrote:

Staging webapp[1] and webapp1 on production [2] are now configured to use swh-counters to display the historical values and the live object counts.

Great news :-)