Support old unittest.mock

Fallback to fsync if fdatasync isn't available

In D1586#36460, @ardumont wrote:

sound good to me
@olasd any thoughts on this?

Rebase; update client_id to match class hierarchy

Rebase

(I'll pass on the underlying limitation of being forced to link to a release object from wikidata, which feels a bit artificial but is out of scope for this task)

Thanks!

I don't think that extras_require is going to work.

After processing the logs of the backfilling process to make sure to redo all the ranges that were interrupted in various database migrations, I'm now confident that this task is complete: we have a full mirror of all contents on Azure, which is kept up to date by the main archive storage backend writing synchronously to it.

In T1815#33533, @anlambert wrote:

I should have read django-simple-captcha doc, indeed its integration is not really straightforward for swh-web.
Currently, only the api endpoint for creating save requests is rate limited while the save code now form is submitted using Javascript
(validating input then setting the appropriate Django CSRF token before sending the POST request).
So without captcha, it will still be difficult for a dumb bot to spam us.
I would go for removing the captcha but add rate limiting to the form submission just in case.

In T1389#33215, @zack wrote:

Thanks @olasd, @ardumont, and @anlambert for this, it's a great plan and I like it a lot !
Just a few comments on the sidelines:

The lister will generate a one-shot task to load each package for the given repository, with the full information needed to do the data fetching.

This seemed clear from a different part of the description, but fwiw here I'm assuming the plan is to only load the version of the packages not already known/ingested in the past.

FWIW https://grafana.softwareheritage.org/d/jScG7g6mk/objstorage-object-counts?orgId=1 is a (not great) implementation of this.

As an alternative, we could just set the Django CSRF token on the form using a bit of Javascript code rather than the view sending it directly in the form, which would thwart most dumb bots (that's the "ReCAPTCHA alternatives for uncustomized spam > Javascript" section of the aforementioned article).

With the post-hoc moderation of Save Code Now requests, do we really need a captcha? Isn't the base rate limiting enough?

• The main archive currently synchronously writes all contents to Azure as well as the local storage (the gap is strictly closing)
• all partitions from uffizi have been copied to azure and mass-injected (except for partition 8 which only got partially mass injected)
• after this process, it looks like azure is missing 10% of all objects (excluding partition 8), which are all on banco
  • I've started a procedure to copy the missing objects from banco directly. Estimated time to completion ~ 1 month
  • The same procedure has been started to copy the missing objects from partition 8 on uffizi. Estimated time to completion ~ 15 days

In T1776#32932, @nahimilega wrote:

OK. Out of curiosity, would you be able to look at the location of dists instead? Just to get a sense of how much overlap there is with archives from GitHub.

I made a short script to analyse the location of dists for packages; here is the result.
Packages iterated - 15336 (~ 6% of total packages)
Number packages whose dists were not hosted on GitHub, or bitbucket or gitlab: 2
VCS for dists: zip, git

In T1776#32914, @nahimilega wrote:

In T1776#32892, @olasd wrote:

In T1776#32739, @nahimilega wrote:

There is a total of 220570 packages.
I made a short script to analyse the VCS and code hosting platform for packages, here is the result.
Packages iterated - 15126 (~ 6% of total packages)
VCS found - git, hg
Number of packages that were not hosted on GitHub, or bitbucket or gitlab: 51
There are expected to be around 744 packages that are not hosted on GitHub, or bitbucket or Gitlab if we calculate using applying unitary method on the above data.

In your analysis, did you look at the source key (which seems to represent the upstream version control repository) or the dist key (which points at the tarball/zipfile that's actually downloaded by the package manager when installing that version of the package)?

I looked at the source key in my analysis.

Once we have the (released) dists figured out, we can also consider -as a second step- having the packagist lister submit tasks for the upstream repositories mentioned in the source key as well, as an extra data source.

In T1776#32739, @nahimilega wrote:

There is a total of 220570 packages.
I made a short script to analyse the VCS and code hosting platform for packages, here is the result.
Packages iterated - 15126 (~ 6% of total packages)
VCS found - git, hg
Number of packages that were not hosted on GitHub, or bitbucket or gitlab: 51
There are expected to be around 744 packages that are not hosted on GitHub, or bitbucket or Gitlab if we calculate using applying unitary method on the above data.

Some part of me wonders if it wouldn't make sense to unify the other way around, as the argument names you picked in the in-memory storage are way more sensible, but we've not succeeded in making a change in this area without breaking some (or all) clients...

In T1734#32762, @anonbnr wrote:

Hello, we are a group of M1 computer science students of the University of Montpellier, France.

kthxbye

Thanks a lot to @hboutemy for your valuable insights on sources in the Maven central repository, and for the pointer to Reproducible Builds on the JVM.

As mentioned in D1516 I don't think DNS provides the proper granularity for this; somerset is a "database replica" server only coincidentally; all its databases that aren't the replica of the main database are actually primaries.

I'm not convinced this is such a good idea; this machine is way more than a "db replica" server (it only has one replica, most its databases are actually primary) and I don't think DNS provides the appropriate granularity level to record this information.

Thanks for this change!

Thanks for this first pass at a Maven lister.

We've discussed a plausible plan for a "base package manager loader" with @ardumont and, to some extent, @anlambert.

I've got two questions that @ardumont or @moranegg probably have an opinion about :)

Obviously needs a core release, but looks good to me.

Please add click in requirements-test.txt :-)

Considering the size of that database, and the fact that we don't have any provisions to automatically spin up a new database server, I think it would make more sense to repatriate it on our main postgres setup, rather than movig it to a new machine on azure.

Expanding on what Dirk Eddelbuettel posted on IRC when we talked about that, a minimal R script to fetch the current package information would be:

In T1709#31492, @nahimilega wrote:

Here is an implementation plan for making R-CRAN lister.
I have taken inspiration from the pypi lister.
To make lister.py for R-CRAN, we need to inherit SimpleLister class and override ingest_data() function and change its first line (where safely_issue_request() is called) to call the function which would run R script to return a json response.
Then after that it is quite like any normal response, we just need to implement following function list_packages, compute url, get_model_from_repo, task_dict and transport_response_simplified.

So that took a few tries in puppet, but adding new brokers to the kafka deployment should now be seamless.

Looks good, thanks!

In T1689#31539, @zack wrote:

I'm not a fan of automated merging when the test suite passes. I think merging code from third parties should always be a manual step by someone. If it can be made a Web UI button, even better, if not we will just standardize to the local arc workflow, which I've recently documented in the wiki.

(looks like Herald can't easily trigger on a comment, because that'd be too easy)

So, according to the fine manual (https://secure.phabricator.com/book/phabricator/article/differential_land/) the "Land Revision" button on the Phabricator user interface depends on setting up:

• Drydock repository automation : https://secure.phabricator.com/book/phabricator/article/drydock_repository_automation/
• Which depends on setting up Drydock working copies : https://secure.phabricator.com/book/phabricator/article/drydock_working_copies/
• Which depends on setting up Drydock hosts : https://secure.phabricator.com/book/phabricator/article/drydock_hosts/
• Which depends on Almanac : https://secure.phabricator.com/book/phabricator/article/almanac/

We're already using LiveDNS for the softwareheritage.org zone, so DNS switchover time should not be an issue (all the more so considering the softwareheritage.org -> www.softwareheritage.org redirect is half-broken already).

The grafana dashboards are stored in the postgresql database on pergamon, which is backed up through the full system backups.