Page MenuHomeSoftware Heritage

Archive coverageFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

stuff related to extend the coverage of the Software Heritage archive

Recent Activity

Wed, Jun 17

civodul added a comment to T1352: ingest Guix (SD) packages.
In T1352#45587, @lewo wrote:

are you suggesting that sources.json itself be an "origin"?

The sources.json URL is an "origin". Each snapshot associated to this origin has several branches. Each branch corresponds to a source of the sources.json file.
There is also special branch named evaluation which points to the commit specified by the attribute revision of your sources.json file: this is to link a snapshot to a nixpkgs/guix commit.

Wed, Jun 17, 7:02 PM · Archive coverage
zack added a comment to T1352: ingest Guix (SD) packages.

@lewo it's used in our DB but also exposed in the swh-web UI in search results (and in the future it is going to be also be a field for user searches, so that you can search, e.g., "emacs" only in the list of packages archived from a given origin type).

Wed, Jun 17, 3:52 PM · Archive coverage
lewo added a comment to T1352: ingest Guix (SD) packages.

@zack

We need a name for this origin type, one of the hardest problem in CS :-)

Where is it used? Is it a new attribute?
We actually had to choose a name for the visit type, and with a lot of inspiration, we choose nixguix :-/

Wed, Jun 17, 3:15 PM · Archive coverage
lewo added a comment to T1352: ingest Guix (SD) packages.

@zimoun

Do you mean filter the unsupported urls for the field "urls" in the "type": "url"?
Or do you mean only export "type": "url" and remove all the other types from 'sources.json', for instance "git"?

Wed, Jun 17, 3:13 PM · Archive coverage
civodul added a comment to T1352: ingest Guix (SD) packages.
In T1352#45536, @zack wrote:
In T1352#45459, @lewo wrote:

So, we can now consider the sources.json file format as stable and you could make the required changes on your sources.json file. A new SHW origin should then be added.

We need a name for this origin type, one of the hardest problem in CS :-)

Can you suggest something that makes sense for both Nix, Guix, and other players in the field? As an outsider I'm a bit at loss at proposing something…

Wed, Jun 17, 2:41 PM · Archive coverage
zimoun added a comment to T1352: ingest Guix (SD) packages.

Thank you for the notification. I have tried to answer by email but I could have failed. Anyway.

Wed, Jun 17, 2:00 AM · Archive coverage

Tue, Jun 16

anadon added a comment to T1352: ingest Guix (SD) packages.

Repology.org went with "Gnu Guix".

Tue, Jun 16, 8:26 PM · Archive coverage
zack added a comment to T1352: ingest Guix (SD) packages.
In T1352#45459, @lewo wrote:

So, we can now consider the sources.json file format as stable and you could make the required changes on your sources.json file. A new SHW origin should then be added.

Tue, Jun 16, 6:34 PM · Archive coverage

Mon, Jun 15

ardumont added a comment to T1352: ingest Guix (SD) packages.

What do you think @ardumont ?

Mon, Jun 15, 7:20 PM · Archive coverage
lewo updated subscribers of T1352: ingest Guix (SD) packages.

The nixguix loader is working well since 2 weeks on the nixpkgs sources.json file!
So, we can now consider the sources.json file format as stable and you could make the required changes on your sources.json file. A new SHW origin should then be added.

Mon, Jun 15, 5:53 PM · Archive coverage

Tue, Jun 9

olasd added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

This task describes in detail what kind of scheduling policy we should implement, but it doesn't help much figure out what the next steps should be.

Tue, Jun 9, 3:04 PM · Archive coverage, Scheduling utilities

May 27 2020

ardumont added a comment to T2313: Archive git.fsfe.org (Gitea).

I've add multiple looks to the proposed gitea lister.
This looks fine to me, i've accepted it but not completely.
If some other team member could do a second pass, that'd be neat.

May 27 2020, 6:06 PM · Archive coverage, Lister

May 26 2020

ardumont added a comment to T1352: ingest Guix (SD) packages.

As a rapid follow up, here is the current structure of the sources.json the
loader nixguix is able to ingest. It's not that much different than what @lewo
initially proposed in the lister diff.

May 26 2020, 3:37 PM · Archive coverage

May 19 2020

zack renamed T682: Ingest Google Code Mercurial repositories from Inject Google Code Mercurial repositories to Ingest Google Code Mercurial repositories.
May 19 2020, 9:56 AM · Archive coverage, Mercurial loader

May 18 2020

ardumont updated subscribers of T1352: ingest Guix (SD) packages.

There has been movement in T1991 (which was not referenced as subtask so that
did now show). I fixed that.

May 18 2020, 9:15 AM · Archive coverage
ardumont added a subtask for T1352: ingest Guix (SD) packages: T1991: Implement a Guix/Nix loader.
May 18 2020, 9:05 AM · Archive coverage

May 16 2020

anadon added a comment to T1352: ingest Guix (SD) packages.

Has there been much movement with this? It looks like only packages relying on git are archived.

May 16 2020, 11:39 PM · Archive coverage

May 10 2020

zack triaged T2400: ingest Ubuntu as Normal priority.
May 10 2020, 8:43 AM · Archive coverage

Apr 23 2020

olasd triaged T2376: Consider archiving GitHub gists as Wishlist priority.
Apr 23 2020, 3:53 PM · Archive coverage

Apr 13 2020

ardumont triaged T2358: Deploy launchpad lister on staging as Normal priority.
Apr 13 2020, 11:32 AM · Lister, Archive coverage

Apr 6 2020

olasd updated the task description for T2345: Improve handling of recurrent loading tasks in scheduler.
Apr 6 2020, 7:06 PM · Archive coverage, Scheduling utilities

Apr 3 2020

olasd updated the task description for T2345: Improve handling of recurrent loading tasks in scheduler.
Apr 3 2020, 9:34 AM · Archive coverage, Scheduling utilities

Apr 2 2020

olasd updated the task description for T2345: Improve handling of recurrent loading tasks in scheduler.
Apr 2 2020, 7:24 PM · Archive coverage, Scheduling utilities
olasd updated the task description for T2345: Improve handling of recurrent loading tasks in scheduler.
Apr 2 2020, 7:23 PM · Archive coverage, Scheduling utilities
olasd updated the task description for T2345: Improve handling of recurrent loading tasks in scheduler.
Apr 2 2020, 10:02 AM · Archive coverage, Scheduling utilities
olasd added projects to T2345: Improve handling of recurrent loading tasks in scheduler: Scheduling utilities, Archive coverage.
Apr 2 2020, 9:39 AM · Archive coverage, Scheduling utilities

Mar 24 2020

legau added a comment to T1734: Create a Lister for launchpad.net.

Hi, I proposed a first version with the new changes (D2799).
@cjwatson it should be coherent with your snippet.
If somebody is used to be working with listers I would be glad to hear some remarks over how I implemented it.

Mar 24 2020, 3:50 PM · Lister, Archive coverage
zack updated the task description for T2313: Archive git.fsfe.org (Gitea).
Mar 24 2020, 1:40 PM · Archive coverage, Lister

Mar 13 2020

olasd added a comment to T2313: Archive git.fsfe.org (Gitea).

Unfortunately, try.gogs.io's API is hidden behind auth so I can't confirm that the responses actually have the same shape between gogs and gitea.

Mar 13 2020, 1:26 PM · Archive coverage, Lister
olasd added a comment to T2313: Archive git.fsfe.org (Gitea).

Thanks for submitting this request. There's a good chance that this can be the same lister as gogs: T1721.

Mar 13 2020, 1:25 PM · Archive coverage, Lister
olasd added projects to T2313: Archive git.fsfe.org (Gitea): Lister, Archive coverage.
Mar 13 2020, 1:21 PM · Archive coverage, Lister

Mar 12 2020

cjwatson added a comment to T1734: Create a Lister for launchpad.net.

Yep, I'm not annoyed, just being emphatic about what we want to see. :-)

Mar 12 2020, 6:29 PM · Lister, Archive coverage
zack added a comment to T1734: Create a Lister for launchpad.net.

Hi Colin (@cjwatson), nice to meet you here !

Mar 12 2020, 1:35 PM · Lister, Archive coverage

Mar 11 2020

cjwatson added a comment to T1734: Create a Lister for launchpad.net.

I'm one of the developers on the Launchpad team. A user identified as "leni" spoke to us about this on IRC last week; it so happened that the Launchpad team were in the middle of an in-person sprint at the time, so we were able to discuss the problem fairly quickly and put together a plan to improve our API. I implemented those improvements shortly afterwards. They aren't quite deployed on production yet, but they should be very soon. Unfortunately I don't have any contact details for leni unless they happen to join IRC, so I'm posting a summary of the discussion and my improvements here, which is probably a useful thing to do anyway.

Mar 11 2020, 11:09 PM · Lister, Archive coverage

Feb 13 2020

legau added a comment to T1734: Create a Lister for launchpad.net.

What is the current status of this task ?

Feb 13 2020, 2:34 PM · Lister, Archive coverage

Feb 6 2020

zack added a comment to T1351: (periodically) ingest GNU package releases.

Given this is done, where can one see the timeline of visits for a given origin coming from GNU?

Feb 6 2020, 8:25 PM · Archive coverage
ardumont closed T1351: (periodically) ingest GNU package releases as Resolved.
Feb 6 2020, 7:16 PM · Archive coverage
ardumont closed T1723: GNU Loader, a subtask of T1351: (periodically) ingest GNU package releases, as Resolved.
Feb 6 2020, 7:16 PM · Archive coverage
ardumont closed T1723: GNU Loader as Resolved.
Feb 6 2020, 7:16 PM · Archive coverage

Jan 27 2020

vlorentz added a project to T1734: Create a Lister for launchpad.net: Lister.
Jan 27 2020, 4:31 PM · Lister, Archive coverage

Jan 21 2020

ardumont closed T2029: cran lister: Align lister to output list of tarballs per origin as Resolved.

Deployed.

Jan 21 2020, 11:45 AM · Origin-CRAN, Archive coverage
ardumont closed T2029: cran lister: Align lister to output list of tarballs per origin, a subtask of T2026: Implement cran loader with package manager mechanism, as Resolved.
Jan 21 2020, 11:45 AM · Origin-CRAN, Archive coverage

Jan 17 2020

ardumont added a comment to T2029: cran lister: Align lister to output list of tarballs per origin.

The original description of this task was adapted by D2531 D2532 D2524.
A sample of what's been listed and ingested can be seen through the staging webapp instance [1].

Jan 17 2020, 12:35 PM · Origin-CRAN, Archive coverage
ardumont renamed T2029: cran lister: Align lister to output list of tarballs per origin from cran lister: Align lister to output list of tarballs per origin (if possible) to cran lister: Align lister to output list of tarballs per origin.
Jan 17 2020, 12:25 PM · Origin-CRAN, Archive coverage

Jan 16 2020

ardumont added a comment to T2029: cran lister: Align lister to output list of tarballs per origin.

In the [2] page, there is an archive link (or something) which lists the old associated artifacts (so apriori, no more need for the mran mirror).

Jan 16 2020, 1:47 PM · Origin-CRAN, Archive coverage
olasd added a comment to T2029: cran lister: Align lister to output list of tarballs per origin.
Jan 16 2020, 1:44 PM · Origin-CRAN, Archive coverage
ardumont added a comment to T2029: cran lister: Align lister to output list of tarballs per origin.

Heads up.

Jan 16 2020, 11:27 AM · Origin-CRAN, Archive coverage

Jan 9 2020

ardumont closed T2026: Implement cran loader with package manager mechanism as Resolved.

Deployed.

Jan 9 2020, 1:48 PM · Origin-CRAN, Archive coverage

Nov 26 2019

ardumont closed T2098: Deploy package loaders as Resolved.
Nov 26 2019, 5:28 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage
ardumont updated the task description for T2098: Deploy package loaders.
Nov 26 2019, 5:28 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage