Page MenuHomeSoftware Heritage

ListerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Recent Activity

Jun 16 2020

olasd added a parent task for T2453: Use the scheduler "lister api" in listers: T2454: Stop creating tasks directly in listers.
Jun 16 2020, 11:25 AM · Lister
olasd added subtasks for T2454: Stop creating tasks directly in listers: T2453: Use the scheduler "lister api" in listers, T2444: Implement the scheduling policy for the recurrent visit scheduler.
Jun 16 2020, 11:25 AM · Lister
olasd triaged T2454: Stop creating tasks directly in listers as Normal priority.
Jun 16 2020, 11:24 AM · Lister
olasd added a subtask for T2453: Use the scheduler "lister api" in listers: T2442: Provide a unified API for listers to interact with the scheduler.
Jun 16 2020, 11:24 AM · Lister
olasd triaged T2453: Use the scheduler "lister api" in listers as High priority.
Jun 16 2020, 11:23 AM · Lister

May 27 2020

ardumont added a comment to T2313: Archive git.fsfe.org (Gitea).

I've add multiple looks to the proposed gitea lister.
This looks fine to me, i've accepted it but not completely.
If some other team member could do a second pass, that'd be neat.

May 27 2020, 6:06 PM · Archive coverage, Lister

May 18 2020

ardumont added a task to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file: T1991: Implement a Guix/Nix loader.
May 18 2020, 9:07 AM · Lister

May 14 2020

lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

What's the status of this patch series? Would be great to deploy it. :-)

May 14 2020, 10:43 AM · Lister

May 12 2020

civodul added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
In D2025#67709, @lewo wrote:

@civodul @zimoun I'm wondering if you generate a sources.json file for any commit of your guix repository or only for those that have been evaluated and pushed to your binary cache by your CI?

May 12 2020, 12:31 PM · Lister

Apr 20 2020

ardumont added a subtask for T1991: Implement a Guix/Nix loader: T2357: nixguix: fails to finish as downloading artifacts step hangs.
Apr 20 2020, 9:44 AM · Package Loader
ardumont added a subtask for T1991: Implement a Guix/Nix loader: T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error.
Apr 20 2020, 9:43 AM · Package Loader

Apr 13 2020

ardumont triaged T2358: Deploy launchpad lister on staging as Normal priority.
Apr 13 2020, 11:32 AM · Lister, Archive coverage

Apr 2 2020

ardumont updated the task description for T1991: Implement a Guix/Nix loader.
Apr 2 2020, 4:34 PM · Package Loader

Mar 24 2020

legau added a comment to T1734: Create a Lister for launchpad.net.

Hi, I proposed a first version with the new changes (D2799).
@cjwatson it should be coherent with your snippet.
If somebody is used to be working with listers I would be glad to hear some remarks over how I implemented it.

Mar 24 2020, 3:50 PM · Lister, Archive coverage
zack updated the task description for T2313: Archive git.fsfe.org (Gitea).
Mar 24 2020, 1:40 PM · Archive coverage, Lister

Mar 19 2020

lewo updated the task description for T1991: Implement a Guix/Nix loader.
Mar 19 2020, 11:09 AM · Package Loader

Mar 18 2020

ardumont added a comment to T1991: Implement a Guix/Nix loader.

First runs triggered on staging, errors will be reported in sentry [1]

Mar 18 2020, 4:50 PM · Package Loader

Mar 13 2020

olasd edited projects for T1721: Implementation of Gogs Lister, added: Lister; removed Archive coverage.
Mar 13 2020, 1:27 PM · Lister
olasd added a comment to T2313: Archive git.fsfe.org (Gitea).

Unfortunately, try.gogs.io's API is hidden behind auth so I can't confirm that the responses actually have the same shape between gogs and gitea.

Mar 13 2020, 1:26 PM · Archive coverage, Lister
olasd added a comment to T2313: Archive git.fsfe.org (Gitea).

Thanks for submitting this request. There's a good chance that this can be the same lister as gogs: T1721.

Mar 13 2020, 1:25 PM · Archive coverage, Lister
olasd added projects to T2313: Archive git.fsfe.org (Gitea): Lister, Archive coverage.
Mar 13 2020, 1:21 PM · Archive coverage, Lister
lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

@civodul @zimoun I'm wondering if you generate a sources.json file for any commit of your guix repository or only for those that have been evaluated and pushed to your binary cache by your CI?

Mar 13 2020, 10:36 AM · Lister

Mar 12 2020

cjwatson added a comment to T1734: Create a Lister for launchpad.net.

Yep, I'm not annoyed, just being emphatic about what we want to see. :-)

Mar 12 2020, 6:29 PM · Lister, Archive coverage
zack added a comment to T1734: Create a Lister for launchpad.net.

Hi Colin (@cjwatson), nice to meet you here !

Mar 12 2020, 1:35 PM · Lister, Archive coverage

Mar 11 2020

cjwatson added a comment to T1734: Create a Lister for launchpad.net.

I'm one of the developers on the Launchpad team. A user identified as "leni" spoke to us about this on IRC last week; it so happened that the Launchpad team were in the middle of an in-person sprint at the time, so we were able to discuss the problem fairly quickly and put together a plan to improve our API. I implemented those improvements shortly afterwards. They aren't quite deployed on production yet, but they should be very soon. Unfortunately I don't have any contact details for leni unless they happen to join IRC, so I'm posting a summary of the discussion and my improvements here, which is probably a useful thing to do anyway.

Mar 11 2020, 11:09 PM · Lister, Archive coverage

Mar 10 2020

lewo updated the task description for T1991: Implement a Guix/Nix loader.
Mar 10 2020, 6:49 PM · Package Loader
lewo updated the task description for T1991: Implement a Guix/Nix loader.
Mar 10 2020, 6:42 PM · Package Loader

Mar 9 2020

lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

Thanks to @zimoun, https://guix.gnu.org/sources.json is now generated periodically (every hour). Each url is now a list.

Mar 9 2020, 2:35 PM · Lister
civodul added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
In D2025#65063, @lewo wrote:

While looking into this with @zimoun, we realized it would be nicer if url were an array of URLs (as is the case at https://guix.gnu.org/packages.json) rather than a single URL.

without changing now the crawler, i.e., the crawler can ingest only the first elem of the array and it will be modified later.

Yeah, I was thinking to introduce this later. But as you said, we could still modify the format without supporting it in the lister.
So, that's fine for me if we generate a list of urls instead of a single url. I could easily update the file NixOS is generating.

Mar 9 2020, 1:52 PM · Lister

Mar 6 2020

lewo abandoned D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

@lewo : Let me know the new diff number. :-)

Mar 6 2020, 5:40 PM · Lister
zimoun added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

After some discussions with the SWH team, it is actually no longer the good way to fill the archive with our sources. Instead, I'm starting to write a loader which will be in charge of reading our sources.json and fill the archive. So, I'm closing this diff and will create a new diff with a loader in the next few days;)

Mar 6 2020, 10:14 AM · Lister

Mar 5 2020

ardumont assigned T1991: Implement a Guix/Nix loader to lewo.
Mar 5 2020, 7:30 PM · Package Loader
ardumont renamed T1991: Implement a Guix/Nix loader from Implement a Guix/GuixSD/Nix/NixOS loader to Implement a Guix/Nix loader.
Mar 5 2020, 7:30 PM · Package Loader
ardumont added a comment to T1991: Implement a Guix/Nix loader.

Implementation of a lister started at D2025.

Mar 5 2020, 7:30 PM · Package Loader
ardumont renamed T1991: Implement a Guix/Nix loader from Implement a Guix/GuixSD/Nix/NixOS lister to Implement a Guix/GuixSD/Nix/NixOS loader.
Mar 5 2020, 7:28 PM · Package Loader
lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

After some discussions with the SWH team, it is actually no longer the good way to fill the archive with our sources. Instead, I'm starting to write a loader which will be in charge of reading our sources.json and fill the archive. So, I'm closing this diff and will create a new diff with a loader in the next few days;)
There are also some advantages of implementing a loader: for instance, we could query the SWH API to know which sources of a specific sources.json file have been archived!

Mar 5 2020, 4:36 PM · Lister

Mar 2 2020

zimoun added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

@lewo: Does the version of the format should be bumped to 2 with this string-to-array modification?

No, I don't think so since it is not used yet.

Mar 2 2020, 7:57 PM · Lister
lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

> @lewo: Does the version of the format should be bumped to 2 with this string-to-array modification?

Mar 2 2020, 6:52 PM · Lister
zimoun added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

@lewo: Does the version of the format should be bumped to 2 with this string-to-array modification?

Mar 2 2020, 5:26 PM · Lister

Feb 27 2020

lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

While looking into this with @zimoun, we realized it would be nicer if url were an array of URLs (as is the case at https://guix.gnu.org/packages.json) rather than a single URL.

without changing now the crawler, i.e., the crawler can ingest only the first elem of the array and it will be modified later.

Feb 27 2020, 10:24 PM · Lister

Feb 18 2020

zimoun added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

While looking into this with @zimoun, we realized it would be nicer if url were an array of URLs (as is the case at https://guix.gnu.org/packages.json) rather than a single URL.

The reason is that in many cases, both Guix and Nixpkgs provide a list of URLs rather than a single URL, which is useful when one of them breaks.

Feb 18 2020, 10:01 PM · Lister
civodul added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
In D2025#61931, @lewo wrote:

A CI job is building a sources.json every day! The file is available at https://nix-community.github.io/nixpkgs-swh/sources.json ;)

Awesome!

Feb 18 2020, 9:23 AM · Lister

Feb 14 2020

legau moved T2260: Fully annotate swh-lister with types from Backlog to In progress on the Easy hack board.
Feb 14 2020, 2:39 PM · Easy hack, Lister

Feb 13 2020

legau added a comment to T1734: Create a Lister for launchpad.net.

What is the current status of this task ?

Feb 13 2020, 2:34 PM · Lister, Archive coverage

Jan 30 2020

lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
> If you are going to the FOSDEM, would be nice to meet you there to talk about next steps!

It would but i'm not going.
Jan 30 2020, 8:56 PM · Lister
lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
> If you are going to the FOSDEM, would be nice to meet you there to talk about next steps!

I'm already in Brussels and would be happy to meet!
Jan 30 2020, 8:53 PM · Lister
ardumont updated subscribers of D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

A CI job is building a sources.json every day! The file is available at https://nix-community.github.io/nixpkgs-swh/sources.json ;)

This is a community CI (not hosted on main NixOS infrascture) which will allow me to iterate quickly on this file.

Jan 30 2020, 10:30 AM · Lister
civodul added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
In D2025#61931, @lewo wrote:

A CI job is building a sources.json every day! The file is available at https://nix-community.github.io/nixpkgs-swh/sources.json ;)

Jan 30 2020, 12:44 AM · Lister

Jan 29 2020

lewo added a comment to D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.

A CI job is building a sources.json every day! The file is available at https://nix-community.github.io/nixpkgs-swh/sources.json ;)
This is a community CI (not hosted on main NixOS infrascture) which will allow me to iterate quickly on this file.

Jan 29 2020, 11:39 PM · Lister
lewo updated the summary of D2025: [wip] swh.lister.functionalPackages: add lister getting sources from a JSON file.
Jan 29 2020, 11:33 PM · Lister