Page MenuHomeSoftware Heritage

Maven listerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

Lister for maven repositories

Recent Activity

Thu, Jun 16

ardumont updated the task description for T4330: Deploy maven stack in production.
Thu, Jun 16, 11:13 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont triaged T4330: Deploy maven stack in production as Normal priority.
Thu, Jun 16, 10:21 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Wed, Jun 15

bchauvet closed T4326: Archive the pom file additionally to the source folder, a subtask of T3746: staging: Deploy maven indexer/lister/loader, as Invalid.
Wed, Jun 15, 5:20 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet closed T4326: Archive the pom file additionally to the source folder as Invalid.
Wed, Jun 15, 5:20 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

Yesss! \o/

Wed, Jun 15, 4:33 PM · Maven loader, Maven lister, System administration, Archive coverage
anlambert added a comment to T4326: Archive the pom file additionally to the source folder.

So in the end, the conclusion is that the loader already does the right thing so it's a noop, right?

Wed, Jun 15, 4:26 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont renamed T4326: Archive the pom file additionally to the source folder from archive the pom file additionally to the source folder to Archive the pom file additionally to the source folder.
Wed, Jun 15, 4:09 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont renamed T4326: Archive the pom file additionally to the source folder from archive the pom file additionnaly to the source folder to archive the pom file additionally to the source folder.
Wed, Jun 15, 4:09 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

Good news *can* happen, ahah! Thanks for notifying me.

Wed, Jun 15, 3:42 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated subscribers of T4326: Archive the pom file additionally to the source folder.

To summarize, the initial intent was to adapt the jar loaded (as extracted directory) to append the pom.xml so we do not lose that reference.

Wed, Jun 15, 3:33 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet added a comment to T4326: Archive the pom file additionally to the source folder.

You're right boris, indeed it's already stored as extrinsic metadata, we hadn't checked properly :)
Thank you for your answer !

Wed, Jun 15, 3:33 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

I'm not sure to understand the intent, as we already keep the pom in the extrinsic metadata (don't we?).
Double-checking in the SWH codebase, I believe you could build upon this: see [1] lines 166-180.

Wed, Jun 15, 3:07 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

Congrats on the work done! I think that downloading the pom file from the same folder is indeed the way to go.

Wed, Jun 15, 2:54 PM · Maven loader, Maven lister, System administration, Archive coverage
anlambert added a comment to T4326: Archive the pom file additionally to the source folder.

I think the simplest way to get the pom file associated to a specific release of a maven package is to download it from the folder where we can find the source jar.

Wed, Jun 15, 1:48 PM · Maven loader, Maven lister, System administration, Archive coverage

Mon, Jun 13

bchauvet updated the task description for T4326: Archive the pom file additionally to the source folder.
Mon, Jun 13, 2:21 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet added a comment to T4326: Archive the pom file additionally to the source folder.
Mon, Jun 13, 2:16 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet updated the task description for T4326: Archive the pom file additionally to the source folder.
Mon, Jun 13, 1:36 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet updated the task description for T4326: Archive the pom file additionally to the source folder.
Mon, Jun 13, 1:30 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet added a comment to T4326: Archive the pom file additionally to the source folder.

in a source.jar, the pom is not inculded by default but can be if specified :

Mon, Jun 13, 1:27 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet added a comment to T4326: Archive the pom file additionally to the source folder.
Mon, Jun 13, 1:25 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet updated the task description for T4326: Archive the pom file additionally to the source folder.
Mon, Jun 13, 12:11 PM · Maven loader, Maven lister, System administration, Archive coverage
bchauvet triaged T4326: Archive the pom file additionally to the source folder as Normal priority.
Mon, Jun 13, 12:10 PM · Maven loader, Maven lister, System administration, Archive coverage

Fri, Jun 3

ardumont added a comment to T3874: staging: Analyze result of the maven listing and ingestion.

There remains git and other dvcs typed origins listed by maven but not github ones.

Fri, Jun 3, 4:11 PM · Maven loader, Maven lister, Archive coverage
ardumont closed T4232: Listers: Canonicalize listed github origins, a subtask of T3874: staging: Analyze result of the maven listing and ingestion, as Resolved.
Fri, Jun 3, 3:19 PM · Maven loader, Maven lister, Archive coverage
ardumont closed T3874: staging: Analyze result of the maven listing and ingestion as Resolved.
Fri, Jun 3, 3:18 PM · Maven loader, Maven lister, Archive coverage
ardumont closed T3874: staging: Analyze result of the maven listing and ingestion, a subtask of T3746: staging: Deploy maven indexer/lister/loader, as Resolved.
Fri, Jun 3, 3:18 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3874: staging: Analyze result of the maven listing and ingestion.
Fri, Jun 3, 3:18 PM · Maven loader, Maven lister, Archive coverage
ardumont added a subtask for T3874: staging: Analyze result of the maven listing and ingestion: T4232: Listers: Canonicalize listed github origins.
Fri, Jun 3, 3:18 PM · Maven loader, Maven lister, Archive coverage
ardumont added a comment to T3874: staging: Analyze result of the maven listing and ingestion.

status: triggered 2 full-maven lister runs on maven central and jboss [1]
And no more exotic github urls are popping up [2].

Fri, Jun 3, 2:18 PM · Maven loader, Maven lister, Archive coverage
ardumont added a comment to T3874: staging: Analyze result of the maven listing and ingestion.

Yesterday, i had fixed, diffed, released and pushed the diff [1] fixing the
canonicalization of remaining exotic urls, cleaned up 'git' (out of a maven listing)
origins and triggered back a listing. Today, checking back those origins (staging
scheduler), there was still noise which should no longer have been there...

Fri, Jun 3, 9:35 AM · Maven loader, Maven lister, Archive coverage

Thu, Jun 2

ardumont added a revision to T3874: staging: Analyze result of the maven listing and ingestion: D7946: github/utils: Deal with exotic urls to canonicalize.
Thu, Jun 2, 3:14 PM · Maven loader, Maven lister, Archive coverage
ardumont added a comment to T3874: staging: Analyze result of the maven listing and ingestion.

Full listing is not finished yet but still there remains origins with exotic starting urls which are not canonicalized.
I'd say the issue lies with the canonicalize swh.core implementation code which only deals with https:// and git:// urls.
So some improvments are needed there.

Thu, Jun 2, 2:08 PM · Maven loader, Maven lister, Archive coverage

Wed, Jun 1

ardumont updated the task description for T3874: staging: Analyze result of the maven listing and ingestion.
Wed, Jun 1, 3:06 PM · Maven loader, Maven lister, Archive coverage
ardumont added a comment to T3874: staging: Analyze result of the maven listing and ingestion.

Plan:

  • P1369: Listing status after first round listing
  • Clean up maven github origins listing [1]
  • Trigger maven full run [2]
  • Wait for listing to finish
  • Listing status after new maven lister round of listing
  • Ping in mailing list discussion with data!
Wed, Jun 1, 3:05 PM · Maven loader, Maven lister, Archive coverage
ardumont updated the task description for T3874: staging: Analyze result of the maven listing and ingestion.
Wed, Jun 1, 3:01 PM · Maven loader, Maven lister, Archive coverage
ardumont updated the task description for T3874: staging: Analyze result of the maven listing and ingestion.
Wed, Jun 1, 10:50 AM · Maven loader, Maven lister, Archive coverage
ardumont added a comment to T3874: staging: Analyze result of the maven listing and ingestion.

Old maven behavior results in origins like git://github.com, ... [1]
The new maven lister behavior should now result in canonical github urls http://github.com/user/repo.
Analysis ongoing and report will go after that comment.

Wed, Jun 1, 10:50 AM · Maven loader, Maven lister, Archive coverage
ardumont updated the task description for T3874: staging: Analyze result of the maven listing and ingestion.
Wed, Jun 1, 10:47 AM · Maven loader, Maven lister, Archive coverage
ardumont created P1369 old maven listing with github origins non-canonicalized.
Wed, Jun 1, 10:46 AM · Maven lister

May 13 2022

anlambert added a project to T4105: Push maven-index-exporter image to docker hub: Maven lister.
May 13 2022, 4:55 PM · Maven lister, System administration, Continuous Integration
anlambert added projects to T4143: staging: Deploy maven stack fixes: Maven lister, Maven loader.
May 13 2022, 4:55 PM · Maven loader, Maven lister, System administration, Archive coverage
anlambert added projects to T3746: staging: Deploy maven indexer/lister/loader: Maven lister, Maven loader.
May 13 2022, 4:54 PM · Maven loader, Maven lister, System administration, Archive coverage
anlambert added projects to T1724: Maven Central repository support: Maven lister, Maven loader.
May 13 2022, 4:54 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
anlambert added projects to T4215: staging: Deploy latest maven stack: Maven lister, Maven loader.
May 13 2022, 4:54 PM · Maven loader, Maven lister, System administration, Archive coverage
anlambert added projects to T3874: staging: Analyze result of the maven listing and ingestion: Maven lister, Maven loader.
May 13 2022, 4:53 PM · Maven loader, Maven lister, Archive coverage
anlambert created Maven lister.
May 13 2022, 4:52 PM