Page MenuHomeSoftware Heritage

GSoC 2019Tag
ActivePublic

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

Stuff related to Software Heritage participation into Google Summer of Code program, 2019 edition.

Recent Activity

Thu, Nov 3

bchauvet added a parent task for T4330: Deploy maven stack in production: T4079: Extend archive coverage.
Thu, Nov 3, 10:17 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Oct 19 2022

gitlab-migration changed the status of T4330: Deploy maven stack in production from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:07 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
gitlab-migration changed the status of T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, from Resolved to Migrated.
Oct 19 2022, 6:07 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
gitlab-migration changed the status of T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, from Resolved to Migrated.
Oct 19 2022, 6:05 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 8 2022

ardumont closed T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, as Resolved.
Sep 8 2022, 3:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont closed T4330: Deploy maven stack in production as Resolved.
Sep 8 2022, 3:03 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
vsellier added a comment to T4330: Deploy maven stack in production.

\o/ great

Sep 8 2022, 2:41 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont moved T4330: Deploy maven stack in production from in-progress to deployed/landed/monitoring on the System administration board.
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Checks:

  • task has been scheduled by the scheduler runner process [1]
  • listing is being consumed by one worker [2]
  • 'maven' listed origins is steadily growing [3]
  • New 'maven' listed origins are getting scheduled for ingestion [4]
  • maven loaders are ingesting those [5]
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:11 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8421: production: Update lister to consume maven listing.
Sep 8 2022, 2:11 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:06 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Schedule maven-central listing:

swhscheduler@saatchi:~$ curl -s https://repo1.maven.org/maven2/ | head -2
<!DOCTYPE html>
<html>
swhscheduler@saatchi:~$ curl -s https://maven-exporter.internal.softwareheritage.org/export-maven-central.fld | head -2
doc 0
  field 0
swhscheduler@saatchi:~$ curl -s http://saatchi.internal.softwareheritage.org:5008/
<html>
<head><title>Software Heritage scheduler RPC server</title></head>
<body>
<p>You have reached the
<a href="https://www.softwareheritage.org/">Software Heritage</a>
scheduler RPC server.<br />
See its
<a href="https://docs.softwareheritage.org/devel/swh-scheduler/">documentation
and API</a> for more information</p>
</body>
</html>swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   task add list-maven-full \
>     url=https://repo1.maven.org/maven2/ \
>     index_url=https://maven-exporter.internal.softwareheritage.org/export-maven-central.fld
Created 1 tasks
Sep 8 2022, 2:04 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 1:47 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Finally, export is done on maven central [1], the fld is computed [2]...
And it's also exposed, hence reachable from lister worker nodes.

Sep 8 2022, 1:47 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, from Open to Work in Progress.
Sep 8 2022, 11:18 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T4330: Deploy maven stack in production from Open to Work in Progress.
Sep 8 2022, 11:18 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a project to T4330: Deploy maven stack in production: System administration.
Sep 8 2022, 11:18 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:30 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:19 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:05 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 9:57 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 7 2022

ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 7 2022, 6:27 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 6 2022

ardumont added a revision to T4330: Deploy maven stack in production: D8399: production: Provision new maven-exporter node.
Sep 6 2022, 2:56 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8397: Deploy maven-exporter production node.
Sep 6 2022, 11:50 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Jun 16 2022

ardumont updated the task description for T4330: Deploy maven stack in production.
Jun 16 2022, 11:13 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont triaged T4330: Deploy maven stack in production as Normal priority.
Jun 16 2022, 10:21 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

May 13 2022

anlambert added projects to T1724: Maven Central repository support: Maven lister, Maven loader.
May 13 2022, 4:54 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Apr 29 2022

ardumont added a parent task for T1724: Maven Central repository support: Unknown Object (Maniphest Task).
Apr 29 2022, 2:26 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Apr 14 2022

ardumont closed T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, as Resolved.
Apr 14 2022, 3:23 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Feb 13 2022

borisbaldassari added a revision to T1724: Maven Central repository support: D7167: maven-docker: update export file to include svn/git entries..
Feb 13 2022, 10:56 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 24 2022

ardumont changed the status of T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, from Open to Work in Progress.
Jan 24 2022, 11:32 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 21 2022

borisbaldassari updated the task description for T1724: Maven Central repository support.
Jan 21 2022, 4:09 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D7014: Fix comment in doco.yml.
Jan 21 2022, 4:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 17 2022

ardumont updated the task description for T1724: Maven Central repository support.
Jan 17 2022, 3:00 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 15 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.

Jan 15 2022, 8:21 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D6955: maven: add index nginx host in docker dev setup.
Jan 15 2022, 8:06 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 10 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!

Jan 10 2022, 8:39 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 7 2022

ardumont added a comment to T1724: Maven Central repository support.

@borisbaldassari Hello, gentle ping about ^

Jan 7 2022, 4:01 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Dec 17 2021

ardumont updated subscribers of T1724: Maven Central repository support.

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual
machine alongside the swh setup to host the text index file, and make sure the swh vm
could access it. I suppose that any vm/docker/baremetal machine with an apache/nginx
server could do for that, as long as the lister can http-fetch the .fld file.

Dec 17 2021, 3:58 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
vlorentz updated the task description for T1724: Maven Central repository support.
Dec 17 2021, 9:49 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
vlorentz updated the task description for T1724: Maven Central repository support.
Dec 17 2021, 9:49 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Dec 8 2021

ardumont updated the task description for T1724: Maven Central repository support.
Dec 8 2021, 12:32 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a comment to T1724: Maven Central repository support.

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual machine alongside the swh setup to host the text index file, and make sure the swh vm could access it.
I suppose that any vm/docker/baremetal machine with an apache/nginx server could do for that, as long as the lister can http-fetch the .fld file.

Dec 8 2021, 12:18 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a comment to T1724: Maven Central repository support.

I'm asking you for a diff with the exact changes you had to make in the
swh-environment/docker/docker-compose.yml (and other folders) to actually make it run.
That will definitely help for the deployment on staging.

Dec 8 2021, 12:06 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D6784: maven: diff docker dev setup.
Dec 8 2021, 11:58 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T1724: Maven Central repository support.
Dec 8 2021, 11:31 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T1724: Maven Central repository support.

I'm not sure what you mean by the docker diff.

Dec 8 2021, 11:25 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Dec 6 2021

borisbaldassari added a comment to T1724: Maven Central repository support.

I'm not sure what you mean by the docker diff. Is that the update of the maven-index-exporter repository at D6740?
The above-mentioned repository has documentation to build, test and run the text index generation. As mentioned there I've also created a bunch of compressed text index exports, that can be used to test the lister/loader without running the docker image immediately. They are all real-world extracts obtained by running the docker image on the list of Maven repositories I could get as of last week. They together represent a few million artefacts.

Dec 6 2021, 10:16 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage