Page MenuHomeSoftware Heritage

GSoC 2019Tag
ActivePublic

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

Stuff related to Software Heritage participation into Google Summer of Code program, 2019 edition.

Recent Activity

Jan 8 2023

gitlab-migration closed T1724: Maven Central repository support as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:21 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
gitlab-migration changed the status of T1709: implement an R-cran lister from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:59 PM · GSoC 2019, Archive coverage
gitlab-migration changed the status of T1909: Enable to manually select the programming language to highlight from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:28 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1900: Improve image and pdf contents rendering from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:28 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1786: Improve popovers display on small screens from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:27 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1786: Improve popovers display on small screens, a subtask of T1477: Improve swh-web design for mobile browsing , from Resolved to Migrated.
Jan 8 2023, 4:27 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1769: Add unit tests for Javascript code, a subtask of T1768: Add end to end tests for the frontend part of swh-web, from Wontfix to Migrated.
Jan 8 2023, 4:27 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1770: Add end to end tests from Duplicate to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:27 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1770: Add end to end tests , a subtask of T1768: Add end to end tests for the frontend part of swh-web, from Duplicate to Migrated.
Jan 8 2023, 4:27 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1768: Add end to end tests for the frontend part of swh-web from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:27 PM · GSoC 2019, Web app
gitlab-migration changed the status of T1477: Improve swh-web design for mobile browsing from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:26 PM · GSoC 2019, Web app

Nov 3 2022

bchauvet added a parent task for T4330: Deploy maven stack in production: T4079: Extend archive coverage.
Nov 3 2022, 10:17 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Oct 19 2022

gitlab-migration changed the status of T4330: Deploy maven stack in production from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:07 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
gitlab-migration changed the status of T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, from Resolved to Migrated.
Oct 19 2022, 6:07 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
gitlab-migration changed the status of T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, from Resolved to Migrated.
Oct 19 2022, 6:05 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 8 2022

ardumont closed T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, as Resolved.
Sep 8 2022, 3:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont closed T4330: Deploy maven stack in production as Resolved.
Sep 8 2022, 3:03 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
vsellier added a comment to T4330: Deploy maven stack in production.

\o/ great

Sep 8 2022, 2:41 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont moved T4330: Deploy maven stack in production from in-progress to deployed/landed/monitoring on the System administration board.
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Checks:

  • task has been scheduled by the scheduler runner process [1]
  • listing is being consumed by one worker [2]
  • 'maven' listed origins is steadily growing [3]
  • New 'maven' listed origins are getting scheduled for ingestion [4]
  • maven loaders are ingesting those [5]
Sep 8 2022, 2:16 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:11 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8421: production: Update lister to consume maven listing.
Sep 8 2022, 2:11 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 2:06 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Schedule maven-central listing:

swhscheduler@saatchi:~$ curl -s https://repo1.maven.org/maven2/ | head -2
<!DOCTYPE html>
<html>
swhscheduler@saatchi:~$ curl -s https://maven-exporter.internal.softwareheritage.org/export-maven-central.fld | head -2
doc 0
  field 0
swhscheduler@saatchi:~$ curl -s http://saatchi.internal.softwareheritage.org:5008/
<html>
<head><title>Software Heritage scheduler RPC server</title></head>
<body>
<p>You have reached the
<a href="https://www.softwareheritage.org/">Software Heritage</a>
scheduler RPC server.<br />
See its
<a href="https://docs.softwareheritage.org/devel/swh-scheduler/">documentation
and API</a> for more information</p>
</body>
</html>swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   task add list-maven-full \
>     url=https://repo1.maven.org/maven2/ \
>     index_url=https://maven-exporter.internal.softwareheritage.org/export-maven-central.fld
Created 1 tasks
Sep 8 2022, 2:04 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 1:47 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a comment to T4330: Deploy maven stack in production.

Finally, export is done on maven central [1], the fld is computed [2]...
And it's also exposed, hence reachable from lister worker nodes.

Sep 8 2022, 1:47 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T4330: Deploy maven stack in production, a subtask of T1724: Maven Central repository support, from Open to Work in Progress.
Sep 8 2022, 11:18 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T4330: Deploy maven stack in production from Open to Work in Progress.
Sep 8 2022, 11:18 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a project to T4330: Deploy maven stack in production: System administration.
Sep 8 2022, 11:18 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:30 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:19 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 10:05 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 8 2022, 9:57 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 7 2022

ardumont updated the task description for T4330: Deploy maven stack in production.
Sep 7 2022, 6:27 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Sep 6 2022

ardumont added a revision to T4330: Deploy maven stack in production: D8399: production: Provision new maven-exporter node.
Sep 6 2022, 2:56 PM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added a revision to T4330: Deploy maven stack in production: D8397: Deploy maven-exporter production node.
Sep 6 2022, 11:50 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

Jun 16 2022

ardumont updated the task description for T4330: Deploy maven stack in production.
Jun 16 2022, 11:13 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont triaged T4330: Deploy maven stack in production as Normal priority.
Jun 16 2022, 10:21 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage

May 13 2022

anlambert added projects to T1724: Maven Central repository support: Maven lister, Maven loader.
May 13 2022, 4:54 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Apr 29 2022

ardumont added a parent task for T1724: Maven Central repository support: Unknown Object (Maniphest Task).
Apr 29 2022, 2:26 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Apr 14 2022

ardumont closed T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, as Resolved.
Apr 14 2022, 3:23 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Feb 13 2022

borisbaldassari added a revision to T1724: Maven Central repository support: D7167: maven-docker: update export file to include svn/git entries..
Feb 13 2022, 10:56 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 24 2022

ardumont changed the status of T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, from Open to Work in Progress.
Jan 24 2022, 11:32 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 21 2022

borisbaldassari updated the task description for T1724: Maven Central repository support.
Jan 21 2022, 4:09 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D7014: Fix comment in doco.yml.
Jan 21 2022, 4:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 17 2022

ardumont updated the task description for T1724: Maven Central repository support.
Jan 17 2022, 3:00 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 15 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.

Jan 15 2022, 8:21 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D6955: maven: add index nginx host in docker dev setup.
Jan 15 2022, 8:06 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 10 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!

Jan 10 2022, 8:39 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage