Page MenuHomeSoftware Heritage

anlambert (Antoine Lambert)
User

Projects (7)

User Details

User Since
Jul 3 2017, 10:14 AM (212 w, 4 d)

Recent Activity

Sat, Jul 24

anlambert added a comment to T3442: Make GitLab lister more robust to HTTP errors .
swh-lister_1                     | [2021-07-24 08:58:25,127: INFO/ForkPoolWorker-1] Task swh.lister.gitlab.tasks.FullGitLabRelister[717c82b2-175a-492c-b701-22b4fd34e5e2] succeeded in 139476.781209187s: {'pages': 27470, 'origins': 2746838}
Sat, Jul 24, 11:47 AM · Lister

Fri, Jul 23

anlambert updated the diff for D6004: misc/coverage: Revamp archive coverage widget (WIP).

Update: Activate origins search links in collapsible elements.

Fri, Jul 23, 5:49 PM
anlambert created P1098 (An Untitled Masterwork).
Fri, Jul 23, 4:59 PM
anlambert closed T3442: Make GitLab lister more robust to HTTP errors as Resolved.

Closing this as requested.

Fri, Jul 23, 4:48 PM · Lister
anlambert closed D6021: gitlab: Update requests query parameters.
Fri, Jul 23, 3:15 PM
anlambert closed D6022: gitlab: Handle HTTP status code 500 when listing projects.
Fri, Jul 23, 3:15 PM
anlambert committed rDLSe904f4760ef1: gitlab: Handle HTTP status code 500 when listing projects (authored by anlambert).
gitlab: Handle HTTP status code 500 when listing projects
Fri, Jul 23, 3:15 PM
anlambert committed rDLS52c315015543: gitlab: Update requests query parameters (authored by anlambert).
gitlab: Update requests query parameters
Fri, Jul 23, 3:15 PM
anlambert updated the diff for D6022: gitlab: Handle HTTP status code 500 when listing projects.

Fix typo.

Fri, Jul 23, 3:07 PM
anlambert added a comment to T3405: GraphQL apis for SWH.

I stumbled across GitLab GraphQL API while working on T3442, could be a great source of inspiration.

Fri, Jul 23, 2:45 PM · Web app
anlambert requested review of D6022: gitlab: Handle HTTP status code 500 when listing projects.
Fri, Jul 23, 2:40 PM
anlambert added a comment to T3442: Make GitLab lister more robust to HTTP errors .

For the record, my lister is still running, 1320500 gitlab.com origins listed so far.

Fri, Jul 23, 2:39 PM · Lister
anlambert added a revision to T3442: Make GitLab lister more robust to HTTP errors : D6022: gitlab: Handle HTTP status code 500 when listing projects.
Fri, Jul 23, 2:37 PM · Lister
anlambert requested review of D6021: gitlab: Update requests query parameters.
Fri, Jul 23, 2:11 PM
anlambert closed D6020: gitlab: Adapt requests retry policy to consider HTTP 50x status codes.
Fri, Jul 23, 2:06 PM
anlambert committed rDLS73f85c0b8a42: gitlab: Adapt requests retry policy to consider HTTP 50x status codes (authored by anlambert).
gitlab: Adapt requests retry policy to consider HTTP 50x status codes
Fri, Jul 23, 2:06 PM
anlambert requested review of D6020: gitlab: Adapt requests retry policy to consider HTTP 50x status codes.
Fri, Jul 23, 1:57 PM
anlambert added a revision to T3442: Make GitLab lister more robust to HTTP errors : D6020: gitlab: Adapt requests retry policy to consider HTTP 50x status codes.
Fri, Jul 23, 1:54 PM · Lister
anlambert updated the task description for T3442: Make GitLab lister more robust to HTTP errors .
Fri, Jul 23, 12:19 PM · Lister
anlambert added a comment to T3127: Compute and display distribution of origins by forge.
In T3127#67581, @anlambert wrote:

    I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Indeed there is something weird here as we have more than one million gitlab.com origins in database.

softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%';
  count  
---------
 1023499
(1 row)

Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.

Indeed, please do look into this, thanks.

Fri, Jul 23, 12:17 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert triaged T3442: Make GitLab lister more robust to HTTP errors as Normal priority.
Fri, Jul 23, 12:14 PM · Lister

Thu, Jul 22

anlambert updated the summary of D6004: misc/coverage: Revamp archive coverage widget (WIP).
Thu, Jul 22, 6:23 PM
anlambert updated the diff for D6004: misc/coverage: Revamp archive coverage widget (WIP).
  • Improve origin category texts
  • Fix code retrieving lister metrics as some of them were dicarded
  • Add display of distribution of origins per origins type, hidden by default, can be displayed by clicking on any collapsible elements (total origin counts)
Thu, Jul 22, 6:16 PM
anlambert updated the summary of D6004: misc/coverage: Revamp archive coverage widget (WIP).
Thu, Jul 22, 6:16 PM
anlambert triaged T3441: Implement query to get origin visit types dynamically as Normal priority.
Thu, Jul 22, 2:41 PM · Archive search
anlambert created P1097 (An Untitled Masterwork).
Thu, Jul 22, 2:05 PM
anlambert created P1096 (An Untitled Masterwork).
Thu, Jul 22, 12:59 PM
anlambert added a comment to T3127: Compute and display distribution of origins by forge.

Thanks for these details: this count is missing the 800k git origins: @ardumont and @olasd should be able to tell you how to find them

Thu, Jul 22, 12:29 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task

Wed, Jul 21

anlambert closed D6013: assets/guided_tour: Improve tour and fix redirection bug.
Wed, Jul 21, 5:58 PM
anlambert committed rDWAPPS481647f863a6: assets/guided_tour: Improve tour and fix redirection bug (authored by anlambert).
assets/guided_tour: Improve tour and fix redirection bug
Wed, Jul 21, 5:58 PM
anlambert updated the diff for D6013: assets/guided_tour: Improve tour and fix redirection bug.

Address @ardumont comment.

Wed, Jul 21, 5:34 PM
anlambert added a comment to T3127: Compute and display distribution of origins by forge.

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Wed, Jul 21, 5:26 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert updated subscribers of D6013: assets/guided_tour: Improve tour and fix redirection bug.
Wed, Jul 21, 3:26 PM
anlambert added a comment to T3127: Compute and display distribution of origins by forge.

Instead, we could split the coverage widget into two tabs

  • one giving a high level overview of the archived origins, similar to what we have now with logos and counters
  • one giving the details of all forges we archived so far, displayed in a table as you suggested with relevant metrics and links to search origins for a given forge
Wed, Jul 21, 3:23 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert requested review of D6013: assets/guided_tour: Improve tour and fix redirection bug.
Wed, Jul 21, 2:31 PM
anlambert added a revision to T3437: Review guided tour and collect feedback from testers: D6013: assets/guided_tour: Improve tour and fix redirection bug.
Wed, Jul 21, 2:13 PM · Web app
anlambert added inline comments to D6005: docs/query-language: Describe search query language syntax.
Wed, Jul 21, 11:18 AM

Tue, Jul 20

anlambert updated the diff for D6004: misc/coverage: Revamp archive coverage widget (WIP).

Rebase

Tue, Jul 20, 4:28 PM
anlambert closed D6008: cypress: Upgrade to 8.0 and fix flaky test.
Tue, Jul 20, 4:27 PM
anlambert committed rDWAPPS914c19e314c7: cypress: Upgrade to 8.0 and fix flaky test (authored by anlambert).
cypress: Upgrade to 8.0 and fix flaky test
Tue, Jul 20, 4:27 PM
anlambert added a comment to D6008: cypress: Upgrade to 8.0 and fix flaky test.

The diff seems to fix the previously observed flaky test but got bitten by another one which does not happen so often thanks to cypress retry features.

Tue, Jul 20, 4:23 PM
anlambert committed rDDOCd200a98043d5: faq: Fix some typos and link displays (authored by anlambert).
faq: Fix some typos and link displays
Tue, Jul 20, 3:27 PM
anlambert requested review of D6008: cypress: Upgrade to 8.0 and fix flaky test.
Tue, Jul 20, 3:22 PM
anlambert updated the summary of D6004: misc/coverage: Revamp archive coverage widget (WIP).
Tue, Jul 20, 12:58 PM
anlambert updated the diff for D6004: misc/coverage: Revamp archive coverage widget (WIP).
  • Add paragraphs detailing each origins category
  • Add missing Phabricator origins in listed ones
  • Address @ardumont comment about doi.org netlocs for deposits
  • Use only png logos to simplify their integration
  • Resize some large logos
  • Remove no longer used logos
Tue, Jul 20, 12:57 PM
anlambert closed D6007: common/utils: Wrap deposits list retrieval in a function.
Tue, Jul 20, 12:52 PM
anlambert committed rDWAPPSc5d441b1a8ec: common/utils: Wrap deposits list retrieval in a function (authored by anlambert).
common/utils: Wrap deposits list retrieval in a function
Tue, Jul 20, 12:52 PM
anlambert added inline comments to D6004: misc/coverage: Revamp archive coverage widget (WIP).
Tue, Jul 20, 11:54 AM
anlambert added inline comments to D6007: common/utils: Wrap deposits list retrieval in a function.
Tue, Jul 20, 11:17 AM

Mon, Jul 19

anlambert requested review of D6004: misc/coverage: Revamp archive coverage widget (WIP).
Mon, Jul 19, 6:03 PM
anlambert requested review of D6007: common/utils: Wrap deposits list retrieval in a function.
Mon, Jul 19, 5:46 PM
anlambert added a revision to T3127: Compute and display distribution of origins by forge: D6007: common/utils: Wrap deposits list retrieval in a function.
Mon, Jul 19, 5:29 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert added a comment to T3202: Help new users discover the features available in the archive browsing view.

First version of guided tour has been deployed to production for ambassador and staff users.

Mon, Jul 19, 4:17 PM · Web app
anlambert added a comment to T3127: Compute and display distribution of origins by forge.

I think we could also get an accurate count of deposit origins (HAL, IPOL) using swh-deposit API

Mon, Jul 19, 3:54 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert committed rDWAPPSf667a87aa90b: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
Mon, Jul 19, 2:25 PM
anlambert committed rDWAPPS779aeb2d2665: templates/layout: Do not display swh-web version in production (authored by anlambert).
templates/layout: Do not display swh-web version in production
Mon, Jul 19, 2:25 PM
anlambert closed D5819: assets/webapp: Add UI guided tour.
Mon, Jul 19, 2:25 PM
anlambert committed rDWAPPS992bea293d25: assets/webapp: Add UI guided tour to help users discover features (authored by anlambert).
assets/webapp: Add UI guided tour to help users discover features
Mon, Jul 19, 2:25 PM
anlambert updated the diff for D5819: assets/webapp: Add UI guided tour.

Fix a guided tour step, element id was removed after a rebase.

Mon, Jul 19, 2:04 PM
anlambert committed rDDOC60c624c0bab4: sphinx/conf.py: Always filter out httpdomain role warnings (authored by anlambert).
sphinx/conf.py: Always filter out httpdomain role warnings
Mon, Jul 19, 12:19 PM
anlambert updated the diff for D5819: assets/webapp: Add UI guided tour.

Rebase

Mon, Jul 19, 11:18 AM

Fri, Jul 16

anlambert committed rCDFJ949068b605f5: base-buster: Install emscripten required by swh-search (authored by anlambert).
base-buster: Install emscripten required by swh-search
Fri, Jul 16, 5:10 PM
anlambert closed D6003: base-buster: Install emscripten required by swh-search.
Fri, Jul 16, 5:10 PM
anlambert updated the diff for D6003: base-buster: Install emscripten required by swh-search.

Remove added empty line

Fri, Jul 16, 5:10 PM
anlambert added inline comments to D6003: base-buster: Install emscripten required by swh-search.
Fri, Jul 16, 5:09 PM
anlambert updated the diff for D6003: base-buster: Install emscripten required by swh-search.

Update:

  • install as root and in /opt
  • update commit message
Fri, Jul 16, 5:01 PM
anlambert requested review of D6003: base-buster: Install emscripten required by swh-search.
Fri, Jul 16, 4:50 PM
anlambert added a comment to T3127: Compute and display distribution of origins by forge.

Only one nit about the display. Using modal windows/popover will mean that there will be no easy way to have, as a user, the full list: one will have to click on each logo one by one, which could be quite annoying. Would it be possible to have a page with a rendering of the table above? (not sure if we want all columns, but at least the last update time and the number of origins per forge instance looks relevant and interesting to me). It coule be either in addition of what you propose (e.g., as a "coverage details" link, leading to the full page), or as a replacement of it (e.g., by making each forge icon just a link to the relevant anchor within the table on the "coverage details" page).

Fri, Jul 16, 11:43 AM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert closed D5995: sphinx/conf.py: Filter out httdomain related warning since sphinx 4.1.
Fri, Jul 16, 11:28 AM
anlambert committed rDDOC3cd7ed54187e: sphinx/conf.py: Filter out httdomain related warning since sphinx 4.1 (authored by anlambert).
sphinx/conf.py: Filter out httdomain related warning since sphinx 4.1
Fri, Jul 16, 11:28 AM
anlambert added a comment to D5990: query_language: Setup tree-sitter and grammar.js.

Yes, that's a good point.
On the other hand, not including it means we install-depend on nodejs+yarn and a C compiler...

Fri, Jul 16, 11:18 AM

Thu, Jul 15

anlambert requested changes to D5990: query_language: Setup tree-sitter and grammar.js.

I added a couple of inline comments for some issues to fix and some improvements to add.

Thu, Jul 15, 3:36 PM
anlambert updated the summary of D5995: sphinx/conf.py: Filter out httdomain related warning since sphinx 4.1.
Thu, Jul 15, 2:33 PM
anlambert updated the summary of D5995: sphinx/conf.py: Filter out httdomain related warning since sphinx 4.1.
Thu, Jul 15, 2:33 PM
anlambert requested review of D5995: sphinx/conf.py: Filter out httdomain related warning since sphinx 4.1.
Thu, Jul 15, 2:32 PM

Tue, Jul 13

anlambert added a comment to D5990: query_language: Setup tree-sitter and grammar.js.

@anlambert We are going to need to precompile some assets in swh-search using JS dependencies. We define the grammar in JS, which can then be compile into a big JSON file, that will be read from Python; see swh/search/parser/package.json. Do you have some advice to make this work nicely?

Tue, Jul 13, 4:39 PM
anlambert added a comment to T3127: Compute and display distribution of origins by forge.

Some reports of what have been done so far and some future directions regarding the display of those data in swh-web.

Tue, Jul 13, 3:39 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert changed the visibility for F4592793: image.png.
Tue, Jul 13, 3:00 PM
anlambert closed D5989: pattern: Use URL network location as instance name when not provided.
Tue, Jul 13, 2:34 PM
anlambert committed rDLS6c123508634c: pattern: Use URL network location as instance name when not provided (authored by anlambert).
pattern: Use URL network location as instance name when not provided
Tue, Jul 13, 2:34 PM
anlambert requested review of D5989: pattern: Use URL network location as instance name when not provided.
Tue, Jul 13, 12:43 PM
anlambert added a revision to T3403: Use forge URL network location as default lister instance name: D5989: pattern: Use URL network location as instance name when not provided.
Tue, Jul 13, 12:40 PM · Scheduling utilities, Lister
anlambert added a comment to T3403: Use forge URL network location as default lister instance name.

It seems the remaining lister instances to process are the phabricator ones that also need credentials.
This is what we currently have in the listers table in scheduler database.

Tue, Jul 13, 11:47 AM · Scheduling utilities, Lister

Fri, Jul 9

anlambert created P1093 (An Untitled Masterwork).
Fri, Jul 9, 6:23 PM
anlambert committed rDWAPPS4a9cf1d61852: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
Fri, Jul 9, 6:00 PM
anlambert closed T3368: Upgrade to highlight.js 11.x as Resolved by committing rDWAPPS66b3a6c480c8: assets: Upgrade highlight.js to version 11.x.
Fri, Jul 9, 6:00 PM · Web app
anlambert closed D5987: assets: Upgrade highlight.js to version 11.x.
Fri, Jul 9, 6:00 PM
anlambert committed rDWAPPS66b3a6c480c8: assets: Upgrade highlight.js to version 11.x (authored by anlambert).
assets: Upgrade highlight.js to version 11.x
Fri, Jul 9, 6:00 PM
anlambert added inline comments to D5987: assets: Upgrade highlight.js to version 11.x.
Fri, Jul 9, 5:43 PM
anlambert requested review of D5987: assets: Upgrade highlight.js to version 11.x.
Fri, Jul 9, 5:28 PM
anlambert added a revision to T3368: Upgrade to highlight.js 11.x: D5987: assets: Upgrade highlight.js to version 11.x.
Fri, Jul 9, 5:11 PM · Web app
anlambert added a comment to D5819: assets/webapp: Add UI guided tour.

Build has FAILED

Patch application report for D5819 (id=21588)

Rebasing onto 3bfc6942fe...

Current branch diff-target is up to date.
Changes applied before test
commit c8e32e745aa3e76fc8d972cc89f6d01757e34e73
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Wed May 12 11:42:40 2021 +0200

    assets/webapp: Add UI guided tour to help users discover features
    
    Add a Web UI guided tour implemented using the intro.js library.
    
    The tour showcases the main browsing features offered by Software
    Heritage web application by highlighting the relevant DOM elements
    and displaying help texts in tooltips.
    
    The tour can span multiple web pages and steps are configured in
    a YAML file for help texts writing commodity.
    
    Currently, the homepage, the main origin browsing view and content
    browsing view are covered by the tour.
    
    Related to T3202

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/962/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/962/console

Fri, Jul 9, 4:39 PM
anlambert updated the diff for D5819: assets/webapp: Add UI guided tour.

Rebase

Fri, Jul 9, 4:17 PM
anlambert closed T3402: Deploy swh-counters v0.8.0 and backfill origins, a subtask of T3127: Compute and display distribution of origins by forge, as Wontfix.
Fri, Jul 9, 2:34 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
anlambert closed T3402: Deploy swh-counters v0.8.0 and backfill origins as Wontfix.

Precise metrics about listed origins and their counts will be retrieved from the scheduler database, no need to backfill origins with swh-counters then, closing this.

Fri, Jul 9, 2:34 PM · Counters, System administration, Metrics/monitoring
anlambert accepted D5984: apidoc: Remove explicit recursion used in visit_field_list.

Better indeed, thanks !

Fri, Jul 9, 1:54 PM
anlambert accepted D5971: apidoc: Stop parsing docutils trees with regexps on its pseudo-XML.

Looks good to me !

Fri, Jul 9, 12:34 PM
anlambert closed D5969: templates/snapshot-context: Visit origin URL in a new browser tab.
Fri, Jul 9, 12:20 PM
anlambert committed rDWAPPSe12202c4b373: templates/snapshot-context: Visit origin URL in a new browser tab (authored by anlambert).
templates/snapshot-context: Visit origin URL in a new browser tab
Fri, Jul 9, 12:20 PM
anlambert closed D5982: cypress/layout: Fix cypress anti-pattern use.
Fri, Jul 9, 12:20 PM