Page MenuHomeSoftware Heritage

vsellier (Vincent Sellier)
UserAdministrator

User Details

User Since
Mar 21 2019, 4:14 PM (97 w, 24 m)
Roles
Administrator

Recent Activity

Today

vsellier changed the status of T2975: Disk replacement on esnode1 from Open to Work in Progress.
Thu, Jan 28, 3:44 PM · System administration
vsellier closed T3001: Webapp is not displaying the origin type on the search results as Resolved.

The fix is deployed on webapp1 and solved the problem.

Thu, Jan 28, 3:33 PM · Storage manager, Web app
vsellier closed D4963: webapp1: use the same deployment pattern than moma.
Thu, Jan 28, 3:19 PM
vsellier committed rSPSITEb82b0d93c2ec: webapp1: use the same deployment pattern than moma (authored by vsellier).
webapp1: use the same deployment pattern than moma
Thu, Jan 28, 3:18 PM
vsellier requested review of D4963: webapp1: use the same deployment pattern than moma.
Thu, Jan 28, 3:10 PM
vsellier added a comment to T3001: Webapp is not displaying the origin type on the search results.

The storage version v0.21.1 is deployed in staging, the problem looks fixed :

❯ curl -s  https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/visit/latest/\?require_snapshot\=true | jq ''
{
  "origin": "https://gitlab.com/miwc/miwc.github.io.git",
  "date": "2020-12-07T18:21:58.967952+00:00",
  "type": "git",
  "visit": 1,
  "status": "full",
  "snapshot": "759b36e0e3e81e8cbf601181829571daa645b5d2",
  "metadata": {},
  "origin_url": "https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/get/",
  "snapshot_url": "https://webapp.staging.swh.network/api/1/snapshot/759b36e0e3e81e8cbf601181829571daa645b5d2/"
}
Thu, Jan 28, 2:36 PM · Storage manager, Web app
vsellier closed T2988: Improve cgit lister to add last modification date of the repos as Resolved.
Thu, Jan 28, 2:10 PM · CGit lister, Lister
vsellier closed D4960: Correctly return origin_visit_status.type value everywhere.
Thu, Jan 28, 2:01 PM
vsellier committed rDSTO76de53cb261f: Correctly return origin_visit_status.type value everywhere (authored by vsellier).
Correctly return origin_visit_status.type value everywhere
Thu, Jan 28, 2:01 PM
vsellier requested review of D4960: Correctly return origin_visit_status.type value everywhere.
Thu, Jan 28, 12:23 PM
vsellier added a revision to T3001: Webapp is not displaying the origin type on the search results: D4960: Correctly return origin_visit_status.type value everywhere.
Thu, Jan 28, 12:12 PM · Storage manager, Web app
vsellier added projects to T3001: Webapp is not displaying the origin type on the search results: Web app, Storage manager.
Thu, Jan 28, 12:11 PM · Storage manager, Web app
vsellier changed the status of T3001: Webapp is not displaying the origin type on the search results from Open to Work in Progress.
Thu, Jan 28, 12:11 PM · Storage manager, Web app
vsellier created P930 (An Untitled Masterwork).
Thu, Jan 28, 10:30 AM

Yesterday

vsellier added a comment to T2920: Document staging infrastructure.

This is a try to generate a global schema of the staging environment (P929):

Wed, Jan 27, 6:09 PM · Documentation, System administration, Staging environment
vsellier created P929 Staging infrastructure.
Wed, Jan 27, 6:07 PM
vsellier accepted D4956: launchpad: Actually mock the anonymous login to launchpad.

It seems to be ok :)

Wed, Jan 27, 4:32 PM
vsellier committed rDSNIP0fe3238bdabf: counters: batch redis calls (authored by vsellier).
counters: batch redis calls
Wed, Jan 27, 3:38 PM
vsellier committed rDSNIPe1076146c645: counters: add local counter to follow the message count (authored by vsellier).
counters: add local counter to follow the message count
Wed, Jan 27, 3:38 PM
vsellier closed D4954: cgit: Don't stop the listing when a repository page is not available.
Wed, Jan 27, 3:06 PM
vsellier committed rDLSf6f9f1ca28a9: cgit: Don't stop the listing when a repository page is not available (authored by vsellier).
cgit: Don't stop the listing when a repository page is not available
Wed, Jan 27, 3:06 PM
vsellier added a comment to D4954: cgit: Don't stop the listing when a repository page is not available.

Thanks :)

Wed, Jan 27, 3:05 PM
vsellier updated the diff for D4954: cgit: Don't stop the listing when a repository page is not available.

Use an exception to validate a repo page can be accessed

Wed, Jan 27, 2:54 PM
vsellier closed D4953: cgit: Add support for last_update information during listing.
Wed, Jan 27, 2:24 PM
vsellier committed rDLS91fcde83410d: cgit: Add support for last_update information during listing (authored by vsellier).
cgit: Add support for last_update information during listing
Wed, Jan 27, 2:24 PM
vsellier updated the diff for D4954: cgit: Don't stop the listing when a repository page is not available.

rebase

Wed, Jan 27, 2:19 PM
vsellier updated the diff for D4953: cgit: Add support for last_update information during listing.

Restore missing log when the date can't be parsed

Wed, Jan 27, 2:18 PM
vsellier updated the diff for D4954: cgit: Don't stop the listing when a repository page is not available.

rebase

Wed, Jan 27, 2:04 PM
vsellier updated the diff for D4953: cgit: Add support for last_update information during listing.

Remove useless variable

Wed, Jan 27, 2:03 PM
vsellier requested review of D4954: cgit: Don't stop the listing when a repository page is not available.
Wed, Jan 27, 12:47 PM
vsellier moved T2944: Deploy swh-search v0.4.1 from in-progress to deployed on the System administration board.
Wed, Jan 27, 12:44 PM · System administration, Journal, Archive search
vsellier added a revision to T2988: Improve cgit lister to add last modification date of the repos: D4954: cgit: Don't stop the listing when a repository page is not available.
Wed, Jan 27, 12:42 PM · CGit lister, Lister
vsellier updated the diff for D4953: cgit: Add support for last_update information during listing.
  • Reorder methods
  • Adapt date parsing according the review
Wed, Jan 27, 12:41 PM
vsellier added a comment to T2944: Deploy swh-search v0.4.1.

To decrease the time to recover the lag, several journal client were launched in // with :

/usr/bin/swh search --config-file /etc/softwareheritage/search/journal_client_objects.yml journal-client objects
Wed, Jan 27, 10:00 AM · System administration, Journal, Archive search
vsellier committed rSPREa7c9c625d98c: Allocate more memory to search1 (authored by vsellier).
Allocate more memory to search1
Wed, Jan 27, 9:40 AM

Tue, Jan 26

vsellier updated the diff for D4953: cgit: Add support for last_update information during listing.

Inline unecessary indirection

Tue, Jan 26, 6:52 PM
vsellier updated the diff for D4953: cgit: Add support for last_update information during listing.

Add missing test coverage

Tue, Jan 26, 6:49 PM
vsellier requested review of D4953: cgit: Add support for last_update information during listing.
Tue, Jan 26, 6:36 PM
vsellier added a revision to T2988: Improve cgit lister to add last modification date of the repos: D4953: cgit: Add support for last_update information during listing.
Tue, Jan 26, 6:33 PM · CGit lister, Lister
vsellier changed the status of T2988: Improve cgit lister to add last modification date of the repos from Open to Work in Progress.
Tue, Jan 26, 6:04 PM · CGit lister, Lister
vsellier accepted D4946: Install scheduler journal client to saatchi.

LGTM

Tue, Jan 26, 12:45 PM
vsellier added a comment to T2944: Deploy swh-search v0.4.1.

Upgrading the index configuration to speedup the indexation :

% cat >/tmp/config.json <<EOF
{
  "index" : {
"translog.sync_interval" : "60s",
"translog.durability": "async",
"refresh_interval": "60s"
  }
}
EOF
% export ES_SERVER=192.168.100.81:9200
% export INDEX=origin            
% curl -s -H "Content-Type: application/json" -XPUT http://${ES_SERVER}/${INDEX}/_settings -d @/tmp/config.json 
{"acknowledged":true}%
Tue, Jan 26, 10:31 AM · System administration, Journal, Archive search
vsellier added a comment to T2944: Deploy swh-search v0.4.1.

Production

  • puppet disabled
  • Services stopped :
root@search1:~# systemctl stop swh-search-journal-client@objects.service 
root@search1:~# systemctl stop gunicorn-swh-search
  • Index deleted and recreated
% export ES_SERVER=search-esnode1.internal.softwareheritage.org:9200
% curl -s http://$ES_SERVER/_cat/indices\?v 
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin Mq8dnlpuRXO4yYoC6CTuQw  90   1  151716299     38861934    260.8gb          131gb
% curl -XDELETE http://$ES_SERVER/origin
{"acknowledged":true}%    
% swh search --config-file /etc/softwareheritage/search/server.yml  initialize
INFO:elasticsearch:PUT http://search-esnode1.internal.softwareheritage.org:9200/origin [status:200 request:2.216s]
INFO:elasticsearch:PUT http://search-esnode3.internal.softwareheritage.org:9200/origin/_mapping [status:200 request:0.151s]
Done.
% curl -s http://$ES_SERVER/_cat/indices\?v                                        
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin yFaqPPCnRFCnc5AA6Ah8lw  90   1          0            0     36.5kb         18.2kb
  • journal client's consumer group delete:
% export SERVER=kafka1.internal.softwareheritage.org:9092  
% ./kafka-consumer-groups.sh --bootstrap-server ${SERVER} --delete --group swh.search.journal_client
Deletion of requested consumer groups ('swh.search.journal_client') was successful.
  • journal client restarted
  • puppet enabled
Tue, Jan 26, 9:39 AM · System administration, Journal, Archive search
vsellier added a comment to T2944: Deploy swh-search v0.4.1.

The filter on visited origins is working correctly on staging. The has_visit flag looks good.
For example for the https://www.npmjs.com/package/@ehmicky/dev-tasks origin

{
  "_index" : "origin",
  "_type" : "_doc",
  "_id" : "019bd314416108304165e82dd92e00bc9ea85a53",
  "_score" : 60.56421,
  "_source" : {
    "url" : "https://www.npmjs.com/package/@ehmicky/dev-tasks",
    "sha1" : "019bd314416108304165e82dd92e00bc9ea85a53"
  },
  "sort" : [
    60.56421,
    "019bd314416108304165e82dd92e00bc9ea85a53"
  ]
}
swh=> select * from origin join origin_visit_status on id=origin where id=469380;
   id   |                       url                        | origin | visit |             date              | status  | metadata |                  snapshot                  | type 
--------+--------------------------------------------------+--------+-------+-------------------------------+---------+----------+--------------------------------------------+------
 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 |     1 | 2021-01-25 13:30:47.221937+00 | created |          |                                            | npm
 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 |     1 | 2021-01-25 13:41:59.435579+00 | partial |          | \xe3f24413d81fd3e9c309686fcfb6c8f5eb549acf | npm
Tue, Jan 26, 9:16 AM · System administration, Journal, Archive search

Mon, Jan 25

vsellier closed D4943: cgit lister: Add missing types on the init method.
Mon, Jan 25, 6:59 PM
vsellier committed rDLSd62e77c1b495: cgit lister: Add missing types on the init method (authored by vsellier).
cgit lister: Add missing types on the init method
Mon, Jan 25, 6:59 PM
vsellier requested review of D4943: cgit lister: Add missing types on the init method.
Mon, Jan 25, 6:58 PM
vsellier added a revision to T2984: Port cgit lister to the new Lister API: D4943: cgit lister: Add missing types on the init method.
Mon, Jan 25, 6:33 PM · Lister, CGit lister, Sprint 2021 01
vsellier added a comment to T2944: Deploy swh-search v0.4.1.

Staging

We are proceeding to a complete index rebuilding

Mon, Jan 25, 5:44 PM · System administration, Journal, Archive search
vsellier added a comment to T2944: Deploy swh-search v0.4.1.

Regarding the index rebuilding process, using a naive approach with aliases with the old and the new index[1] returns duplicated results when the search is done.
Using an alias with only the old index, rebuilding a new index and switching the alias to the new index[2] can be a first approach with the default the old index will not be updated until the alias is switched to the new index.
It also requires the swh-search code is able to use different names for the read and write operations.

Mon, Jan 25, 4:07 PM · System administration, Journal, Archive search
vsellier changed the status of T2920: Document staging infrastructure from Open to Work in Progress.
Mon, Jan 25, 3:32 PM · Documentation, System administration, Staging environment
vsellier changed the status of T2944: Deploy swh-search v0.4.1, a subtask of T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit, from Open to Work in Progress.
Mon, Jan 25, 3:32 PM · Journal, Archive search
vsellier changed the status of T2944: Deploy swh-search v0.4.1 from Open to Work in Progress.
Mon, Jan 25, 3:32 PM · System administration, Journal, Archive search
vsellier renamed T2944: Deploy swh-search v0.4.1 from Deploy swh-search v0.4.1 in staging to Deploy swh-search v0.4.1.
Mon, Jan 25, 3:32 PM · System administration, Journal, Archive search
vsellier accepted D4939: gitlab: Adapt celery task implementations to the new lister api.

LGTM

Mon, Jan 25, 3:13 PM
vsellier closed D4926: Port cgit lister to the new lister api.
Mon, Jan 25, 3:06 PM · Sprint 2021 01, Lister
vsellier committed rDLSe4a590fc7f3e: Port cgit lister to the new lister api (authored by vsellier).
Port cgit lister to the new lister api
Mon, Jan 25, 3:06 PM
vsellier updated the diff for D4926: Port cgit lister to the new lister api.
  • rebase
  • update tests according to the review feedbacks
Mon, Jan 25, 2:58 PM · Sprint 2021 01, Lister
vsellier committed rSPSITE8e78d5b0edd4: Increase the deposit end-to-end monitoring timeouts (authored by vsellier).
Increase the deposit end-to-end monitoring timeouts
Mon, Jan 25, 2:10 PM
vsellier added a comment to T2912: Next generation archive counters.

It seems redis has a Hyperloglog functionnality[1] that can match with the requirements (bloom filter / limited deviation / small memory footprint / efficiency).

Mon, Jan 25, 12:52 PM · System administration, Monitoring, Web app
vsellier committed rDSNIPc5aa5b9d15e5: POC for counters with redis hyperloglog (authored by vsellier).
POC for counters with redis hyperloglog
Mon, Jan 25, 12:42 PM
vsellier added inline comments to D4926: Port cgit lister to the new lister api.
Mon, Jan 25, 12:17 PM · Sprint 2021 01, Lister
vsellier added a revision to T2976: Deposit tests end-to-end are failing in icinga: Unknown Object (Differential Revision).
Mon, Jan 25, 11:48 AM · System administration, SWORD deposit
vsellier triaged T2988: Improve cgit lister to add last modification date of the repos as Normal priority.
Mon, Jan 25, 11:40 AM · CGit lister, Lister
vsellier updated the task description for T2967: Write journal client subcribed to origin_visit_status topics .
Mon, Jan 25, 10:56 AM · Sprint 2021 01, Scheduling utilities

Sat, Jan 23

vsellier committed rDSCH3d13cda421b9: Solve uneventful/eventful with unordered messages with snapshots (authored by vsellier).
Solve uneventful/eventful with unordered messages with snapshots
Sat, Jan 23, 8:00 PM
vsellier committed rDSCHd528998df596: Do not consider duplicated messages as uneventful event (authored by vsellier).
Do not consider duplicated messages as uneventful event
Sat, Jan 23, 8:00 PM

Fri, Jan 22

vsellier requested review of D4926: Port cgit lister to the new lister api.
Fri, Jan 22, 4:24 PM · Sprint 2021 01, Lister
vsellier added a revision to T2984: Port cgit lister to the new Lister API: D4926: Port cgit lister to the new lister api.
Fri, Jan 22, 3:53 PM · Lister, CGit lister, Sprint 2021 01
vsellier closed T2978: Deploy visit-stats journal client on staging as Resolved.
Fri, Jan 22, 11:12 AM · Sprint 2021 01, Scheduling utilities
vsellier closed T2978: Deploy visit-stats journal client on staging, a subtask of T2967: Write journal client subcribed to origin_visit_status topics , as Resolved.
Fri, Jan 22, 11:12 AM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Fri, Jan 22, 11:12 AM · Sprint 2021 01, Scheduling utilities
vsellier moved T2984: Port cgit lister to the new Lister API from Backlog to in-progress on the Sprint 2021 01 board.
Fri, Jan 22, 11:09 AM · Lister, CGit lister, Sprint 2021 01
vsellier changed the status of T2984: Port cgit lister to the new Lister API from Open to Work in Progress.
Fri, Jan 22, 11:09 AM · Lister, CGit lister, Sprint 2021 01

Thu, Jan 21

vsellier closed D4908: Do not consider duplicated messages as uneventful event.
Thu, Jan 21, 7:28 PM
vsellier closed D4918: Solve uneventful/eventful with unordered messages with snapshots.
Thu, Jan 21, 7:28 PM
vsellier committed rDSCH25d036ef4503: Do not consider duplicated messages as uneventful event (authored by vsellier).
Do not consider duplicated messages as uneventful event
Thu, Jan 21, 7:28 PM
vsellier committed rDSCH82b7a8a42367: Solve uneventful/eventful with unordered messages with snapshots (authored by vsellier).
Solve uneventful/eventful with unordered messages with snapshots
Thu, Jan 21, 7:28 PM
vsellier updated the diff for D4918: Solve uneventful/eventful with unordered messages with snapshots.

rebase

Thu, Jan 21, 7:17 PM
vsellier updated the diff for D4908: Do not consider duplicated messages as uneventful event.

rebase

Thu, Jan 21, 7:16 PM
vsellier requested review of D4918: Solve uneventful/eventful with unordered messages with snapshots.
Thu, Jan 21, 7:10 PM
vsellier added a revision to T2978: Deploy visit-stats journal client on staging: D4918: Solve uneventful/eventful with unordered messages with snapshots.
Thu, Jan 21, 7:06 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the diff for D4908: Do not consider duplicated messages as uneventful event.

Test with "several" database upserts as it's more realistic

Thu, Jan 21, 7:06 PM
vsellier requested review of D4908: Do not consider duplicated messages as uneventful event.
Thu, Jan 21, 5:19 PM
vsellier added a revision to T2978: Deploy visit-stats journal client on staging: D4908: Do not consider duplicated messages as uneventful event.
Thu, Jan 21, 5:17 PM · Sprint 2021 01, Scheduling utilities
vsellier closed D4884: Deploy the scheduler's journal client.
Thu, Jan 21, 2:42 PM
vsellier committed rSPSITE9a67834c5b94: Deploy the scheduler's journal client (authored by vsellier).
Deploy the scheduler's journal client
Thu, Jan 21, 2:42 PM
vsellier closed D4901: scheduler: deploy journal client service.
Thu, Jan 21, 12:16 PM
vsellier committed rDENV6687fd022604: scheduler: deploy journal client service (authored by vsellier).
scheduler: deploy journal client service
Thu, Jan 21, 12:16 PM
vsellier updated the diff for D4901: scheduler: deploy journal client service.

split the long entrypoint's command line

Thu, Jan 21, 12:14 PM
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Thu, Jan 21, 11:48 AM · Sprint 2021 01, Scheduling utilities
vsellier requested review of D4901: scheduler: deploy journal client service.
Thu, Jan 21, 11:47 AM
vsellier added a revision to T2978: Deploy visit-stats journal client on staging: D4901: scheduler: deploy journal client service.
Thu, Jan 21, 11:47 AM · Sprint 2021 01, Scheduling utilities
vsellier updated the diff for D4884: Deploy the scheduler's journal client.

rebase

Thu, Jan 21, 9:57 AM
vsellier moved T2919: Decomission webapp0 node (azure) from deployed to done on the System administration board.
Thu, Jan 21, 9:47 AM · System administration
vsellier moved T2958: Use all the disks on esnode2 and esnode3 from in-progress to deployed on the System administration board.
Thu, Jan 21, 9:46 AM · System administration
vsellier moved T2905: Deploy swh-search for production from in-progress to deployed on the System administration board.
Thu, Jan 21, 9:46 AM · System administration, Journal, Archive search
vsellier moved T2903: Test different disk configuration on esnode1 from in-progress to deployed on the System administration board.
Thu, Jan 21, 9:46 AM · System administration
vsellier moved T2888: Elasticsearch cluster failure during a rolling restart from in-progress to deployed on the System administration board.
Thu, Jan 21, 9:46 AM · System administration
vsellier moved T2976: Deposit tests end-to-end are failing in icinga from in-progress to deployed on the System administration board.
Thu, Jan 21, 9:46 AM · System administration, SWORD deposit