- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Jan 27 2021
Jan 27 2021
vsellier committed rDLSf6f9f1ca28a9: cgit: Don't stop the listing when a repository page is not available (authored by vsellier).
cgit: Don't stop the listing when a repository page is not available
vsellier added a comment to D4954: cgit: Don't stop the listing when a repository page is not available.
Thanks :)
vsellier updated the diff for D4954: cgit: Don't stop the listing when a repository page is not available.
Use an exception to validate a repo page can be accessed
vsellier committed rDLS91fcde83410d: cgit: Add support for last_update information during listing (authored by vsellier).
cgit: Add support for last_update information during listing
vsellier updated the diff for D4954: cgit: Don't stop the listing when a repository page is not available.
rebase
Restore missing log when the date can't be parsed
vsellier updated the diff for D4954: cgit: Don't stop the listing when a repository page is not available.
rebase
Remove useless variable
vsellier requested review of D4954: cgit: Don't stop the listing when a repository page is not available.
vsellier moved T2944: Deploy swh-search v0.4.1 from in-progress to deployed/landed/monitoring on the System administration board.
- Reorder methods
- Adapt date parsing according the review
To decrease the time to recover the lag, several journal client were launched in // with :
/usr/bin/swh search --config-file /etc/softwareheritage/search/journal_client_objects.yml journal-client objects
Allocate more memory to search1
Jan 26 2021
Jan 26 2021
Inline unecessary indirection
Add missing test coverage
vsellier changed the status of T2988: Improve cgit lister to add last modification date of the repos from Open to Work in Progress.
Upgrading the index configuration to speedup the indexation :
% cat >/tmp/config.json <<EOF
{
"index" : {
"translog.sync_interval" : "60s",
"translog.durability": "async",
"refresh_interval": "60s"
}
}
EOF
% export ES_SERVER=192.168.100.81:9200
% export INDEX=origin
% curl -s -H "Content-Type: application/json" -XPUT http://${ES_SERVER}/${INDEX}/_settings -d @/tmp/config.json
{"acknowledged":true}%Production
- puppet disabled
- Services stopped :
root@search1:~# systemctl stop swh-search-journal-client@objects.service root@search1:~# systemctl stop gunicorn-swh-search
- Index deleted and recreated
% export ES_SERVER=search-esnode1.internal.softwareheritage.org:9200
% curl -s http://$ES_SERVER/_cat/indices\?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open origin Mq8dnlpuRXO4yYoC6CTuQw 90 1 151716299 38861934 260.8gb 131gb
% curl -XDELETE http://$ES_SERVER/origin
{"acknowledged":true}%
% swh search --config-file /etc/softwareheritage/search/server.yml initialize
INFO:elasticsearch:PUT http://search-esnode1.internal.softwareheritage.org:9200/origin [status:200 request:2.216s]
INFO:elasticsearch:PUT http://search-esnode3.internal.softwareheritage.org:9200/origin/_mapping [status:200 request:0.151s]
Done.
% curl -s http://$ES_SERVER/_cat/indices\?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open origin yFaqPPCnRFCnc5AA6Ah8lw 90 1 0 0 36.5kb 18.2kb- journal client's consumer group delete:
% export SERVER=kafka1.internal.softwareheritage.org:9092
% ./kafka-consumer-groups.sh --bootstrap-server ${SERVER} --delete --group swh.search.journal_client
Deletion of requested consumer groups ('swh.search.journal_client') was successful.- journal client restarted
- puppet enabled
The filter on visited origins is working correctly on staging. The has_visit flag looks good.
For example for the https://www.npmjs.com/package/@ehmicky/dev-tasks origin
{
"_index" : "origin",
"_type" : "_doc",
"_id" : "019bd314416108304165e82dd92e00bc9ea85a53",
"_score" : 60.56421,
"_source" : {
"url" : "https://www.npmjs.com/package/@ehmicky/dev-tasks",
"sha1" : "019bd314416108304165e82dd92e00bc9ea85a53"
},
"sort" : [
60.56421,
"019bd314416108304165e82dd92e00bc9ea85a53"
]
}swh=> select * from origin join origin_visit_status on id=origin where id=469380; id | url | origin | visit | date | status | metadata | snapshot | type --------+--------------------------------------------------+--------+-------+-------------------------------+---------+----------+--------------------------------------------+------ 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 | 1 | 2021-01-25 13:30:47.221937+00 | created | | | npm 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 | 1 | 2021-01-25 13:41:59.435579+00 | partial | | \xe3f24413d81fd3e9c309686fcfb6c8f5eb549acf | npm
Jan 25 2021
Jan 25 2021
vsellier committed rDLSd62e77c1b495: cgit lister: Add missing types on the init method (authored by vsellier).
cgit lister: Add missing types on the init method
Staging
We are proceeding to a complete index rebuilding
Regarding the index rebuilding process, using a naive approach with aliases with the old and the new index[1] returns duplicated results when the search is done.
Using an alias with only the old index, rebuilding a new index and switching the alias to the new index[2] can be a first approach with the default the old index will not be updated until the alias is switched to the new index.
It also requires the swh-search code is able to use different names for the read and write operations.
vsellier changed the status of T2920: Document staging infrastructure from Open to Work in Progress.
vsellier changed the status of T2944: Deploy swh-search v0.4.1, a subtask of T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit, from Open to Work in Progress.
vsellier renamed T2944: Deploy swh-search v0.4.1 from Deploy swh-search v0.4.1 in staging to Deploy swh-search v0.4.1.
LGTM
Port cgit lister to the new lister api
- rebase
- update tests according to the review feedbacks
vsellier committed rSPSITE8e78d5b0edd4: Increase the deposit end-to-end monitoring timeouts (authored by vsellier).
Increase the deposit end-to-end monitoring timeouts
It seems redis has a Hyperloglog functionnality[1] that can match with the requirements (bloom filter / limited deviation / small memory footprint / efficiency).
vsellier committed rDSNIPc5aa5b9d15e5: POC for counters with redis hyperloglog (authored by vsellier).
POC for counters with redis hyperloglog
vsellier added a revision to T2976: Deposit tests end-to-end are failing in icinga: Unknown Object (Differential Revision).
vsellier triaged T2988: Improve cgit lister to add last modification date of the repos as Normal priority.
vsellier updated the task description for T2967: Write journal client subcribed to origin_visit_status topics .
Jan 23 2021
Jan 23 2021
vsellier committed rDSCH3d13cda421b9: Solve uneventful/eventful with unordered messages with snapshots (authored by vsellier).
Solve uneventful/eventful with unordered messages with snapshots
vsellier committed rDSCHd528998df596: Do not consider duplicated messages as uneventful event (authored by vsellier).
Do not consider duplicated messages as uneventful event
Jan 22 2021
Jan 22 2021
vsellier closed T2978: Deploy visit-stats journal client on staging, a subtask of T2967: Write journal client subcribed to origin_visit_status topics , as Resolved.
vsellier moved T2984: Port cgit lister to the new Lister API from Backlog to in-progress on the Sprint 2021 01 board.
vsellier changed the status of T2984: Port cgit lister to the new Lister API from Open to Work in Progress.
Jan 21 2021
Jan 21 2021
vsellier committed rDSCH25d036ef4503: Do not consider duplicated messages as uneventful event (authored by vsellier).
Do not consider duplicated messages as uneventful event
vsellier committed rDSCH82b7a8a42367: Solve uneventful/eventful with unordered messages with snapshots (authored by vsellier).
Solve uneventful/eventful with unordered messages with snapshots
vsellier updated the diff for D4918: Solve uneventful/eventful with unordered messages with snapshots.
rebase
rebase
vsellier requested review of D4918: Solve uneventful/eventful with unordered messages with snapshots.
Test with "several" database upserts as it's more realistic
vsellier committed rSPSITE9a67834c5b94: Deploy the scheduler's journal client (authored by vsellier).
Deploy the scheduler's journal client
vsellier committed rDENV6687fd022604: scheduler: deploy journal client service (authored by vsellier).
scheduler: deploy journal client service
split the long entrypoint's command line
rebase
vsellier moved T2919: Decomission webapp0 node (azure) from deployed/landed/monitoring to done on the System administration board.
vsellier moved T2958: Use all the disks on esnode2 and esnode3 from in-progress to deployed/landed/monitoring on the System administration board.
vsellier moved T2905: Deploy swh-search for production from in-progress to deployed/landed/monitoring on the System administration board.
vsellier moved T2903: Test different disk configuration on esnode1 from in-progress to deployed/landed/monitoring on the System administration board.
vsellier moved T2888: Elasticsearch cluster failure during a rolling restart from in-progress to deployed/landed/monitoring on the System administration board.
vsellier moved T2976: Deposit tests end-to-end are failing in icinga from in-progress to deployed/landed/monitoring on the System administration board.
With the longer warning threshold, the monitoring is now green.
vsellier committed rSPSITE2aeb1821b9ab: deposit monitoring: Increase warning threshold (authored by vsellier).
deposit monitoring: Increase warning threshold
vsellier changed the status of T2976: Deposit tests end-to-end are failing in icinga from Open to Work in Progress.
Jan 20 2021
Jan 20 2021
Backfill launched from storage1 with this script : P927 (10 ranges in //) and finished in ~15mn
All staging worker stopped:
root@pergamon:~# sudo clush -b -w @staging-workers 'puppet agent --disable "Deploy new storage version"; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl disable $unit; done; systemctl stop swh-worker@*'
it seems it's the scheduler running that is taking time to scheduler the deposit task :
08:37:53 -> task is created
08:43:05 -> the runner is scheduling the task
08:43:24 -> the worker acknowledge the task
Jan 19 2021
Jan 19 2021