Done scheduling:
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Feb 18 2021
Feb 18 2021
vlorentz closed T3058: Metadata search is failing with "failed to parse date field", a subtask of T2590: Finish the indexer -> swh-search pipeline, as Resolved.
vsellier moved T2944: Deploy swh-search v0.4.1 from deployed/landed/monitoring to done on the System administration board.
Feb 17 2021
Feb 17 2021
anlambert closed T3047: Enable to search in origin metadata with swh-search in webapp, a subtask of T2590: Finish the indexer -> swh-search pipeline, as Resolved.
Feb 15 2021
Feb 15 2021
vsellier renamed T3040: [production] Enable swh-search's journal-client for indexed objects from Enable swh-search's journal-client for indexed objects to [production] Enable swh-search's journal-client for indexed objects.
Feb 12 2021
Feb 12 2021
ardumont closed T3037: Reschedule origin-intrinsic-metadata tasks for all origins, a subtask of T2590: Finish the indexer -> swh-search pipeline, as Resolved.
Feb 11 2021
Feb 11 2021
vsellier placed T3040: [production] Enable swh-search's journal-client for indexed objects up for grabs.
vsellier added a comment to T3040: [production] Enable swh-search's journal-client for indexed objects.
T3041 needs to be done before this one (for the production environment)
vlorentz triaged T3040: [production] Enable swh-search's journal-client for indexed objects as Normal priority.
Running:
swhscheduler@saatchi:~$ /usr/bin/swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task schedule_ origins --storage-url http://saam.internal.softwareheritage.org:5002 --batch-size 20 index-origin-metadata | tee /tmp/schedule-origins.txt
vlorentz changed the status of T2590: Finish the indexer -> swh-search pipeline from Open to Work in Progress.
@ardumont no, OriginMetadataIndexer lacks a filter step.
@ardumont no, OriginMetadataIndexer lacks a filter step.
ardumont closed T2780: Enable the journal-writer for the swh-idx-storage in production, a subtask of T2590: Finish the indexer -> swh-search pipeline, as Resolved.
ardumont closed T2780: Enable the journal-writer for the swh-idx-storage in production, a subtask of T3037: Reschedule origin-intrinsic-metadata tasks for all origins, as Resolved.
ardumont changed the status of T3037: Reschedule origin-intrinsic-metadata tasks for all origins from Open to Work in Progress.
ardumont changed the status of T3037: Reschedule origin-intrinsic-metadata tasks for all origins, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Open to Work in Progress.
Although, now i'm wondering something.
Is that enough to write what's not in the topics?
Ah no! I misused the cli, with the right flags:
This needs a storage access so edit a dedicated configuration file.
That's it! [1]
Feb 10 2021
Feb 10 2021
try swh scheduler task schedule_origins
That suggested cli does not show up but i've only took a quick glance ¯\_(ツ)_/¯:
Rather than write code to read from the database to kafka (like we did with swh-storage), this can be done simply by re-indexing all the origins, using swh scheduler schedule_origins
vlorentz triaged T3037: Reschedule origin-intrinsic-metadata tasks for all origins as Normal priority.
Indexer related topics status can be seen in in the indexer ingestion status board [1]
We'll prepare the topics with the following first and we'll improve later if need be:
I did too much here. Finish the pipeline swh-indexer -> swh-search on staging (so that's good nonetheless)
Note that the "docs.count" grew though (from 496619 to 786803) and the reason are
unclear.The same index is used to store the metadata out of the indexer with the same origin url
as key [1] and we are computing index metadata on origins already seen (thus already present
in the index afaiui). So I would have expect the docs.count stay roughly (or even
exactly?) the same as before?
swh-search-journal-client@indexed kept up with its topic:
swh.search.journal_client.indexed swh.journal.indexed.origin_intrinsic_metadata 0 13653216 13653216 0 rdkafka-7c45245c-814f-47f1-ba67-041e4f426373 /192.168.130.90 rdkafka
Feb 9 2021
Feb 9 2021
tl; dr deployed on staging and it seems ok.
I just mention some on T2912#58067 but it's unclear whether that's actually true of me misremembering things.
ardumont changed the status of T2780: Enable the journal-writer for the swh-idx-storage in production, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Open to Work in Progress.
ardumont changed the status of T2780: Enable the journal-writer for the swh-idx-storage in production from Open to Work in Progress.
vlorentz closed T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit, a subtask of T2905: Deploy swh-search for production, as Resolved.
Feb 5 2021
Feb 5 2021
ardumont moved T2780: Enable the journal-writer for the swh-idx-storage in production from Backlog to Weekly backlog on the System administration board.
Feb 4 2021
Feb 4 2021
olasd added a comment to T3012: Check all objects in the production storage/journal have a correct hash.
This is a duplicate of T75, the history of which would probably be useful to take into account (I suspect it can be closed).
Feb 3 2021
Feb 3 2021
Is there some remaining blocker on this?
(If not i'll attend to it next week)
Feb 2 2021
Feb 2 2021
seirl triaged T3021: Investigate why reading the journal of the content table takes so long as Normal priority.
Feb 1 2021
Feb 1 2021
vlorentz triaged T3012: Check all objects in the production storage/journal have a correct hash as Normal priority.
vsellier closed T2944: Deploy swh-search v0.4.1, a subtask of T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit, as Resolved.
The backfill is done.
Jan 29 2021
Jan 29 2021
The journal_client has almost ingested the topics[1] it listens. It took some more time because a backfill of the origin_visit_status was launched for T2993.
It should be done by the end of the day.
vsellier moved T2905: Deploy swh-search for production from deployed/landed/monitoring to done on the System administration board.
Jan 27 2021
Jan 27 2021
Let's consider it as done.
vsellier moved T2944: Deploy swh-search v0.4.1 from in-progress to deployed/landed/monitoring on the System administration board.
To decrease the time to recover the lag, several journal client were launched in // with :
/usr/bin/swh search --config-file /etc/softwareheritage/search/journal_client_objects.yml journal-client objects
Jan 26 2021
Jan 26 2021
Back on this, the plan is now to make swh-journal not depend on the actual model definition, which is currently mostly due to the presence of the journal_data.py in swh-journal. So the plan is to move this file in swh-model so it's kept up to date with swh-model, even if it's mostly used for testing other packages (like swh-journal).
Upgrading the index configuration to speedup the indexation :
% cat >/tmp/config.json <<EOF
{
"index" : {
"translog.sync_interval" : "60s",
"translog.durability": "async",
"refresh_interval": "60s"
}
}
EOF
% export ES_SERVER=192.168.100.81:9200
% export INDEX=origin
% curl -s -H "Content-Type: application/json" -XPUT http://${ES_SERVER}/${INDEX}/_settings -d @/tmp/config.json
{"acknowledged":true}%Production
- puppet disabled
- Services stopped :
root@search1:~# systemctl stop swh-search-journal-client@objects.service root@search1:~# systemctl stop gunicorn-swh-search
- Index deleted and recreated
% export ES_SERVER=search-esnode1.internal.softwareheritage.org:9200
% curl -s http://$ES_SERVER/_cat/indices\?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open origin Mq8dnlpuRXO4yYoC6CTuQw 90 1 151716299 38861934 260.8gb 131gb
% curl -XDELETE http://$ES_SERVER/origin
{"acknowledged":true}%
% swh search --config-file /etc/softwareheritage/search/server.yml initialize
INFO:elasticsearch:PUT http://search-esnode1.internal.softwareheritage.org:9200/origin [status:200 request:2.216s]
INFO:elasticsearch:PUT http://search-esnode3.internal.softwareheritage.org:9200/origin/_mapping [status:200 request:0.151s]
Done.
% curl -s http://$ES_SERVER/_cat/indices\?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open origin yFaqPPCnRFCnc5AA6Ah8lw 90 1 0 0 36.5kb 18.2kb- journal client's consumer group delete:
% export SERVER=kafka1.internal.softwareheritage.org:9092
% ./kafka-consumer-groups.sh --bootstrap-server ${SERVER} --delete --group swh.search.journal_client
Deletion of requested consumer groups ('swh.search.journal_client') was successful.- journal client restarted
- puppet enabled
The filter on visited origins is working correctly on staging. The has_visit flag looks good.
For example for the https://www.npmjs.com/package/@ehmicky/dev-tasks origin
{
"_index" : "origin",
"_type" : "_doc",
"_id" : "019bd314416108304165e82dd92e00bc9ea85a53",
"_score" : 60.56421,
"_source" : {
"url" : "https://www.npmjs.com/package/@ehmicky/dev-tasks",
"sha1" : "019bd314416108304165e82dd92e00bc9ea85a53"
},
"sort" : [
60.56421,
"019bd314416108304165e82dd92e00bc9ea85a53"
]
}swh=> select * from origin join origin_visit_status on id=origin where id=469380; id | url | origin | visit | date | status | metadata | snapshot | type --------+--------------------------------------------------+--------+-------+-------------------------------+---------+----------+--------------------------------------------+------ 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 | 1 | 2021-01-25 13:30:47.221937+00 | created | | | npm 469380 | https://www.npmjs.com/package/@ehmicky/dev-tasks | 469380 | 1 | 2021-01-25 13:41:59.435579+00 | partial | | \xe3f24413d81fd3e9c309686fcfb6c8f5eb549acf | npm
Jan 25 2021
Jan 25 2021
Staging
We are proceeding to a complete index rebuilding
Regarding the index rebuilding process, using a naive approach with aliases with the old and the new index[1] returns duplicated results when the search is done.
Using an alias with only the old index, rebuilding a new index and switching the alias to the new index[2] can be a first approach with the default the old index will not be updated until the alias is switched to the new index.
It also requires the swh-search code is able to use different names for the read and write operations.
vsellier changed the status of T2944: Deploy swh-search v0.4.1, a subtask of T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit, from Open to Work in Progress.
vsellier renamed T2944: Deploy swh-search v0.4.1 from Deploy swh-search v0.4.1 in staging to Deploy swh-search v0.4.1.
Jan 21 2021
Jan 21 2021
vsellier moved T2905: Deploy swh-search for production from in-progress to deployed/landed/monitoring on the System administration board.
Jan 13 2021
Jan 13 2021
vsellier closed T2905: Deploy swh-search for production, a subtask of T2904: Create a new production webapp using the frozen index on the staging ES, as Resolved.
I close this issue as there is not more action to perform at the moment.
Diagnosis and eventual fixes will be followed on dedicated issues
Jan 11 2021
Jan 11 2021
ardumont renamed T2944: Deploy swh-search v0.4.1 from Deploy T2936 in staging to Deploy swh-search v0.4.1 in staging.
ardumont moved T2944: Deploy swh-search v0.4.1 from Backlog to Weekly backlog on the System administration board.
Jan 7 2021
Jan 7 2021
vsellier added a comment to T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit.
version v0.4.1 created with the last commit (rDSEA47db624364d4e781f8fa157b2d72d0eb9929b7a0)
vlorentz renamed T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit from Update the swh-search journal client to use visit statuses to Update the swh-search journal client to only set "has_visit" on "full" status of the visit.
Oh right, they were wrongfully set to True. I guess we can write a small script to set them all to False before we re-consume stasuses
how doing it without killing all the search
It depends of what will be implemented in T2936, but a new reindex will probably have to be done to fix the search. It will be the opportunity to think on how doing it without killing all the search
vlorentz triaged T2936: Update the swh-search journal client to only set "has_visit" on "full" status of the visit as Normal priority.
Yes indeed. swh-search was written before we had origin visit statuses, and I forgot to update it.
@vlorentz I was checking some differences between swh-search and the current search. does the journal client has to listen the origin_visit topic? It seems that `origin_visit_status should be enough to match the behavior of the search in the webapp.
Jan 6 2021
Jan 6 2021
webapp1 is now plugged on the real live production index
Let monitor the behavior with real searches.
First constatation, the search retrieves all the documents and is not as progressive as the random search script.
The response times are longer than expected:
Jan 06 09:59:46 search1 python3[813]: 2021-01-06 09:59:46 [813] elasticsearch:INFO GET http://search-esnode1.internal.softwareheritage.org:9200/origin/_search?size=100 [status:200 request:3.399s] Jan 06 10:06:18 search1 python3[848]: 2021-01-06 10:06:18 [848] elasticsearch:INFO GET http://search-esnode1.internal.softwareheritage.org:9200/origin/_search?size=100 [status:200 request:7.422s] Jan 06 10:06:21 search1 python3[813]: 2021-01-06 10:06:21 [813] elasticsearch:INFO GET http://search-esnode3.internal.softwareheritage.org:9200/origin/_search?size=100 [status:200 request:5.077s] Jan 06 10:07:32 search1 python3[813]: 2021-01-06 10:07:32 [813] elasticsearch:INFO GET http://search-esnode2.internal.softwareheritage.org:9200/origin/_search?size=100 [status:200 request:4.819s] Jan 06 10:08:06 search1 python3[813]: 2021-01-06 10:08:06 [813] elasticsearch:INFO GET http://search-esnode1.internal.softwareheritage.org:9200/origin/_search?size=100 [status:200 request:2.700s] Jan 06 10:08:15 search1 python3[813]: 2021-01-06 10:08:15 [813] elasticsearch:INFO GET http://search-esnode3.internal.softwareheritage.org:9200/origin/_search?size=100 [status:200 request:2.414s]
the performances looks acceptable as it for a small number of parallel searches (~10), let's try now with real searches, it will also help to adapt the cluster configuration and validate the behavior
Jan 5 2021
Jan 5 2021
vsellier moved T2905: Deploy swh-search for production from Backlog to in-progress on the System administration board.
In the new configuration, after a few time without search, the first ones are taking some time before stabilizing to the old values :
❯ ./random_search.sh 12:36:37
the index configuration was reset to its default :
cat >/tmp/config.json <<EOF
{
"index" : {
"translog.sync_interval" : null,
"translog.durability": null,
"refresh_interval": null
}
}
EOF❯ curl -s http://192.168.100.81:9200/origin/_settings\?pretty
{
"origin" : {
"settings" : {
"index" : {
"refresh_interval" : "60s",
"number_of_shards" : "90",
"translog" : {
"sync_interval" : "60s",
"durability" : "async"
},
"provided_name" : "origin",
"creation_date" : "1608761881782",
"number_of_replicas" : "1",
"uuid" : "Mq8dnlpuRXO4yYoC6CTuQw",
"version" : {
"created" : "7090399"
}
}
}
}
}
❯ curl -s -H "Content-Type: application/json" -XPUT http://192.168.100.81:9200/origin/_settings\?pretty -d @/tmp/config.json
{
"acknowledged" : true
}
❯ curl -s http://192.168.100.81:9200/origin/_settings\?pretty
{
"origin" : {
"settings" : {
"index" : {
"creation_date" : "1608761881782",
"number_of_shards" : "90",
"number_of_replicas" : "1",
"uuid" : "Mq8dnlpuRXO4yYoC6CTuQw",
"version" : {
"created" : "7090399"
},
"provided_name" : "origin"
}
}
}
}A *simple* search doesn't looked impacted (it's not a real benchmark):
❯ ./random_search.sh