Looks good to me.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed All Stories
All Stories
All Stories
Aug 25 2021
Aug 25 2021
Regenerate private data
swh-public-ci added a comment to D6131: Remove .wasm from all distributions, and .so from source distributions.
Build is green
vlorentz updated the diff for D6131: Remove .wasm from all distributions, and .so from source distributions.
fix bdist_wheel support
Build is green
anlambert added a comment to D6131: Remove .wasm from all distributions, and .so from source distributions.
I still got an error when packaging swh-search, see below:
(swh) ✔ ~/swh/swh-environment/swh-search [arcpatch-D6131 L|…1⚑ 3] 12:13 $ python3 setup.py sdist bdist_wheel running sdist running egg_info writing swh.search.egg-info/PKG-INFO writing dependency_links to swh.search.egg-info/dependency_links.txt writing entry points to swh.search.egg-info/entry_points.txt writing requirements to swh.search.egg-info/requires.txt writing top-level names to swh.search.egg-info/top_level.txt reading manifest template 'MANIFEST.in' warning: no files found matching 'version.txt' adding license file 'LICENSE' adding license file 'AUTHORS' writing manifest file 'swh.search.egg-info/SOURCES.txt' running check creating swh.search-0.11.3.dev2+g7f5026d creating swh.search-0.11.3.dev2+g7f5026d/docs creating swh.search-0.11.3.dev2+g7f5026d/docs/_static creating swh.search-0.11.3.dev2+g7f5026d/docs/_templates creating swh.search-0.11.3.dev2+g7f5026d/es_config creating swh.search-0.11.3.dev2+g7f5026d/swh creating swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info creating swh.search-0.11.3.dev2+g7f5026d/swh/search creating swh.search-0.11.3.dev2+g7f5026d/swh/search/api creating swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language creating swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language/test creating swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language/test/corpus creating swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying files to swh.search-0.11.3.dev2+g7f5026d... copying .gitignore -> swh.search-0.11.3.dev2+g7f5026d copying .pre-commit-config.yaml -> swh.search-0.11.3.dev2+g7f5026d copying AUTHORS -> swh.search-0.11.3.dev2+g7f5026d copying CODE_OF_CONDUCT.md -> swh.search-0.11.3.dev2+g7f5026d copying CONTRIBUTORS -> swh.search-0.11.3.dev2+g7f5026d copying LICENSE -> swh.search-0.11.3.dev2+g7f5026d copying MANIFEST.in -> swh.search-0.11.3.dev2+g7f5026d copying Makefile -> swh.search-0.11.3.dev2+g7f5026d copying Makefile.local -> swh.search-0.11.3.dev2+g7f5026d copying README.md -> swh.search-0.11.3.dev2+g7f5026d copying mypy.ini -> swh.search-0.11.3.dev2+g7f5026d copying package.json -> swh.search-0.11.3.dev2+g7f5026d copying pyproject.toml -> swh.search-0.11.3.dev2+g7f5026d copying pytest.ini -> swh.search-0.11.3.dev2+g7f5026d copying requirements-swh.txt -> swh.search-0.11.3.dev2+g7f5026d copying requirements-test.txt -> swh.search-0.11.3.dev2+g7f5026d copying requirements.txt -> swh.search-0.11.3.dev2+g7f5026d copying setup.cfg -> swh.search-0.11.3.dev2+g7f5026d copying setup.py -> swh.search-0.11.3.dev2+g7f5026d copying tox.ini -> swh.search-0.11.3.dev2+g7f5026d copying yarn.lock -> swh.search-0.11.3.dev2+g7f5026d copying docs/.gitignore -> swh.search-0.11.3.dev2+g7f5026d/docs copying docs/Makefile -> swh.search-0.11.3.dev2+g7f5026d/docs copying docs/cli.rst -> swh.search-0.11.3.dev2+g7f5026d/docs copying docs/conf.py -> swh.search-0.11.3.dev2+g7f5026d/docs copying docs/index.rst -> swh.search-0.11.3.dev2+g7f5026d/docs copying docs/query-language.rst -> swh.search-0.11.3.dev2+g7f5026d/docs copying docs/_static/.placeholder -> swh.search-0.11.3.dev2+g7f5026d/docs/_static copying docs/_templates/.placeholder -> swh.search-0.11.3.dev2+g7f5026d/docs/_templates copying es_config/elasticsearch.keystore -> swh.search-0.11.3.dev2+g7f5026d/es_config copying es_config/elasticsearch.yml -> swh.search-0.11.3.dev2+g7f5026d/es_config copying es_config/jvm.options -> swh.search-0.11.3.dev2+g7f5026d/es_config copying es_config/log4j2.properties -> swh.search-0.11.3.dev2+g7f5026d/es_config copying swh/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh copying swh.search.egg-info/PKG-INFO -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh.search.egg-info/SOURCES.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh.search.egg-info/dependency_links.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh.search.egg-info/entry_points.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh.search.egg-info/not-zip-safe -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh.search.egg-info/requires.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh.search.egg-info/top_level.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info copying swh/search/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/cli.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/elasticsearch.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/in_memory.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/interface.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/journal_client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/metrics.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/py.typed -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/translator.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/utils.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search copying swh/search/api/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/api copying swh/search/api/client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/api copying swh/search/api/server.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/api copying swh/search/query_language/.gitignore -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language copying swh/search/query_language/grammar.js -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language copying swh/search/query_language/sample_query -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language copying swh/search/query_language/tokens.js -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language copying swh/search/query_language/test/corpus/combinations.txt -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language/test/corpus copying swh/search/tests/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/conftest.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_api_client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_cli.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_elasticsearch.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_in_memory.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_init.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_journal_client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_search.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_server.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests copying swh/search/tests/test_translator.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests Writing swh.search-0.11.3.dev2+g7f5026d/setup.cfg running ts_install yarn install v1.22.5 [1/4] Resolving packages... success Already up-to-date. Done in 0.06s. Copying parser files Getting path Generating Creating tar archive removing 'swh.search-0.11.3.dev2+g7f5026d' (and everything under it) running bdist_wheel running build running build_py running ts_build running ts_build_so error: [Errno 2] No such file or directory: 'swh/search/query_language/src/parser.c'
anlambert retitled D6004: misc/coverage: Revamp and improve archive coverage widget from misc/coverage: Revamp archive coverage widget (WIP) to misc/coverage: Revamp and improve archive coverage widget.
- improve test to check origins logos and search links are present in the rendered HTML page
vlorentz requested review of D6131: Remove .wasm from all distributions, and .so from source distributions.
status.io incident closed
Save code now requests rescheduled:
swh-web=> select * from save_origin_request where loading_task_status='scheduled' limit 100; ... <output loast due to the psql pager :( ...
softwareheritage-scheduler=> select * from task where id in (398244739, 398244740, 398244742, 398244744, 398244745, 398244748, 398095676, 397470401, 397470402, 397470404, 397470399);
few minutes later:
swh-web=> select * from save_origin_request where loading_task_status='scheduled' limit 100; id | request_date | visit_type | origin_url | status | loading_task_id | visit_date | loading_task_status | visit_status | user_ids ----+--------------+------------+------------+--------+-----------------+------------+---------------------+--------------+---------- (0 rows)
- all the workers are restarted
- Several save code now requests look stuck in the scheduled status, currently looking how to unblock them
D6130 landed and applied one kafka at a time
anlambert added inline comments to D6113: vault API: Rename bundle types and use SWHIDs to identify objects.
Build is green
kafka: increase the open file limit
vsellier added a revision to T3501: Too many open files error on kafka: D6130: kafka: increase the open file limit.
swh-public-ci added a comment to D6113: vault API: Rename bundle types and use SWHIDs to identify objects.
Build is green
ok roger that :).
I will increase to 524288 in the diff
The kafka servers are only running kafka and zookeeper, so the limit of open files isn't that critical. I think we can bump the limit more substantially than just x2 (maybe go directly with x8?), as I expect we'll still be adding more topics in the future.
vlorentz requested review of D6113: vault API: Rename bundle types and use SWHIDs to identify objects.
vlorentz added inline comments to D6113: vault API: Rename bundle types and use SWHIDs to identify objects.
all the loaders are restarted on worker01 and workers02, it seems the cluster is ok.
vlorentz closed D6126: vault.spec.js: Add test checking the LocalStorage is used to show the task list.
773d6c2e067fa6a494d2bd4e34317fff779affe8
7dde8550288892a23ea3128a52e4a67d3aad9802
vlorentz committed rDWAPPS773d6c2e067f: vault.spec.js: Add test checking the LocalStorage is used to show the task list (authored by vlorentz).
vault.spec.js: Add test checking the LocalStorage is used to show the task list
vlorentz committed rDWAPPS7dde85502888: vault.spec.js: Remove vaultItems from global variables (authored by vlorentz).
vault.spec.js: Remove vaultItems from global variables
rebase
vlorentz updated the diff for D6113: vault API: Rename bundle types and use SWHIDs to identify objects.
remove debug log + remove useless code from tests + fix deprecation notices
The open file limit was manually increased to stabilize the cluster:
# puppet agent --disable T3501 # diff -U3 /tmp/kafka.service kafka.service --- /tmp/kafka.service 2021-08-25 07:32:28.068928972 +0000 +++ kafka.service 2021-08-25 07:32:31.384955246 +0000 @@ -15,7 +15,7 @@ Environment='LOG_DIR=/var/log/kafka' Type=simple ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties -LimitNOFILE=65536 +LimitNOFILE=131072
- Incident created on status.io
- Loader disabled:
root@pergamon:~# clush -b -w @swh-workers 'puppet agent --disable "Kafka incident T3501"; systemctl stop cron; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@loader_*; do systemctl disable $unit; done; systemctl stop "swh-worker@loader_*"'
vsellier changed the status of T3501: Too many open files error on kafka from Open to Work in Progress.
Aug 24 2021
Aug 24 2021
DanSeraf committed rDTSCN979d7c803a14: retrieve additional information about software artifacts (authored by DanSeraf).
retrieve additional information about software artifacts
swh-public-ci added a comment to D6114: swh-scanner: retrieve additional information about software artifacts.
Build is green
DanSeraf updated the diff for D6114: swh-scanner: retrieve additional information about software artifacts.
fix docstring
Build is green
Fix hypothesis warning
Build is green
Looks good to me.
Ensure lru caches are cleared before each test.
Fix $command must be an array of strings and not a plain string.
Vagrantfile: Add belvedere node
LGTM, thanks! But please note that remaining "-a/--add" reference in the new docstring, which should be changed to "-e/..." for consistency. Please fix that before landing this change.
Build is green
vsellier committed rSENVeb3a616b885b: vagrant: update debian image to debian 10.10 (authored by vsellier).
vagrant: update debian image to debian 10.10
Great, thanks !
Harbormaster failed remote builds in B23129: Diff 22172 for D6004: misc/coverage: Revamp and improve archive coverage widget!
Build has FAILED
Update tests
Rebase and add test for coverage view.
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTOdac2450db794: Merge tag 'debian/0.36.0-1_swh1' into debian/buster-swh (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Merge tag 'debian/0.36.0-1_swh1' into debian/buster-swh
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO89d28d5ef621: Updated backport on buster-swh from debian/0.36.0-1_swh1 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on buster-swh from debian/0.36.0-1_swh1 (unstable-swh)
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO0ed3003b5ead: pristine-tar data for swh-storage_0.36.0.orig.tar.gz (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
pristine-tar data for swh-storage_0.36.0.orig.tar.gz
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTOffe636f347c6: Updated debian changelog for version 0.36.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated debian changelog for version 0.36.0
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO6924a7ed049b: Update upstream source from tag 'debian/upstream/0.36.0' (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Update upstream source from tag 'debian/upstream/0.36.0'
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO3a224a92dfb8: New upstream version 0.36.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
New upstream version 0.36.0
ardumont moved T3495: The weekly report bot is down from in-progress to code-review/await-feedback/pause on the System administration board.
ardumont moved T3497: Allow systemd service status monitoring from Backlog to in-progress on the System administration board.
Add cvs as supported revision_type
Build is green
This looked familiar, there is one diff D5991 doing this already.
This seems stuck though.
vlorentz committed rDSTO8f1cdf65a105: Add test for origin_visit_get_latest in presence of mismatched id and date… (authored by vlorentz).
Add test for origin_visit_get_latest in presence of mismatched id and date…
vlorentz committed rDSTOcf880db30bb5: cassandra: Bump next_visit_id when origin_visit_add is called by a replayer (authored by vlorentz).
cassandra: Bump next_visit_id when origin_visit_add is called by a replayer
vlorentz committed rDSTO54b5abfb2626: cassandra: Make content_missing query in batches (authored by vlorentz).
cassandra: Make content_missing query in batches
olasd@journal0:~$ /opt/kafka/bin/kafka-configs.sh --bootstrap-server journal0.internal.staging.swh.network:9092 --alter --add-config 'cleanup.policy=[compact,delete],retention.ms=86400000' --entity-type=topics --entity-name swh.journal.objects.extid
The backfill was stopped as the performance was (much) lower than expected (worked around with D6127).
swh-public-ci added a comment to D6121: Add test for origin_visit_get_latest in presence of mismatched id and date orders.
Build is green
swh-public-ci added a comment to D6120: cassandra: Bump next_visit_id when origin_visit_add is called by a replayer.
Build is green
Harbormaster failed remote builds in B23119: Diff 22162 for D6118: cassandra: Make content_missing query in batches!
Build has FAILED
Some live data from a git loader with a batch size of 1000 for each object types (with D6118 applied):
"object type";"input count";"missing_id duration (s)";"_missing_id count","_add duration(s)" content;1000;0.4928;999;35.3384 content;1000;0.4095;1000;34.1440 content;1000;0.4374;998;35.6249 content;492;0.2960;488;16.7028 directory;1000;0.3978;999;71.2518 directory;1000;0.4484;1000;39.6845 directory;1000;0.4356;1000;54.0077 directory;1000;0.3833;1000;36.1437 directory;1000;0.4319;1000;30.5690 directory;402;0.1718;402;19.2335 revision;1000;0.8671;1000;10.3417 revision;575;0.4639;575;4.0819
vlorentz updated the diff for D6121: Add test for origin_visit_get_latest in presence of mismatched id and date orders.
rebase
vlorentz updated the diff for D6120: cassandra: Bump next_visit_id when origin_visit_add is called by a replayer.
rebase
rebase
The performance are ok now for the read part with a batch size of 1000 for content, directory and revision.
vsellier committed rDSTO7113198fd65e: backfill: add extra where clause to use the right index for extid requests (authored by vsellier).
backfill: add extra where clause to use the right index for extid requests
vsellier changed the status of T3476: One of the system disks of beaubourg is out of order, a subtask of T3444: 26/07/2021: Unstuck infrastructure outage then post-mortem, from Open to Work in Progress.
vsellier changed the status of T3476: One of the system disks of beaubourg is out of order from Open to Work in Progress.
An alert was sent by email the 2021-05-22 at 05:30 AM so the monitoring has well detected the issue ;) :
This message was generated by the smartd daemon running on:
vsellier closed T3499: Move firewall storage to local hypervisor storage, a subtask of T3444: 26/07/2021: Unstuck infrastructure outage then post-mortem, as Resolved.
on hypervisor3 and branly
- A new lvm volume was created and mounted on /var/lib/vz (40G on hypervisor3 / 100G on branly)
- local storage type was activated on proxmox via the ui (Datacenter / storage / local, check enable)
- pushkin and glytotek disks moved via to ui to the local storage (<vm> / hardware click on the disk / move disk button / target storage 'local')