Build is green
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed All Stories
All Stories
All Stories
Aug 19 2021
Aug 19 2021
vlorentz removed a parent task for T2513: Copy metadata on revisions to the extrinsic metadata storage: T2202: Collect extrinsic metadata.
Aug 18 2021
Aug 18 2021
DanSeraf requested review of D6114: swh-scanner: retrieve additional information about software artifacts.
swh-public-ci added a comment to D6113: vault API: Rename bundle types and use SWHIDs to identify objects.
vlorentz published D6113: vault API: Rename bundle types and use SWHIDs to identify objects for review.
vlorentz requested review of D6112: Rename bundle types and use SWHIDs everywhere instead of raw sha1_git.
olasd committed rCJSWH253b42a0232b: automatic-backport: use simpler logic to detect whether the branch needs… (authored by olasd).
automatic-backport: use simpler logic to detect whether the branch needs…
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rPTS08a56c888086: Updated backport on buster-swh from debian/0.19.0-1+swh1 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on buster-swh from debian/0.19.0-1+swh1 (unstable-swh)
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rPTS3c6730eb159d: Merge tag 'debian/0.19.0-1+swh1' into debian/buster-swh (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Merge tag 'debian/0.19.0-1+swh1' into debian/buster-swh
olasd committed rPTS8493b1527058: Merge tag 'debian/0.19.0-1+swh1_bpo10+1' into debian/buster-swh (authored by olasd).
Merge tag 'debian/0.19.0-1+swh1_bpo10+1' into debian/buster-swh
@sdruskat Good news about the GitHub support.
moranegg triaged T3489: Implement iframe view for content and directory elements as Normal priority.
vsellier committed rPTSc2e0c5766b2c: pristine-tar data for tree-sitter_0.19.0.orig.tar.gz (authored by vsellier).
pristine-tar data for tree-sitter_0.19.0.orig.tar.gz
vsellier committed rPTS7702e942c810: initialize the backport build configuration (authored by vsellier).
initialize the backport build configuration
vsellier committed rPTS08f03d9d6057: Initial packaging for python3-tree-sitter (authored by vsellier).
Initial packaging for python3-tree-sitter
New upstream version 0.19.0
vsellier committed rCJSWH9a81390eda36: Declare debian package build for tree-sitter (authored by vsellier).
Declare debian package build for tree-sitter
In D6083#157897, @vlorentz wrote:Sounds good, thanks! Should I close this diff?
Sounds good, thanks! Should I close this diff?
I need help to fix these failures...
The back fill was relaunched using the script pasted in P1124
vlorentz requested review of D6111: Add support for releases pointing to other releases or contents..
anlambert committed rDWAPPS885be5dde5b5: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
browse/utils: Remove dead code
anlambert committed rDWAPPSc9d89d93b7df: browse/utils: Fix edge case in prepare_content_for_display (authored by anlambert).
browse/utils: Fix edge case in prepare_content_for_display
Build is green
rebase
aeviso committed rDPROVe9206efef3a1: Fix origin type when querying ProstgreSQL backend (authored by aeviso).
Fix origin type when querying ProstgreSQL backend
The backfill process was interrupted by a restart of kafka on kafka1 (!).
2021-08-18T09:20:05 ERROR swh.journal.writer.kafka FAIL [swh.storage.journal_writer.getty#producer-1] [thrd:kafka1.internal.softwareheritage.org:9092/bootstrap]: kafka1.internal.softwareheritage.org:9092/1: Connect to ipv4#192.168.100.201:9092 failed: Connectio
n refused (after 0ms in state CONNECT, 12 identical error(s) suppressed)
2021-08-18T09:20:05 INFO swh.journal.writer.kafka Received non-fatal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="kafka1.internal.softwareheritage.org:9092/1: Connect to ipv4#192.168.100.201:9092 failed: Connection refused (after 0ms in state CONNECT,
12 identical error(s) suppressed)"}
2021-08-18T09:20:05 ERROR swh.journal.writer.kafka FAIL [swh.storage.journal_writer.getty#producer-1] [thrd:kafka1.internal.softwareheritage.org:9092/bootstrap]: kafka1.internal.softwareheritage.org:9092/1: Connect to ipv4#192.168.100.201:9092 failed: Connectio
n refused (after 0ms in state CONNECT, 5 identical error(s) suppressed)
2021-08-18T09:20:05 INFO swh.journal.writer.kafka Received non-fatal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="kafka1.internal.softwareheritage.org:9092/1: Connect to ipv4#192.168.100.201:9092 failed: Connection refused (after 0ms in state CONNECT,
5 identical error(s) suppressed)"}
2021-08-18T09:20:07 INFO swh.journal.writer.kafka PARTCNT [swh.storage.journal_writer.getty#producer-1] [thrd:main]: Topic swh.journal.objects.extid partition count changed from 256 to 128
2021-08-18T09:20:07 WARNING swh.journal.writer.kafka BROKER [swh.storage.journal_writer.getty#producer-1] [thrd:main]: swh.journal.objects.extid [128] is unknown (partition_cnt 128): ignoring leader (-1) update
2021-08-18T09:20:07 WARNING swh.journal.writer.kafka BROKER [swh.storage.journal_writer.getty#producer-1] [thrd:main]: swh.journal.objects.extid [130] is unknown (partition_cnt 128): ignoring leader (-1) update
2021-08-18T09:20:07 WARNING swh.journal.writer.kafka BROKER [swh.storage.journal_writer.getty#producer-1] [thrd:main]: swh.journal.objects.extid [132] is unknown (partition_cnt 128): ignoring leader (-1) update
...
2021-08-18T09:20:07 WARNING swh.journal.writer.kafka BROKER [swh.storage.journal_writer.getty#producer-1] [thrd:main]: swh.journal.objects.extid [253] is unknown (partition_cnt 128): ignoring leader (-1) update
Traceback (most recent call last):
File "/usr/bin/swh", line 11, in <module>
load_entry_point('swh.core==0.13.0', 'console_scripts', 'swh')()
File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
return swh(auto_envvar_prefix="SWH")
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/storage/cli.py", line 145, in backfill
dry_run=dry_run,
File "/usr/lib/python3/dist-packages/swh/storage/backfill.py", line 637, in run
writer.write_additions(object_type, objects)
File "/usr/lib/python3/dist-packages/swh/storage/writer.py", line 67, in write_additions
self.journal.write_additions(object_type, values)
File "/usr/lib/python3/dist-packages/swh/journal/writer/kafka.py", line 249, in write_additions
self.flush()
File "/usr/lib/python3/dist-packages/swh/journal/writer/kafka.py", line 215, in flush
raise self.delivery_error("Failed deliveries after flush()")
swh.journal.writer.kafka.KafkaDeliveryError: KafkaDeliveryError(Failed deliveries after flush(), [extid 344a2795951fabbf1f898b1a5fc54c4b57293cd5 (Local: Unknown partition)])
2021-08-18T09:20:07 INFO swh.journal.writer.kafka PARTCNT [swh.storage.journal_writer.getty#producer-1] [thrd:main]: Topic swh.journal.objects.extid partition count changed from 256 to 128
...
swh.journal.writer.kafka.KafkaDeliveryError: KafkaDeliveryError(flush() exceeded timeout (120s), [extid 6e1a1317c35b971ef88e052a8b1b78d57bc71a2e (No delivery before flush() timeout), extid a5052a247a0af7926b8e33224ecf7ab12c148eb5 (No delivery before flush() timeout), extid 4f5ed974e8691d340724782b01bc9bb63781176f (No delivery before flush() timeout)])
Traceback (most recent call last):
File "/usr/bin/swh", line 11, in <module>
load_entry_point('swh.core==0.13.0', 'console_scripts', 'swh')()
File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
return swh(auto_envvar_prefix="SWH")
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/storage/cli.py", line 145, in backfill
dry_run=dry_run,
File "/usr/lib/python3/dist-packages/swh/storage/backfill.py", line 637, in run
writer.write_additions(object_type, objects)
File "/usr/lib/python3/dist-packages/swh/storage/writer.py", line 67, in write_additions
self.journal.write_additions(object_type, values)
File "/usr/lib/python3/dist-packages/swh/journal/writer/kafka.py", line 249, in write_additions
self.flush()
File "/usr/lib/python3/dist-packages/swh/journal/writer/kafka.py", line 212, in flush
"flush() exceeded timeout (%ss)" % self.flush_timeout,
swh.journal.writer.kafka.KafkaDeliveryError: KafkaDeliveryError(flush() exceeded timeout (120s), [extid b3f5a81891b2be4bf487ff1f8418110fd87d1042 (No delivery before flush() timeout), extid 5c165ffa4bb15bde37d0652cee9e19c5f0cda09b (No delivery before flush() timeout)])The backfill will be restarted from the last positions (need to figure how to do that without taking too much time)
KShivendu committed rDSEAdb47029f6c91: grammar.js: Improve grammar and export tokens (authored by KShivendu).
grammar.js: Improve grammar and export tokens
thanks
vsellier committed rDSNIPf4c8abe97ccc: grid5000/cassandra: add a script to refresh the besteffort node list (authored by vsellier).
grid5000/cassandra: add a script to refresh the besteffort node list
vsellier committed rDSNIPdb9574d46037: grid5000/cassadra: declare the best effort nodes only when they are fully… (authored by vsellier).
grid5000/cassadra: declare the best effort nodes only when they are fully…
vsellier committed rDSNIP19515afeb074: grid5000/cassadra: count best_effort jobs in waiting/launching state (authored by vsellier).
grid5000/cassadra: count best_effort jobs in waiting/launching state
Fix module requirements
aeviso committed rDPROV68e1c8ffddf6: Fix bugs in ProvenanceStoragePostgreSql class (authored by aeviso).
Fix bugs in ProvenanceStoragePostgreSql class
aeviso committed rDPROV2086effea842: Rename PostgreSQL backend and code styling (authored by aeviso).
Rename PostgreSQL backend and code styling
vsellier changed the status of T3487: Installation of the new provenance server from Open to Work in Progress.
Build is green
- tokens.js: Use concat in fields
cli.py: Add rpc-serve command
Build is green
- Changes suggested by vlorentz
- Polish the code (add comments, improve formatting)
- Squash
Aug 17 2021
Aug 17 2021
Build is green
- Improve commit message
Build is green
- Fix ql_rel_paths in translator.py (Development path was wrong and was throwing error when used with the rpc server)
- Squash
One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends > is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, and pybuild internally sets the http_proxy and https_proxy environment variables (to 127.0.0.1:9) to prevent this from happening.
The pypi build is still working well with the 2 last diff.
Now there is a new error during the debian ones:
dh: warning: Compatibility levels before 10 are deprecated (level 9 in use)
dh_auto_clean -O--buildsystem=pybuild
dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use)
I: pybuild base:232: python3.9 setup.py clean
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101bcd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101beb0>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b850>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b730>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b610>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
ERROR: Could not find a version that satisfies the requirement tree-sitter==0.19.0
ERROR: No matching distribution found for tree-sitter==0.19.0
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/setuptools/installer.py", line 75, in fetch_build_egg
subprocess.check_call(cmd)
File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpdrbws3hq', '--quiet', 'tree-sitter==0.19.0']' returned non-zero exit status 1.current status:
borisbaldassari committed rDDOC2b51cac2fd63: T3339 Fix various formatting typos in getting started with SWH API. (authored by borisbaldassari).
T3339 Fix various formatting typos in getting started with SWH API.
borisbaldassari updated the diff for D6106: T3339 Fix various formatting typos in getting started with SWH API..
Updating D6106: T3339 Fix various formatting typos in getting started with SWH API.
vsellier committed rDSNIPa31433b334ad: grid5000/cassandra: replay extid topic (authored by vsellier).
grid5000/cassandra: replay extid topic
vsellier committed rDSNIP35813e5a8fcc: grid5000/cassandra: adapt the number of replayers (authored by vsellier).
grid5000/cassandra: adapt the number of replayers
Thanks for this, and for the screenshots, they look gorgeous!
vsellier moved T3485: extid topic is misconfigured in staging and production from Backlog to in-progress on the System administration board.
Build is green
Update:
- rebase
- implement search links for all origin types
- polish widget rendering: add scrollbar for origin type with lot of instances
Production backfill in progress:
root@getty:~/T3485# ./backfill.sh | tee output.log Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 000000 --end-object 080000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 080001 --end-object 100000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 100001 --end-object 180000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 180001 --end-object 200000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 200001 --end-object 280000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 280001 --end-object 300000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 300001 --end-object 380000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 380001 --end-object 400000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 400001 --end-object 480000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 480001 --end-object 500000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 500001 --end-object 580000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 580001 --end-object 600000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 600001 --end-object 680000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 680001 --end-object 700000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 700001 --end-object 780000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 780001 --end-object 800000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 800001 --end-object 880000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 880001 --end-object 900000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 900001 --end-object 980000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object 980001 --end-object a00000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object a00001 --end-object a80000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object a80001 --end-object b00000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object b00001 --end-object b80000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object b80001 --end-object c00000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object c00001 --end-object c80000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object c80001 --end-object d00000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object d00001 --end-object d80000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object d80001 --end-object e00000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object e00001 --end-object e80000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object e80001 --end-object f00000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object f00001 --end-object f80000 Starting swh --log-config /etc/softwareheritage/journal/backfill_logger.yml storage backfill extid --start-object f80001
Production
Unfortunately, the replication factor can't be changed directly, the partition assignment must be reconfigured to change it.
It was done before increasing the number of partition to limit the number of move to perform
Looks good to me, thanks !
vlorentz committed rDVAU079a57b4d3cf: git_bare: Add support for contents pointed by snapshots (authored by vlorentz).
git_bare: Add support for contents pointed by snapshots
vlorentz committed rDVAUc2692385873a: git_bare: Statically check for exhaustiveness of enum matching (authored by vlorentz).
git_bare: Statically check for exhaustiveness of enum matching
Nice !
borisbaldassari added a comment to D6106: T3339 Fix various formatting typos in getting started with SWH API..
Hiho good people, here is a quick fix for some minor typos, mostly linked to the output rendering of pre's.
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rPMK1c862dcbf51a: Updated backport on bullseye-swh from debian/2.1.0-2__swh2 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on bullseye-swh from debian/2.1.0-2__swh2 (unstable-swh)