Page MenuHomeSoftware Heritage
Feed Advanced Search

Nov 12 2019

zack updated the diff for D2244: REST server: set content-type to text or ndjson where appropriate.

app.py: inline MIME types, they are single use anyway

Nov 12 2019, 8:22 AM

Nov 11 2019

zack added a revision to T2077: add random walk endpoint with limited retries: D2249: swh-graph: add random walk endpoint.
Nov 11 2019, 7:05 PM · Compressed graph service
zack created D2249: swh-graph: add random walk endpoint.
Nov 11 2019, 7:05 PM
zack committed rDGRPH2ad5d55ec4e0: CLI: add "swh graph map lookup" to lookup values in binary maps (authored by zack).
CLI: add "swh graph map lookup" to lookup values in binary maps
Nov 11 2019, 4:55 PM
zack added a comment to T2055: swh-graph CI hangs badly when py4j doesn't find needed files.

AFAICT this is a more general problem, the Java backend can hang forever in case of unexpected situations (uncaught exceptions? I really don't know…), which will make it not respond to any incoming request with no visible output.
We should make this visible and debuggable.

Nov 11 2019, 1:45 PM · Continuous Integration, Compressed graph service
zack committed rDGRPH0b0e84ad0b10: backend.py: log which JAR is being used and warn if multiple ones exist (authored by zack).
backend.py: log which JAR is being used and warn if multiple ones exist
Nov 11 2019, 1:37 PM
zack changed the status of T2077: add random walk endpoint with limited retries from Open to Work in Progress.

initial skeleton (not yet working) in rDGRPHbc368c1775e6, branch feature/random-walk

Nov 11 2019, 12:45 PM · Compressed graph service
zack triaged T2077: add random walk endpoint with limited retries as Normal priority.
Nov 11 2019, 12:42 PM · Compressed graph service
zack renamed T1969: graph: reduce RAM usage for /walk from Reduce RAM usage for graph backtracking to reduce RAM usage for /walk.
Nov 11 2019, 12:39 PM · Compressed graph service

Nov 9 2019

zack closed T2056: fix swh-graph sphinx table of content as Invalid.

This was actually a false alarm. Due to the lack of --separate in the build toolchain for the entire docs.s.o, submodules (as opposed to sub-*packages*) of swh.graph were not visible in the TOC and only visible by scrolling down the page. I've fixed this with 6547df80508fa8d467475a8ca8db307ceb2f9972 in swh-docs.

Nov 9 2019, 6:28 PM · Documentation, Compressed graph service
zack committed rDDOC6547df80508f: sphinx: unify apidoc flags when building individual modules and full doc (authored by zack).
sphinx: unify apidoc flags when building individual modules and full doc
Nov 9 2019, 6:26 PM
zack committed rDGRPH6290eda4721b: doc: stop building javadoc for now, as we do not ship it anyway (authored by zack).
doc: stop building javadoc for now, as we do not ship it anyway
Nov 9 2019, 6:23 PM
zack committed rDGRPHddf77c8d263b: doc: integrate git2graph doc into top-level doc and toc (authored by zack).
doc: integrate git2graph doc into top-level doc and toc
Nov 9 2019, 2:42 PM
zack committed rDGRPH222bfbd615a8: git2graph doc: explain why only HEAD is supported among symbolic refs (authored by zack).
git2graph doc: explain why only HEAD is supported among symbolic refs
Nov 9 2019, 2:42 PM
zack committed rDGRPH5ddbc352be39: git2graph: update doc and benchmark to use zstd (authored by zack).
git2graph: update doc and benchmark to use zstd
Nov 9 2019, 2:00 PM
zack created D2247: REST server: support HEAD requests.
Nov 9 2019, 11:08 AM
zack committed rDGRPHfcf6f1bf0776: javadoc: fix docstring syntax error in NodeTypesMap (authored by zack).
javadoc: fix docstring syntax error in NodeTypesMap
Nov 9 2019, 10:46 AM
zack committed rDGRPH87af24e3fd3d: Makefile: add convenience target "java-doc" (authored by zack).
Makefile: add convenience target "java-doc"
Nov 9 2019, 10:43 AM
zack committed rDGRPH353b91bf9e8a: doc: update docker-related documentation and scripts (authored by zack).
doc: update docker-related documentation and scripts
Nov 9 2019, 10:16 AM

Nov 8 2019

zack committed rCJSWH7429d91626e2: swh-pypy job: be more robust in how pom.xml are searched (authored by zack).
swh-pypy job: be more robust in how pom.xml are searched
Nov 8 2019, 9:24 PM
zack updated the summary of D2246: REST server: add validation for PID parameters.
Nov 8 2019, 8:39 PM
zack created D2246: REST server: add validation for PID parameters.
Nov 8 2019, 8:36 PM
zack created D2245: REST API doc: update to match current aiohttp implementation.
Nov 8 2019, 8:08 PM
zack updated the diff for D2244: REST server: set content-type to text or ndjson where appropriate.

REST server: set content-type to text or ndjson where appropriate

Nov 8 2019, 7:42 PM
zack updated the diff for D2244: REST server: set content-type to text or ndjson where appropriate.

REST server: set content-type to text or ndjson where appropriate

Nov 8 2019, 7:39 PM
zack created D2244: REST server: set content-type to text or ndjson where appropriate.
Nov 8 2019, 6:24 PM
zack triaged T2072: common configuration file for swh graph rpc-serve, compress, … as Normal priority.
Nov 8 2019, 3:03 PM · Compressed graph service
zack closed T1937: nicer landing page for the swh-graph REST API as Resolved.

this is now done in the aiohttp server, which says (with links):

Nov 8 2019, 2:25 PM · Compressed graph service
zack renamed Archive search from Search service to Archive search.
Nov 8 2019, 1:22 PM
zack committed rDSEA5f2a19336877: in_memory.py: make flake8 happy again (authored by zack).
in_memory.py: make flake8 happy again
Nov 8 2019, 1:18 PM
zack closed D2230: typing: add type annotations to make mypy pass.
Nov 8 2019, 1:18 PM
zack committed rDGRPH48f1802d062d: webgraph.py: use named pipes to read zst decompression output (authored by zack).
webgraph.py: use named pipes to read zst decompression output
Nov 8 2019, 11:06 AM

Nov 7 2019

zack closed T1944: use a compact, binary format for node ids mapping files as Resolved.

Closed in rDGRPH998a44353612

Nov 7 2019, 11:36 PM · Compressed graph service
zack closed T1944: use a compact, binary format for node ids mapping files, a subtask of T1950: Reduce RAM usage for generating mapping files, as Resolved.
Nov 7 2019, 11:36 PM · Compressed graph service
zack closed T1950: Reduce RAM usage for generating mapping files as Resolved.

Closed in 6d2f04b4d5a4

Nov 7 2019, 11:35 PM · Compressed graph service
zack added a comment to T2053: support graph export for the cassandra backend.

Added parallelism. 450k/s with 16 workers and no compression. I won't try with 32 workers because Python processes would use too much CPU on my machine.

Nov 7 2019, 5:18 PM · Compressed graph service, Storage manager
zack accepted D2233: Migrate to PEP 526 variable annotations (Python 3.6+).
Nov 7 2019, 4:35 PM
zack committed rDDOC10256806cfb1: Makefile.local: port to new swh-environment typecheck naming (authored by zack).
Makefile.local: port to new swh-environment typecheck naming
Nov 7 2019, 1:45 PM
zack committed rDWAPPS44e353eb7012: Makefile.local: port to new swh-environment typecheck naming (authored by zack).
Makefile.local: port to new swh-environment typecheck naming
Nov 7 2019, 1:43 PM
zack committed rDDEP45a982ddc5be: Makefile.local: port to new swh-environment typecheck naming (authored by zack).
Makefile.local: port to new swh-environment typecheck naming
Nov 7 2019, 1:42 PM
zack updated the diff for D2230: typing: add type annotations to make mypy pass.
  • in_memory.py: make flake8 happy again
Nov 7 2019, 1:31 PM
zack abandoned D2231: in_memory.py: make flake8 happy again.
Nov 7 2019, 1:31 PM
zack created D2231: in_memory.py: make flake8 happy again.
Nov 7 2019, 1:29 PM
zack created D2230: typing: add type annotations to make mypy pass.
Nov 7 2019, 1:27 PM
zack committed rDENV5e20c707195b: Makefile.python: remove obsolete comment about mypy (authored by zack).
Makefile.python: remove obsolete comment about mypy
Nov 7 2019, 10:05 AM
zack committed rDENVe15a08b7b911: Makefile.python: make "make check" also run "make typecheck" (authored by zack).
Makefile.python: make "make check" also run "make typecheck"
Nov 7 2019, 10:04 AM
zack renamed T2066: when pushing several commits, only run CI on the most recent commit from when pushing several commits, do not run CI on all of them to when pushing several commits, only run CI on the most recent commit.
Nov 7 2019, 9:50 AM · Continuous Integration
zack triaged T2066: when pushing several commits, only run CI on the most recent commit as Normal priority.
Nov 7 2019, 9:50 AM · Continuous Integration
zack closed T1024: Proper continuous integration setup as Resolved.

closing this, as we have "proper" CI since quite a while now (it could always be improved of course, but no need to keep the meta task open at this point)

Nov 7 2019, 9:48 AM · Restricted Project, Continuous Integration, System administration
zack committed rDDEP17b3ee474e11: mypy: ignore psycopg2 (authored by zack).
mypy: ignore psycopg2
Nov 7 2019, 9:39 AM
zack committed rDDEPdd06930548f3: tox.ini: pass DJANGO_SETTINGS_MODULE to mypy environment (authored by zack).
tox.ini: pass DJANGO_SETTINGS_MODULE to mypy environment
Nov 7 2019, 9:39 AM
zack committed rDDEP39d124190019: mypy: properly type get/put methods, as well deposit_requests_types dict (authored by zack).
mypy: properly type get/put methods, as well deposit_requests_types dict
Nov 7 2019, 9:39 AM
zack committed rDDEP34153ef59b77: tox: add mypy environment (authored by zack).
tox: add mypy environment
Nov 7 2019, 9:39 AM
zack committed rDDEP4bb1de4415b0: Makefile.local: sync typecheck target with global Makefile.python (authored by zack).
Makefile.local: sync typecheck target with global Makefile.python
Nov 7 2019, 9:39 AM
zack committed rDDEP00440eb99622: requirements-test.txt: add back deps from master (authored by zack).
requirements-test.txt: add back deps from master
Nov 7 2019, 9:39 AM
zack committed rDDEP422b92fc1910: init.py: switch to documented way of extending path (authored by zack).
init.py: switch to documented way of extending path
Nov 7 2019, 9:39 AM
zack committed rDDEP0b3d526e956a: typing: convert leftover variable annotation to comments (authored by zack).
typing: convert leftover variable annotation to comments
Nov 7 2019, 9:39 AM
zack committed rDDEPfd8db57691c3: typing: minimal changes to make a no-op mypy run pass (authored by zack).
typing: minimal changes to make a no-op mypy run pass
Nov 7 2019, 9:39 AM
zack closed D2032: typing: minimal changes to make a no-op mypy run pass.
Nov 7 2019, 9:39 AM
zack updated the diff for D2032: typing: minimal changes to make a no-op mypy run pass.
  • mypy: properly type get/put methods, as well deposit_requests_types dict
Nov 7 2019, 9:31 AM
zack updated the diff for D2032: typing: minimal changes to make a no-op mypy run pass.
  • tox.ini: pass DJANGO_SETTINGS_MODULE to mypy environment
Nov 7 2019, 9:25 AM

Nov 6 2019

zack committed rCDFJ51522d682684: install zstd in base-jdk11 image (upcoming swh-graph dependency) (authored by zack).
install zstd in base-jdk11 image (upcoming swh-graph dependency)
Nov 6 2019, 4:44 PM
zack closed D2227: install zstd in base-jdk11 image (upcoming swh-graph dependency).
Nov 6 2019, 4:44 PM
zack created D2227: install zstd in base-jdk11 image (upcoming swh-graph dependency).
Nov 6 2019, 4:43 PM
zack triaged T2063: Investigate stream-based execution model as a replacement for current scheduler/celery/task based approach as Normal priority.
Nov 6 2019, 1:37 PM · Scheduling utilities

Nov 5 2019

zack added a comment to D2220: typing: minimal changes to make a no-op mypy run pass.

Currently, mypy checks will fail as new tags including the typing info for swh-vault and swh-indexer need to be created.

Nov 5 2019, 5:48 PM
zack accepted D2220: typing: minimal changes to make a no-op mypy run pass.

I'm approving this as it LGTM, but indeed we should wait for the CI update to 3.7 before merging.

Nov 5 2019, 5:27 PM
zack reopened T1944: use a compact, binary format for node ids mapping files as "Open".

reopen, as it's not fixed in master yet

Nov 5 2019, 4:18 PM · Compressed graph service
zack reopened T1944: use a compact, binary format for node ids mapping files, a subtask of T1950: Reduce RAM usage for generating mapping files, as Open.
Nov 5 2019, 4:18 PM · Compressed graph service
zack reopened T1950: Reduce RAM usage for generating mapping files as "Open".

reopen, as it's not closed in master yet

Nov 5 2019, 4:18 PM · Compressed graph service
zack committed rDGRPHf246fa859213: Makefile: add generic java-* dispatcher target (authored by zack).
Makefile: add generic java-* dispatcher target
Nov 5 2019, 3:46 PM
zack closed T1950: Reduce RAM usage for generating mapping files as Resolved by committing rDGRPH6d2f04b4d5a4: Setup.java: shell out node2pid map generation to sort.
Nov 5 2019, 3:46 PM · Compressed graph service
zack committed rDGRPH6d2f04b4d5a4: Setup.java: shell out node2pid map generation to sort (authored by zack).
Setup.java: shell out node2pid map generation to sort
Nov 5 2019, 3:46 PM
zack added a comment to T2053: support graph export for the cassandra backend.

Looks good, thanks !

Nov 5 2019, 2:05 PM · Compressed graph service, Storage manager
zack updated the task description for T2053: support graph export for the cassandra backend.
Nov 5 2019, 2:00 PM · Compressed graph service, Storage manager
zack committed rDSEA15145bf90c56: mypy: add skeleton (authored by zack).
mypy: add skeleton
Nov 5 2019, 11:38 AM
zack committed rDTPL91355c3f32f6: __init__.py: type extend_path properly (authored by zack).
__init__.py: type extend_path properly
Nov 5 2019, 11:33 AM
zack committed rDTPL6509d7bf815b: tox.ini: run mypy environment by default (authored by zack).
tox.ini: run mypy environment by default
Nov 5 2019, 11:31 AM
zack committed rDGRPH92bfc4d0eabb: Setup.java: remove unused import (authored by zack).
Setup.java: remove unused import
Nov 5 2019, 11:01 AM
zack committed rDGRPHd7747d412429: switch compression pipeline from gzip to zstd (authored by zack).
switch compression pipeline from gzip to zstd
Nov 5 2019, 10:40 AM
zack committed rDGRPHd6d5ef95a7b8: webgraph.py: use shell=True in compression step execution (authored by zack).
webgraph.py: use shell=True in compression step execution
Nov 5 2019, 10:40 AM
zack committed rDGRPHac15459387fe: rename int2pid/pid2int to node2pid/pid2node on the Python side (authored by zack).
rename int2pid/pid2int to node2pid/pid2node on the Python side
Nov 5 2019, 10:40 AM
zack triaged T2056: fix swh-graph sphinx table of content as Low priority.
Nov 5 2019, 10:24 AM · Documentation, Compressed graph service

Nov 4 2019

zack committed rDGRPH6f8266a5caaa: pid2int2int2pid: new tool to generate int->PID map from PID->int one (authored by zack).
pid2int2int2pid: new tool to generate int->PID map from PID->int one
Nov 4 2019, 8:45 PM
zack committed rDGRPHe970329f8ccd: CLI: add new "dumb" sequential map writer "swh graph map write" (authored by zack).
CLI: add new "dumb" sequential map writer "swh graph map write"
Nov 4 2019, 8:45 PM
zack committed rDGRPH55ba50a27da6: binary maps: change type IDs on Python side, to be compatible with Java (authored by zack).
binary maps: change type IDs on Python side, to be compatible with Java
Nov 4 2019, 2:39 PM
zack committed rDGRPHdb4a926489be: cli.py: document configuration parameter and reorder args (authored by zack).
cli.py: document configuration parameter and reorder args
Nov 4 2019, 2:39 PM
zack committed rDGRPH8b5cd0130f5d: test data: remove obsolete textual maps (authored by zack).
test data: remove obsolete textual maps
Nov 4 2019, 2:39 PM
zack committed rDGRPH78868a54b619: cli.py: update docstring doc about available compression steps (authored by zack).
cli.py: update docstring doc about available compression steps
Nov 4 2019, 2:39 PM
zack committed rDGRPH998a44353612: switch Java map generation from CSV to binary format (authored by zack).
switch Java map generation from CSV to binary format
Nov 4 2019, 2:39 PM
zack committed rDGRPHc997a4ce628e: test_cli.py: reduce memory requirements (authored by zack).
test_cli.py: reduce memory requirements
Nov 4 2019, 2:37 PM
zack committed rDGRPH8e9ecc3487b4: cli.py: add support for --config-file and compression configuration (authored by zack).
cli.py: add support for --config-file and compression configuration
Nov 4 2019, 2:37 PM
zack renamed T2055: swh-graph CI hangs badly when py4j doesn't find needed files from swh-graph CI hangs badly on test_api_client.py to swh-graph CI hangs badly when py4j doesn't find needed files.
Nov 4 2019, 1:45 PM · Continuous Integration, Compressed graph service
zack committed rDGRPHc1c1ad1fb25b: tox: more robust detection of swh-graph JAR (authored by zack).
tox: more robust detection of swh-graph JAR
Nov 4 2019, 1:39 PM
zack accepted D2213: Update README.md with basic info about the package..
Nov 4 2019, 1:18 PM
zack committed rDLDHG887adf10aeb2: typing: minimal changes to make a no-op mypy run pass (authored by zack).
typing: minimal changes to make a no-op mypy run pass
Nov 4 2019, 12:19 PM
zack closed D2210: typing: minimal changes to make a no-op mypy run pass.
Nov 4 2019, 12:19 PM
zack committed rDLDNPM2d3057edbe75: MANIFEST.in: ship py.typed (authored by zack).
MANIFEST.in: ship py.typed
Nov 4 2019, 12:18 PM
zack committed rDLDNPMd2a8f15f582c: typing: minimal changes to make a no-op mypy run pass (authored by zack).
typing: minimal changes to make a no-op mypy run pass
Nov 4 2019, 12:18 PM
zack closed D2211: typing: minimal changes to make a no-op mypy run pass.
Nov 4 2019, 12:18 PM