Page MenuHomeSoftware Heritage
Feed Advanced Search

Aug 14 2019

haltode updated the task description for T1951: Reduce RAM usage in graph API endpoints.
Aug 14 2019, 9:47 PM · Compressed graph service
haltode committed rDGRPH6a84a51b6f6a: server: benchmark: add proper CLI args management (authored by haltode).
server: benchmark: add proper CLI args management
Aug 14 2019, 9:39 PM
haltode closed D1852: Add proper CLI args management in benchmarking tools.
Aug 14 2019, 9:39 PM
haltode updated subscribers of T1951: Reduce RAM usage in graph API endpoints.
Aug 14 2019, 9:33 PM · Compressed graph service
haltode updated the diff for D1852: Add proper CLI args management in benchmarking tools.

Use null instead of -1 value.

Aug 14 2019, 9:32 PM
haltode added inline comments to D1852: Add proper CLI args management in benchmarking tools.
Aug 14 2019, 9:27 PM
haltode triaged T1951: Reduce RAM usage in graph API endpoints as High priority.
Aug 14 2019, 9:26 PM · Compressed graph service
haltode updated the diff for D1852: Add proper CLI args management in benchmarking tools.

Use default Random constructor if no seed is passed by CLI argument.

Aug 14 2019, 9:08 PM
haltode created D1852: Add proper CLI args management in benchmarking tools.
Aug 14 2019, 2:53 PM
haltode committed rDGRPH9b02f4137cb6: server: benchmark: point to specific section in docs URL (authored by haltode).
server: benchmark: point to specific section in docs URL
Aug 14 2019, 2:14 PM
haltode committed rDGRPH2da9b1a05395: server: add provenance use-cases benchmarks (authored by haltode).
server: add provenance use-cases benchmarks
Aug 14 2019, 2:09 PM
haltode closed D1851: Add provenance use-cases benchmarks.
Aug 14 2019, 2:09 PM
haltode created D1851: Add provenance use-cases benchmarks.
Aug 14 2019, 2:06 PM
haltode committed rDGRPH258aaf24354a: server: add input wrapper class for endpoints methods (authored by haltode).
server: add input wrapper class for endpoints methods
Aug 14 2019, 2:05 PM
haltode closed D1850: Add input wrapper class for endpoints methods.
Aug 14 2019, 2:05 PM
haltode updated the diff for D1850: Add input wrapper class for endpoints methods.

Remove useless comment.

Aug 14 2019, 1:31 PM
haltode created D1850: Add input wrapper class for endpoints methods.
Aug 14 2019, 1:27 PM
haltode added a comment to D1849: Add vault use-case benchmark.

As told on IRC, I was waiting on adding all benchmarking interfaces before implementing the CLI options.

Aug 14 2019, 1:25 PM
haltode closed D1849: Add vault use-case benchmark.
Aug 14 2019, 1:24 PM
haltode committed rDGRPH9ca54a873d7a: server: add vault use-case benchmark (authored by haltode).
server: add vault use-case benchmark
Aug 14 2019, 1:24 PM
haltode created P505 Browsing benchmarks early results in the S1 Public space.
Aug 14 2019, 11:22 AM · Compressed graph service
haltode created D1849: Add vault use-case benchmark.
Aug 14 2019, 10:55 AM

Aug 13 2019

haltode committed rDGRPH9ea36f08ce24: server: move utils/ into benchmark/utils (authored by haltode).
server: move utils/ into benchmark/utils
Aug 13 2019, 3:05 PM
haltode closed D1846: Add browsing use-cases benchmarks.
Aug 13 2019, 3:05 PM
haltode committed rDGRPH6aee508c4cdf: server: add browsing use-cases benchmarks (authored by haltode).
server: add browsing use-cases benchmarks
Aug 13 2019, 3:05 PM
haltode updated the diff for D1846: Add browsing use-cases benchmarks.
  • Add a new meta info nbEdgesAccessed
  • Move utils/ into benchmark/utils/
Aug 13 2019, 2:24 PM
haltode added inline comments to D1846: Add browsing use-cases benchmarks.
Aug 13 2019, 11:33 AM
haltode added inline comments to D1846: Add browsing use-cases benchmarks.
Aug 13 2019, 10:41 AM

Aug 12 2019

haltode created D1846: Add browsing use-cases benchmarks.
Aug 12 2019, 4:43 PM
haltode committed rDGRPH13d53c66452a: server: use double instead of float in timings (authored by haltode).
server: use double instead of float in timings
Aug 12 2019, 2:48 PM
haltode committed rDTPLc0caae801b1c: add missing swh/foo/__init__.py (authored by haltode).
add missing swh/foo/__init__.py
Aug 12 2019, 1:22 PM
haltode closed D1844: Add missing swh/foo/__init__.py.
Aug 12 2019, 1:21 PM
haltode committed rDGRPH22b9258fb2cc: server: archive LinuxLog outdated benchmark (authored by haltode).
server: archive LinuxLog outdated benchmark
Aug 12 2019, 1:05 PM
haltode added a comment to D1844: Add missing swh/foo/__init__.py.

I don't think you should put yourself in the contributors of a template repo

Aug 12 2019, 12:16 PM
haltode updated the diff for D1844: Add missing swh/foo/__init__.py.

Remove CONTRIBUTORS changes.

Aug 12 2019, 12:15 PM
haltode created D1844: Add missing swh/foo/__init__.py.
Aug 12 2019, 12:06 PM
haltode committed rDGRPHfb57ec5143a8: server: run clang-format (authored by haltode).
server: run clang-format
Aug 12 2019, 12:02 PM
haltode committed rDGRPHf12a5aafff26: client: add missing __init__.py file (authored by haltode).
client: add missing __init__.py file
Aug 12 2019, 12:02 PM
haltode committed rDGRPH88bfedaf9883: server: rename SwhId to SwhPID (authored by haltode).
server: rename SwhId to SwhPID
Aug 12 2019, 11:52 AM
haltode closed D1842: Rename SwhId to SwhPID.
Aug 12 2019, 11:52 AM
haltode created D1842: Rename SwhId to SwhPID.
Aug 12 2019, 10:53 AM
haltode committed rDGRPH1fd382b69889: client: tests: update to new JSON return format (authored by haltode).
client: tests: update to new JSON return format
Aug 12 2019, 10:18 AM
haltode committed rDGRPH8ff03d608db6: client: remove warning about unused variable (authored by haltode).
client: remove warning about unused variable
Aug 12 2019, 10:18 AM

Aug 10 2019

haltode added a subtask for T1950: Reduce RAM usage for generating mapping files: T1944: use a compact, binary format for node ids mapping files.
Aug 10 2019, 9:23 AM · Compressed graph service
haltode added a parent task for T1944: use a compact, binary format for node ids mapping files: T1950: Reduce RAM usage for generating mapping files.
Aug 10 2019, 9:23 AM · Compressed graph service
haltode triaged T1950: Reduce RAM usage for generating mapping files as Normal priority.
Aug 10 2019, 9:22 AM · Compressed graph service
haltode abandoned D1802: [WIP] server: setup: use RandomAccessFile instead of temporary array.

Results from Azure with random write using custom mmap:

Aug 10 2019, 9:20 AM

Aug 9 2019

haltode committed rDGRPH7e1917a236f3: server: add endpoints wrapper class to return metadata (authored by haltode).
server: add endpoints wrapper class to return metadata
Aug 9 2019, 4:12 PM
haltode closed T1945: Return timings instead of simply logging them as Resolved by committing rDGRPH7e1917a236f3: server: add endpoints wrapper class to return metadata.
Aug 9 2019, 4:12 PM · Compressed graph service
haltode closed D1832: Endpoints now return timings instead of logging them.
Aug 9 2019, 4:12 PM
haltode updated the diff for D1832: Endpoints now return timings instead of logging them.

Update JSON output format in api.rst (content -> result, add meta dict).

Aug 9 2019, 9:51 AM
haltode updated the diff for D1832: Endpoints now return timings instead of logging them.

Update documentation (api.rst and server README).

Aug 9 2019, 9:48 AM

Aug 8 2019

haltode updated the diff for D1832: Endpoints now return timings instead of logging them.
  • Timings are now optional
  • Improve JSON output structure (rename Result class to Output, rename

content member to result, add meta member)

Aug 8 2019, 11:54 PM
haltode committed rCJSWH7e4693a09bef: jobs/swh-packages: add swh-graph package (authored by haltode).
jobs/swh-packages: add swh-graph package
Aug 8 2019, 4:39 PM
haltode closed D1833: jobs/swh-packages: add swh-graph package.
Aug 8 2019, 4:39 PM
haltode added a comment to D1832: Endpoints now return timings instead of logging them.
In D1832#42599, @zack wrote:

I hadn't in mind to lift the timings up to the REST layer, but why not, it will enable doing interesting stuff using non Java clients.

Aug 8 2019, 4:39 PM
haltode planned changes to D1802: [WIP] server: setup: use RandomAccessFile instead of temporary array.
Aug 8 2019, 4:30 PM
Herald added a reviewer for D1833: jobs/swh-packages: add swh-graph package: Reviewers.
Aug 8 2019, 4:25 PM
haltode added a revision to T1945: Return timings instead of simply logging them: D1832: Endpoints now return timings instead of logging them.
Aug 8 2019, 3:53 PM · Compressed graph service
haltode created D1832: Endpoints now return timings instead of logging them.
Aug 8 2019, 3:53 PM
haltode committed rDGRPHba9b3913c5a9: server: use LazyLongIterator instead of array in NeighborsIterator (authored by haltode).
server: use LazyLongIterator instead of array in NeighborsIterator
Aug 8 2019, 3:03 PM
haltode committed rDGRPH8f19048ff75e: server: add method to pick random nodes with specific type (authored by haltode).
server: add method to pick random nodes with specific type
Aug 8 2019, 3:03 PM
haltode committed rDGRPHa49549868917: server: add benchmark to test edge access time (authored by haltode).
server: add benchmark to test edge access time
Aug 8 2019, 3:03 PM
haltode closed D1831: Add benchmark to test edge access time.
Aug 8 2019, 3:03 PM
haltode updated the diff for D1831: Add benchmark to test edge access time.

Add method to pick random nodes with specific type

Aug 8 2019, 11:34 AM
haltode updated the diff for D1831: Add benchmark to test edge access time.

Use LazyLongIterator instead of array in NeighborsIterator

Aug 8 2019, 11:05 AM
haltode triaged T1945: Return timings instead of simply logging them as Normal priority.
Aug 8 2019, 10:34 AM · Compressed graph service
haltode triaged T1944: use a compact, binary format for node ids mapping files as Normal priority.
Aug 8 2019, 10:29 AM · Compressed graph service

Aug 7 2019

haltode updated the diff for D1831: Add benchmark to test edge access time.

Add reference to T1885 in commit message.

Aug 7 2019, 10:10 PM
haltode added a comment to D1831: Add benchmark to test edge access time.

I rebooted the azure VM and did the benchmark, here is the result for 1M edges:

Aug 7 2019, 8:01 PM
haltode created D1831: Add benchmark to test edge access time.
Aug 7 2019, 7:02 PM

Aug 6 2019

haltode closed D1822: Add detailed internal explanations in javadoc.
Aug 6 2019, 7:39 PM
haltode committed rDGRPH02e4f44f91e0: server: add detailed internal explanations in javadoc (authored by haltode).
server: add detailed internal explanations in javadoc
Aug 6 2019, 7:39 PM
haltode created D1822: Add detailed internal explanations in javadoc.
Aug 6 2019, 4:30 PM
haltode closed D1821: Add javadoc generation in swh-graph docs assets.
Aug 6 2019, 10:43 AM
haltode committed rDGRPH81b6c0032c8c: docs: generate javadoc in assets (authored by haltode).
docs: generate javadoc in assets
Aug 6 2019, 10:43 AM
haltode updated the diff for D1821: Add javadoc generation in swh-graph docs assets.

Rebasing on master.

Aug 6 2019, 10:42 AM
haltode updated the diff for D1821: Add javadoc generation in swh-graph docs assets.

Add clean-javadoc target.

Aug 6 2019, 10:36 AM
haltode added a comment to D1802: [WIP] server: setup: use RandomAccessFile instead of temporary array.

Last solution I tried: using a custom mmap output file instead of the temporary array. On monster, results seem better!

Aug 6 2019, 10:29 AM
haltode committed rDGRPH78f18d06f734: server: test: add all assets for mock graph (authored by haltode).
server: test: add all assets for mock graph
Aug 6 2019, 10:17 AM
haltode committed rDGRPHf5d245f4e312: server: tests: fix multiple possible solution unit test (authored by haltode).
server: tests: fix multiple possible solution unit test
Aug 6 2019, 10:13 AM
haltode created D1821: Add javadoc generation in swh-graph docs assets.
Aug 6 2019, 12:32 AM

Aug 5 2019

haltode closed T1877: Add contextual info to compression pipeline as Resolved by committing rDGRPH403f1e010c3e: dockerfiles: add contextual info to compression script.
Aug 5 2019, 3:37 PM · Compressed graph service
haltode committed rDGRPH403f1e010c3e: dockerfiles: add contextual info to compression script (authored by haltode).
dockerfiles: add contextual info to compression script
Aug 5 2019, 3:37 PM
haltode closed D1817: Add contextual info to compression script.
Aug 5 2019, 3:37 PM
haltode updated the diff for D1817: Add contextual info to compression script.
  • Refactor into a step_info function
  • Use "* swh-graph:" prefix
  • Add step numbering
Aug 5 2019, 3:32 PM
haltode added a revision to T1877: Add contextual info to compression pipeline: D1817: Add contextual info to compression script.
Aug 5 2019, 2:37 PM · Compressed graph service
haltode created D1817: Add contextual info to compression script.
Aug 5 2019, 2:37 PM
haltode closed T1933: bad invocation of o.s.graph.backend.Setup in docker doc as Resolved by committing rDGRPH4013dbc12e43: docs: fix backend.Setup command arguments.
Aug 5 2019, 2:11 PM · Compressed graph service
haltode committed rDGRPH4013dbc12e43: docs: fix backend.Setup command arguments (authored by haltode).
docs: fix backend.Setup command arguments
Aug 5 2019, 2:11 PM
haltode closed T1867: compress Merkle DAG and origin nodes together as Resolved.
Aug 5 2019, 10:24 AM · Compressed graph service
haltode added a comment to T1867: compress Merkle DAG and origin nodes together.

Due to multiple server maintenance, the process was re-started a few times, but it is now finished and results are uploaded in the annex: https://annex.softwareheritage.org/public/dataset/graph/latest/compressed/all+ori/

Aug 5 2019, 10:24 AM · Compressed graph service
haltode added a comment to D1802: [WIP] server: setup: use RandomAccessFile instead of temporary array.

Some updates on the running experiments, I tried with both monster and our new azure VM.

Aug 5 2019, 10:20 AM

Aug 3 2019

haltode added a comment to T1904: build developer documentation for swh-graph.

What was tried so far:

Aug 3 2019, 11:13 PM · Documentation, Compressed graph service

Aug 2 2019

haltode created D1802: [WIP] server: setup: use RandomAccessFile instead of temporary array.
Aug 2 2019, 1:47 PM
haltode committed rDGRPH223650ed7e6a: server: move time measurement methods to a dedicated class (authored by haltode).
server: move time measurement methods to a dedicated class
Aug 2 2019, 11:21 AM

Aug 1 2019

haltode committed rDCOREff794e8d946f: api: tests: use aiohttp unused_port() (authored by haltode).
api: tests: use aiohttp unused_port()
Aug 1 2019, 2:55 PM
haltode closed D1800: api: tests: use aiohttp unused_port().
Aug 1 2019, 2:55 PM
haltode updated the diff for D1800: api: tests: use aiohttp unused_port().

Add CONTRIBUTORS file

Aug 1 2019, 2:52 PM
haltode closed T1920: graph service: add tests for the python client as Resolved by committing rDGRPHc57819ce34ae: client: add API endpoints integration tests.
Aug 1 2019, 2:49 PM · Compressed graph service