Page MenuHomeSoftware Heritage

Replace `entry_to_bytes` with psycopg2's typecast
ClosedPublic

Authored by vlorentz on Feb 1 2019, 11:06 AM.

Details

Summary

entry_to_bytes and its friends were called many times (eg.
entry_to_bytes alone was called 40k times while indexing 500
origins with the metadata indexer), and its use of isinstance
used a non-negligible amount of CPU time.

Instead of using *_to_bytes function as post-processing on
all bits of data returned by postgresql, this patch tells psycopg2
to use a new typecast_bytea function when needed (in adapt_conn).
This function deffers the decoding work to psycopg2, which returns a
memoryview, which is turned into bytes.

Replaces D1053.

Diff Detail

Repository
rDCORE Foundations and core functionalities
Branch
typecast-bytea
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 3928
Build 5149: tox-on-jenkinsJenkins
Build 5148: arc lint + arc unit

Event Timeline

vlorentz created this revision.Feb 1 2019, 11:06 AM
ardumont accepted this revision.Feb 1 2019, 11:11 AM
This revision is now accepted and ready to land.Feb 1 2019, 11:11 AM
vlorentz updated this revision to Diff 3342.Feb 1 2019, 1:28 PM
  • rebase
This revision was automatically updated to reflect the committed changes.