Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T2782: add a "Filter Pull Requests" checkbox (or similar) in the Branches view of an origin in the web UI

Commits

rDSTOb565201dcfe2: storage: Allow to filter out branches by prefix when counting them

Summary

Add an optional branch_name_exclude_prefix parameter to the
snapshot_count_branches method of the Storage interface.

It enables to filter out branches whose name starts with a
given prefix when counting.

The purpose is to get accurate counters in swh-web as pull
request branches will be filtered out by default.

Related to T2782

Depends on D4615

Diff Detail

Repository

rDSTO Storage manager

Branch

snapshot-count-branches-exclude-prefix

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 19770
Build 30694: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 30693: arc lint + arc unit

Event Timeline

anlambert created this revision.Mar 5 2021, 5:27 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 5 2021, 5:27 PM

Build is green

Patch application report for D5208 (id=18658)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..047061b1
Fast-forward
 sql/upgrades/168.sql               |  39 +++++++++++
 swh/storage/cassandra/cql.py       |  31 ++++++++-
 swh/storage/cassandra/storage.py   |  27 +++++++-
 swh/storage/in_memory.py           |   9 ++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++++--
 swh/storage/postgresql/storage.py  |  19 +++++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 +++-
 swh/storage/tests/storage_tests.py | 128 +++++++++++++++++++++++++++++++++++++
 10 files changed, 285 insertions(+), 20 deletions(-)
 create mode 100644 sql/upgrades/168.sql

Changes applied before test

commit 047061b14e37f4674394a24fc06a32c847b476be
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit 2f1fdfe322435b59d08bd56e4a521d01caddacd6
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1188/ for more details.

Harbormaster completed remote builds in B19751: Diff 18658.Mar 5 2021, 5:32 PM

anlambert requested review of this revision.Mar 5 2021, 5:32 PM

Simplify in memory implementation.

anlambert mentioned this in D4616: common/archive: Add branch names filtering support in lookup_snapshot.Mar 5 2021, 5:53 PM

anlambert added a child revision: D4616: common/archive: Add branch names filtering support in lookup_snapshot.Mar 5 2021, 5:53 PM

Build is green

Patch application report for D5208 (id=18660)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..2d8a6484
Fast-forward
 sql/upgrades/168.sql               |  39 +++++++++++
 swh/storage/cassandra/cql.py       |  31 ++++++++-
 swh/storage/cassandra/storage.py   |  27 +++++++-
 swh/storage/in_memory.py           |   8 ++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++++--
 swh/storage/postgresql/storage.py  |  19 +++++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 +++-
 swh/storage/tests/storage_tests.py | 128 +++++++++++++++++++++++++++++++++++++
 10 files changed, 284 insertions(+), 20 deletions(-)
 create mode 100644 sql/upgrades/168.sql

Changes applied before test

commit 2d8a648438a9540e91ac7b637f7eb7a26c0fd091
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit 2f1fdfe322435b59d08bd56e4a521d01caddacd6
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1190/ for more details.

Harbormaster completed remote builds in B19753: Diff 18660.Mar 5 2021, 6:01 PM

Thanks.

We could probably improve the perfs by restricting what rows are scanned (in both pg and cass), but we can do it in the future diff without changing this interface.

Just one comment though:

swh/storage/interface.py
683	Should be `bytes`, not `str`.

This revision now requires changes to proceed.Mar 8 2021, 10:01 AM

Bump db version and add missing migration file

Build is green

Patch application report for D5208 (id=18674)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..c8a1c643
Fast-forward
 sql/upgrades/168.sql               |  39 +++++++++++
 sql/upgrades/169.sql               |  19 ++++++
 swh/storage/cassandra/cql.py       |  31 ++++++++-
 swh/storage/cassandra/storage.py   |  27 +++++++-
 swh/storage/in_memory.py           |   8 ++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++++--
 swh/storage/postgresql/storage.py  |  19 +++++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 +++-
 swh/storage/tests/storage_tests.py | 128 +++++++++++++++++++++++++++++++++++++
 11 files changed, 303 insertions(+), 20 deletions(-)
 create mode 100644 sql/upgrades/168.sql
 create mode 100644 sql/upgrades/169.sql

Changes applied before test

commit c8a1c643f9460d1dd5f97e47828c2d26f2ea6dff
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit 2f1fdfe322435b59d08bd56e4a521d01caddacd6
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1191/ for more details.

Harbormaster completed remote builds in B19767: Diff 18674.Mar 8 2021, 10:09 AM

Change branch_name_exclude_prefix parameter type from str to bytes.

Build is green

Patch application report for D5208 (id=18677)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..97edab89
Fast-forward
 sql/upgrades/168.sql               |  39 ++++++++++++
 sql/upgrades/169.sql               |  19 ++++++
 swh/storage/cassandra/cql.py       |  31 ++++++++-
 swh/storage/cassandra/storage.py   |  25 +++++++-
 swh/storage/in_memory.py           |   8 ++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++++--
 swh/storage/postgresql/storage.py  |  19 +++++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 +++-
 swh/storage/tests/storage_tests.py | 127 +++++++++++++++++++++++++++++++++++++
 11 files changed, 300 insertions(+), 20 deletions(-)
 create mode 100644 sql/upgrades/168.sql
 create mode 100644 sql/upgrades/169.sql

Changes applied before test

commit 97edab890047ae614950b1febbd17ac92d92be2b
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit b1f0caaee70b5291fbe6b90f7ae1eea434ae821b
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1193/ for more details.

Harbormaster completed remote builds in B19770: Diff 18677.Mar 8 2021, 11:17 AM

vlorentz accepted this revision.Mar 8 2021, 12:45 PM

This revision is now accepted and ready to land.Mar 8 2021, 12:45 PM

anlambert planned changes to this revision.Mar 11 2021, 2:20 PM

anlambert mentioned this in D4615: storage: Add branch names filtering support in snapshot_get_branches.

Update:

Rebase

Properly implement branches count with branch name prefix exclude pattern in cassandra backend

This revision is now accepted and ready to land.Mar 11 2021, 3:42 PM

Build is green

Patch application report for D5208 (id=18750)

Could not rebase; Attempt merge onto b8e10f00cf...

Updating b8e10f00..b41e3f20
Fast-forward
 sql/upgrades/169.sql               |  39 ++++++++++++
 sql/upgrades/170.sql               |  19 ++++++
 swh/storage/cassandra/cql.py       |  87 +++++++++++++++++++++++--
 swh/storage/cassandra/storage.py   |  26 +++++++-
 swh/storage/in_memory.py           |  19 +++++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++++--
 swh/storage/postgresql/storage.py  |  19 +++++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 +++-
 swh/storage/tests/storage_tests.py | 127 +++++++++++++++++++++++++++++++++++++
 11 files changed, 362 insertions(+), 26 deletions(-)
 create mode 100644 sql/upgrades/169.sql
 create mode 100644 sql/upgrades/170.sql

Changes applied before test

commit b41e3f200b6362ad34caa68a4dd13ddff11b4188
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit 017055d705b3bd7b5cda2548d17e3afb2d9a75cd
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1216/ for more details.

Harbormaster completed remote builds in B19846: Diff 18750.Mar 11 2021, 3:57 PM

vlorentz requested changes to this revision.Mar 11 2021, 5:20 PM

vlorentz added inline comments.

swh/storage/cassandra/cql.py
660	doesn't work if the last byte of the prefix is `\xff`. You'll need to either implement the carry, or convert to an int

This revision now requires changes to proceed.Mar 11 2021, 5:20 PM

vlorentz added inline comments.Mar 11 2021, 5:22 PM

swh/storage/cassandra/cql.py
660	and you'll also need to handle the issue of prefixes that are made entirely of `\xff` :/

Update: Fix edge cases for branch name exclude filter in cassandra backend and add test.

Build is green

Patch application report for D5208 (id=18755)

Could not rebase; Attempt merge onto b8e10f00cf...

Updating b8e10f00..83d8630c
Fast-forward
 sql/upgrades/169.sql               |  39 +++++++++
 sql/upgrades/170.sql               |  19 ++++
 swh/storage/cassandra/cql.py       |  92 ++++++++++++++++++--
 swh/storage/cassandra/storage.py   |  26 +++++-
 swh/storage/in_memory.py           |  19 +++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++--
 swh/storage/postgresql/storage.py  |  19 +++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 ++-
 swh/storage/tests/storage_tests.py | 174 +++++++++++++++++++++++++++++++++++++
 11 files changed, 414 insertions(+), 26 deletions(-)
 create mode 100644 sql/upgrades/169.sql
 create mode 100644 sql/upgrades/170.sql

Changes applied before test

commit 83d8630cf299ed4188c3b4011ae58be3a153863a
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit 09a017827eb31cbec05252a7b200fd6af245a0a8
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1219/ for more details.

Harbormaster completed remote builds in B19851: Diff 18755.Mar 11 2021, 7:32 PM

Looks correct now!

Good you just add some a short comment in snapshot_count_branches and a docstring to _next_prefix to explain what they do?

This revision is now accepted and ready to land.Mar 11 2021, 9:30 PM

Update: Add comments and rename _next_prefix to _next_bytes_value.

Rebase

Build was aborted

Patch application report for D5208 (id=18781)

Could not rebase; Attempt merge onto b8e10f00cf...

Updating b8e10f00..61fc4bfc
Fast-forward
 sql/upgrades/169.sql               |  39 ++++++++
 sql/upgrades/170.sql               |  19 ++++
 swh/storage/cassandra/cql.py       | 104 ++++++++++++++++++++--
 swh/storage/cassandra/storage.py   |  26 +++++-
 swh/storage/in_memory.py           |  19 +++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++--
 swh/storage/postgresql/storage.py  |  19 +++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 ++-
 swh/storage/tests/storage_tests.py | 178 +++++++++++++++++++++++++++++++++++++
 11 files changed, 430 insertions(+), 26 deletions(-)
 create mode 100644 sql/upgrades/169.sql
 create mode 100644 sql/upgrades/170.sql

Changes applied before test

commit 61fc4bfc75ca07dab08c893b1d72e3c8fac87f95
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit d080085e7ff75d41972e4106b76c02183fcce987
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_pattern parameter to snapshot_get_branches,
    if provided only branches whose name contains the given pattern will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1221/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1221/console

Harbormaster failed remote builds in B19879: Diff 18781!Mar 12 2021, 3:46 PM

Build is green

Patch application report for D5208 (id=18783)

Could not rebase; Attempt merge onto b8e10f00cf...

Updating b8e10f00..b565201d
Fast-forward
 sql/upgrades/169.sql               |  39 ++++++++
 sql/upgrades/170.sql               |  19 ++++
 swh/storage/cassandra/cql.py       | 104 ++++++++++++++++++++--
 swh/storage/cassandra/storage.py   |  26 +++++-
 swh/storage/in_memory.py           |  19 +++-
 swh/storage/interface.py           |  10 ++-
 swh/storage/postgresql/db.py       |  28 ++++--
 swh/storage/postgresql/storage.py  |  19 +++-
 swh/storage/sql/30-schema.sql      |   2 +-
 swh/storage/sql/40-funcs.sql       |  12 ++-
 swh/storage/tests/storage_tests.py | 178 +++++++++++++++++++++++++++++++++++++
 11 files changed, 430 insertions(+), 26 deletions(-)
 create mode 100644 sql/upgrades/169.sql
 create mode 100644 sql/upgrades/170.sql

Changes applied before test

commit b565201dcfe2a007a628de94323b84c1045b7b0c
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Mar 5 16:33:29 2021 +0100

    storage: Allow to filter out branches by prefix when counting them
    
    Add an optional branch_name_exclude_prefix parameter to the
    snapshot_count_branches method of the Storage interface.
    
    It enables to filter out branches whose name starts with a
    given prefix when counting.
    
    The purpose is to get accurate counters in swh-web as pull
    request branches will be filtered out by default.
    
    Related to T2782

commit 93301a1f67acba349319c383dac9031a132a7470
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Mar 2 14:42:57 2021 +0100

    storage: Add branch names filtering support in snapshot_get_branches
    
    Add optional branch_name_include_substring parameter to snapshot_get_branches,
    if provided only branches whose name contains the given substring will be
    returned.
    
    Add optional branch_name_exclude_prefix parameter to snapshot_get_branches,
    if provided branches whose name starts with the given prefix will not be
    returned.
    
    Purpose of these new features: add a search form in the branches view
    of swh-web and filter out pull request branches (whose names start with
    "refs/pull/") by default.
    
    Related to T2782

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1223/ for more details.

Harbormaster completed remote builds in B19881: Diff 18783.Mar 12 2021, 4:09 PM

Closed by commit rDSTOb565201dcfe2: storage: Allow to filter out branches by prefix when counting them (authored by anlambert). · Explain WhyMar 15 2021, 11:26 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDSTOb565201dcfe2: storage: Allow to filter out branches by prefix when counting them.

Diff 18677

View Options

sql/upgrades/169.sql

This file was added.

				-- SWH DB schema upgrade
				-- from_version: 168
				-- to_version: 169
				-- description: Make origin_visit_status.type not null

				insert into dbversion(version, release, description)
				values(169, now(), 'Work In Progress');

				create or replace function swh_snapshot_count_branches(id sha1_git,
				branch_name_exclude_prefix bytea default NULL)
				returns setof snapshot_size
				language sql
				stable
				as $$
				SELECT target_type, count(name)
				from swh_snapshot_get_by_id(swh_snapshot_count_branches.id,
				branch_name_exclude_prefix => swh_snapshot_count_branches.branch_name_exclude_prefix)
				group by target_type;
				$$;

View Options

swh/storage/cassandra/cql.py

Show First 20 Lines • Show All 608 Lines • ▼ Show 20 Lines	class CqlRunner:

@_prepared_insert_statement(SnapshotBranchRow)		@_prepared_insert_statement(SnapshotBranchRow)
def snapshot_branch_add_one(self, branch: SnapshotBranchRow, *, statement) -> None:		def snapshot_branch_add_one(self, branch: SnapshotBranchRow, *, statement) -> None:
self._add_one(statement, branch)		self._add_one(statement, branch)

@_prepared_statement(		@_prepared_statement(
"SELECT ascii_bins_count(target_type) AS counts "		"SELECT ascii_bins_count(target_type) AS counts "
"FROM snapshot_branch "		"FROM snapshot_branch "
"WHERE snapshot_id = ? "		"WHERE snapshot_id = ? AND name >= ? AND name < ?"
)		)
def snapshot_count_branches(		def snapshot_count_branches(
self, snapshot_id: Sha1Git, *, statement		self,
		snapshot_id: Sha1Git,
		branch_name_exclude_prefix: Optional[bytes],
		*,
		statement,
) -> Dict[Optional[str], int]:		) -> Dict[Optional[str], int]:
"""Returns a dictionary from type names to the number of branches		"""Returns a dictionary from type names to the number of branches
of that type."""		of that type."""
row = self._execute_with_retries(statement, [snapshot_id]).one()		prefix = b""
		next_prefix = b"\xff"
		row = self._execute_with_retries(
		statement, [snapshot_id, prefix, next_prefix],
		).one()
(nb_none, counts) = row["counts"]		(nb_none, counts) = row["counts"]

		# count branch names starting with prefix and subtstract values
		# from global counters
		if branch_name_exclude_prefix:
		prefix = branch_name_exclude_prefix
		next_prefix = prefix[:-1] + bytes([prefix[-1] + 1])
		row = self._execute_with_retries(
		statement, [snapshot_id, prefix, next_prefix],
		).one()
		(nb_none_prefix, counts_prefix) = row["counts"]

		for target_type, count in list(counts_prefix.items()):
		counts[target_type] -= count
		if counts[target_type] == 0:
		del counts[target_type]
		nb_none -= nb_none_prefix

return {None: nb_none, **counts}		return {None: nb_none, **counts}

@_prepared_select_statement(		@_prepared_select_statement(
SnapshotBranchRow, "WHERE snapshot_id = ? AND name >= ? LIMIT ?"		SnapshotBranchRow, "WHERE snapshot_id = ? AND name >= ? LIMIT ?"
)		)
def snapshot_branch_get(		def snapshot_branch_get(
self, snapshot_id: Sha1Git, from_: bytes, limit: int, *, statement		self, snapshot_id: Sha1Git, from_: bytes, limit: int, *, statement
) -> Iterable[SnapshotBranchRow]:		) -> Iterable[SnapshotBranchRow]:
return map(		return map(
SnapshotBranchRow.from_dict,		SnapshotBranchRow.from_dict,
		vlorentzUnsubmitted Not Done Inline Actions doesn't work if the last byte of the prefix is `\xff`. You'll need to either implement the carry, or convert to an int vlorentz: doesn't work if the last byte of the prefix is `\xff`. You'll need to either implement the…
		vlorentzUnsubmitted Not Done Inline Actions and you'll also need to handle the issue of prefixes that are made entirely of `\xff` :/ vlorentz: and you'll also need to handle the issue of prefixes that are made entirely of `\xff` :/
self._execute_with_retries(statement, [snapshot_id, from_, limit]),		self._execute_with_retries(statement, [snapshot_id, from_, limit]),
)		)

##########################		##########################
# 'origin' table		# 'origin' table
##########################		##########################

@_prepared_insert_statement(OriginRow)		@_prepared_insert_statement(OriginRow)
▲ Show 20 Lines • Show All 334 Lines • Show Last 20 Lines

View Options

swh/storage/cassandra/storage.py

Show First 20 Lines • Show All 658 Lines • ▼ Show 20 Lines	def snapshot_get(self, snapshot_id: Sha1Git) -> Optional[Dict[str, Any]]:
"branches": {		"branches": {
name: branch.to_dict() if branch else None		name: branch.to_dict() if branch else None
for (name, branch) in d["branches"].items()		for (name, branch) in d["branches"].items()
},		},
"next_branch": d["next_branch"],		"next_branch": d["next_branch"],
}		}

def snapshot_count_branches(		def snapshot_count_branches(
self, snapshot_id: Sha1Git		self, snapshot_id: Sha1Git, branch_name_exclude_prefix: Optional[bytes] = None,
) -> Optional[Dict[Optional[str], int]]:		) -> Optional[Dict[Optional[str], int]]:
if self._cql_runner.snapshot_missing([snapshot_id]):		if self._cql_runner.snapshot_missing([snapshot_id]):
# Makes sure we don't fetch branches for a snapshot that is		# Makes sure we don't fetch branches for a snapshot that is
# being added.		# being added.
return None		return None
return self._cql_runner.snapshot_count_branches(snapshot_id)
		return self._cql_runner.snapshot_count_branches(
		snapshot_id, branch_name_exclude_prefix
		)

def snapshot_get_branches(		def snapshot_get_branches(
self,		self,
snapshot_id: Sha1Git,		snapshot_id: Sha1Git,
branches_from: bytes = b"",		branches_from: bytes = b"",
branches_count: int = 1000,		branches_count: int = 1000,
target_types: Optional[List[str]] = None,		target_types: Optional[List[str]] = None,
branch_name_include_pattern: Optional[bytes] = None,		branch_name_include_pattern: Optional[bytes] = None,
▲ Show 20 Lines • Show All 670 Lines • Show Last 20 Lines

View Options

swh/storage/in_memory.py

Show First 20 Lines • Show All 380 Lines • ▼ Show 20 Lines	class InMemoryCqlRunner:

##########################		##########################
# 'snapshot_branch' table		# 'snapshot_branch' table
##########################		##########################

def snapshot_branch_add_one(self, branch: SnapshotBranchRow) -> None:		def snapshot_branch_add_one(self, branch: SnapshotBranchRow) -> None:
self._snapshot_branches.insert(branch)		self._snapshot_branches.insert(branch)

def snapshot_count_branches(self, snapshot_id: Sha1Git) -> Dict[Optional[str], int]:		def snapshot_count_branches(
		self, snapshot_id: Sha1Git, branch_name_exclude_prefix: Optional[bytes] = None,
		) -> Dict[Optional[str], int]:
"""Returns a dictionary from type names to the number of branches		"""Returns a dictionary from type names to the number of branches
of that type."""		of that type."""
counts: Dict[Optional[str], int] = defaultdict(int)		counts: Dict[Optional[str], int] = defaultdict(int)
for branch in self._snapshot_branches.get_from_partition_key((snapshot_id,)):		for branch in self._snapshot_branches.get_from_partition_key((snapshot_id,)):
		if branch_name_exclude_prefix and branch.name.startswith(
		branch_name_exclude_prefix
		):
		continue
if branch.target_type is None:		if branch.target_type is None:
target_type = None		target_type = None
else:		else:
target_type = branch.target_type		target_type = branch.target_type
counts[target_type] += 1		counts[target_type] += 1
return counts		return counts

def snapshot_branch_get(		def snapshot_branch_get(
▲ Show 20 Lines • Show All 227 Lines • Show Last 20 Lines

View Options

swh/storage/interface.py

View Options

swh/storage/postgresql/db.py

View Options

swh/storage/postgresql/storage.py

View Options

swh/storage/sql/30-schema.sql

View Options

swh/storage/sql/40-funcs.sql

View Options

swh/storage/tests/storage_tests.py

storage: Allow to filter out branches by prefix when counting them
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D5208 (id=18658)

Changes applied before test

Patch application report for D5208 (id=18660)

Changes applied before test

Patch application report for D5208 (id=18674)

Changes applied before test

Patch application report for D5208 (id=18677)

Changes applied before test

Patch application report for D5208 (id=18750)

Changes applied before test

Patch application report for D5208 (id=18755)

Changes applied before test

Patch application report for D5208 (id=18781)

Changes applied before test

Patch application report for D5208 (id=18783)

Changes applied before test

Revision Contents
Changeset List

Diff 18677

sql/upgrades/169.sql

swh/storage/cassandra/cql.py

swh/storage/cassandra/storage.py

swh/storage/in_memory.py

swh/storage/interface.py

swh/storage/postgresql/db.py

swh/storage/postgresql/storage.py

swh/storage/sql/30-schema.sql

swh/storage/sql/40-funcs.sql

swh/storage/tests/storage_tests.py

storage: Allow to filter out branches by prefix when counting themClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D5208 (id=18658)

Changes applied before test

Patch application report for D5208 (id=18660)

Changes applied before test

Patch application report for D5208 (id=18674)

Changes applied before test

Patch application report for D5208 (id=18677)

Changes applied before test

Patch application report for D5208 (id=18750)

Changes applied before test

Patch application report for D5208 (id=18755)

Changes applied before test

Patch application report for D5208 (id=18781)

Changes applied before test

Patch application report for D5208 (id=18783)

Changes applied before test

Revision ContentsChangeset List

Diff 18677

sql/upgrades/169.sql

swh/storage/cassandra/cql.py

swh/storage/cassandra/storage.py

swh/storage/in_memory.py

swh/storage/interface.py

swh/storage/postgresql/db.py

swh/storage/postgresql/storage.py

swh/storage/sql/30-schema.sql

swh/storage/sql/40-funcs.sql

swh/storage/tests/storage_tests.py

storage: Allow to filter out branches by prefix when counting them
ClosedPublic
Actions

Revision Contents
Changeset List