Enhance web app behavior when encountering snapshots with a large number of branches
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	olasd
	Sep 22 2018, 2:47 PM

Description

The web app is not really usable on origins that yield snapshots containing a large number of branches.

(Example of an extreme case: https://github.com/v8/v8 has 110k branches)

Right now, the snapshot handling will try to cache objects for all branches, however many there is, which times out in extreme cases such as v8. Even just looking at the snapshot through the API times out because of the quantity of data to process.

We'll need some storage backend adaptations (e.g. to provide a paginated/filtered snapshot view), and some frontend adaptations to use these.

Revisions and Commits

rDWAPPS Web applications
	rDWAPPS8e5af45cc06f swh-web: Improve querying / browsing of large snapshots

Related Objects

Mentioned In: rDSTOe9e5e73fe421: storage: Enable to paginate, filter and count snapshot content
D487: Storage: Enable to paginate, filter and count snapshot content
rDSTO5193e6d59017: Drop backwards-compatibility features between snapshots and occurrences
D466: Drop backwards-compatibility features between snapshots and occurrences
T1221: Migrate away from revision_get_by / revision_log_by

Event Timeline

olasd triaged this task as Normal priority.Sep 22 2018, 2:47 PM

olasd created this task.

in terms of UX, I think this should do something like:

show the default branch (HEAD) if no explicit branch has been requested, some branches (according to whatever pagination is implemented by the back-end) and tell the user "sorry, this snapshot has too many branches to be shown…". GitHub does something similar for very large dirs, and it's totally fine. (BTW, do we have something similar for very large dirs? we should!)
if an explicit branch has been requested, show that one instead of the default one, and do as above for everything else

olasd mentioned this in T1221: Migrate away from revision_get_by / revision_log_by.Oct 2 2018, 10:36 AM

anlambert changed the task status from Open to Work in Progress.Oct 3 2018, 10:45 AM

anlambert claimed this task.

olasd mentioned this in D466: Drop backwards-compatibility features between snapshots and occurrences.Oct 3 2018, 5:30 PM

olasd mentioned this in rDSTO5193e6d59017: Drop backwards-compatibility features between snapshots and occurrences.Oct 3 2018, 6:18 PM

anlambert mentioned this in D487: Storage: Enable to paginate, filter and count snapshot content.Oct 4 2018, 6:00 PM

anlambert mentioned this in rDSTOe9e5e73fe421: storage: Enable to paginate, filter and count snapshot content.Oct 10 2018, 2:43 PM

anlambert closed this task as Resolved by committing rDWAPPS8e5af45cc06f: swh-web: Improve querying / browsing of large snapshots.Oct 16 2018, 6:03 PM

anlambert added a commit: rDWAPPS8e5af45cc06f: swh-web: Improve querying / browsing of large snapshots.