Page MenuHomeSoftware Heritage
Differential D6691

common/archive: Add lookup_origin_snapshots function
ClosedPublic
Actions

Authored by anlambert on Nov 25 2021, 4:01 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3631: Make snapshot query parameter mandatory when browsing objects in an origin context
Commits
rDWAPPSd7f3a3a4f8ce: common/archive: Add lookup_origin_snapshots function
Summary

This wraps the call to origin_snapshot_get_all method from the
storage interface.

Related to T3631

Diff Detail

Repository
rDWAPPS Web applications
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 25217
Build 39408: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 39407: arc lint + arc unit

Event Timeline

anlambert created this revision.Nov 25 2021, 4:01 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 25 2021, 4:01 PM
swh-public-ci added a comment.Nov 25 2021, 4:19 PM

Build is green

Patch application report for D6691 (id=24305)

Rebasing onto 20a9d18559...

Current branch diff-target is up to date.
Changes applied before test
commit c91fc8e02848ff3852d0bc76dfa0963157eff9d9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 25 16:00:39 2021 +0100

    common/archive: Add lookup_origin_snapshots function
    
    This wraps the call to origin_snapshot_get_all method from the
    storage interface.
    
    Related to T3631

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1245/ for more details.

Harbormaster completed remote builds in B25175: Diff 24305.Nov 25 2021, 4:19 PM
anlambert requested review of this revision.Nov 25 2021, 4:19 PM
vlorentz added a subscriber: vlorentz.Nov 26 2021, 1:32 PM

Where is it called from?

anlambert added a comment.Nov 26 2021, 1:36 PM
In D6691#174048, @vlorentz wrote:

Where is it called from?

I will use it in a future diff to check if a snapshot is associated to an origin.

vlorentz accepted this revision.Nov 29 2021, 10:55 AM
vlorentz added inline comments.
swh/web/common/archive.py
298
This revision is now accepted and ready to land.Nov 29 2021, 10:55 AM
anlambert updated this revision to Diff 24341.Nov 29 2021, 10:59 AM

Update docstring

swh-public-ci added a comment.Nov 29 2021, 11:22 AM

Build is green

Patch application report for D6691 (id=24341)

Rebasing onto 806623eb53...

First, rewinding head to replay your work on top of it...
Applying: common/archive: Add lookup_origin_snapshots function
Changes applied before test
commit 26ae6c0ad2acfdbe214eae3b212b03640b5e5663
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 25 16:00:39 2021 +0100

    common/archive: Add lookup_origin_snapshots function
    
    This wraps the call to origin_snapshot_get_all method from the
    storage interface.
    
    Related to T3631

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1249/ for more details.

Harbormaster completed remote builds in B25215: Diff 24341.Nov 29 2021, 11:22 AM
anlambert updated this revision to Diff 24343.Nov 29 2021, 11:27 AM

Rebase

This revision was landed with ongoing or failed builds.Nov 29 2021, 11:48 AM
Closed by commit rDWAPPSd7f3a3a4f8ce: common/archive: Add lookup_origin_snapshots function (authored by anlambert). · Explain Why
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPSd7f3a3a4f8ce: common/archive: Add lookup_origin_snapshots function.
swh-public-ci added a comment.Nov 29 2021, 11:53 AM

Build was aborted

Patch application report for D6691 (id=24343)

Rebasing onto 806623eb53...

Current branch diff-target is up to date.
Changes applied before test
commit d7f3a3a4f8ce9b55f1b06aeea92b883126a2915d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 25 16:00:39 2021 +0100

    common/archive: Add lookup_origin_snapshots function
    
    This wraps the call to origin_snapshot_get_all method from the
    storage interface.
    
    Related to T3631

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1250/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1250/console

Harbormaster failed remote builds in B25217: Diff 24343!Nov 29 2021, 11:53 AM

Revision Contents
Changeset List

PathSize
swh/
web/
common/
15 lines
tests/
common/
8 lines

Diff 24343

View Options

swh/web/common/archive.py

  • This file was added.
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 2,
"id": 18,
"iteration": 1551112370226,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"fill": 1,
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 0
},
"id": 4,
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": true,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/uncaught/",
"color": "#bf1b00"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(rate(swh_task_called_count{worker=~\"$worker\"}[$interval])) by (task)",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{task}}",
"refId": "A"
},
{
"expr": "sum(rate(swh_task_failure_count{worker=~\"$worker\"}[$interval])) by (task)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{task}} uncaught exceptions",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Task counts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "short",
"label": "tasks per second",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"cards": {
"cardPadding": null,
"cardRound": null
},
"color": {
"cardColor": "#c15c17",
"colorScale": "sqrt",
"colorScheme": "interpolateViridis",
"exponent": 0.5,
"mode": "spectrum"
},
"dataFormat": "tsbuckets",
"datasource": "Prometheus",
"description": "Each square's color represents the number of tasks that completed within that duration range.",
"gridPos": {
"h": 14,
"w": 24,
"x": 0,
"y": 9
},
"heatmap": {},
"hideTimeOverride": false,
"highlightCards": true,
"id": 2,
"legend": {
"show": true
},
"links": [],
"repeat": "task",
"repeatDirection": "v",
"scopedVars": {
"task": {
"selected": true,
"text": "swh.loader.git.tasks.UpdateGitRepository",
"value": "swh.loader.git.tasks.UpdateGitRepository"
}
},
"targets": [
{
"expr": "sum(increase(swh_task_duration_seconds_bucket{task=~\"$task\",worker=~\"$worker\"}[$interval])) by (le)",
"format": "heatmap",
"instant": false,
"interval": "$interval",
"intervalFactor": 1,
"legendFormat": "{{le}}",
"refId": "A"
}
],
"title": "$task durations",
"tooltip": {
"show": true,
"showHistogram": true
},
"tooltipDecimals": null,
"transparent": false,
"type": "heatmap",
"xAxis": {
"show": true
},
"xBucketNumber": null,
"xBucketSize": null,
"yAxis": {
"decimals": 0,
"format": "s",
"logBase": 1,
"max": null,
"min": null,
"show": true,
"splitFactor": null
},
"yBucketBound": "upper",
"yBucketNumber": null,
"yBucketSize": null
}
],
"refresh": false,
"schemaVersion": 16,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"auto": true,
"auto_count": 100,
"auto_min": "2m",
"current": {
"text": "auto",
"value": "$__auto_interval_interval"
},
"hide": 0,
"label": "Interval",
"name": "interval",
"options": [
{
"selected": true,
"text": "auto",
"value": "$__auto_interval_interval"
},
{
"selected": false,
"text": "2m",
"value": "2m"
},
{
"selected": false,
"text": "5m",
"value": "5m"
},
{
"selected": false,
"text": "10m",
"value": "10m"
},
{
"selected": false,
"text": "30m",
"value": "30m"
},
{
"selected": false,
"text": "1h",
"value": "1h"
},
{
"selected": false,
"text": "6h",
"value": "6h"
},
{
"selected": false,
"text": "12h",
"value": "12h"
},
{
"selected": false,
"text": "1d",
"value": "1d"
},
{
"selected": false,
"text": "7d",
"value": "7d"
},
{
"selected": false,
"text": "14d",
"value": "14d"
},
{
"selected": false,
"text": "30d",
"value": "30d"
}
],
"query": "2m,5m,10m,30m,1h,6h,12h,1d,7d,14d,30d",
"refresh": 2,
"skipUrlSync": false,
"type": "interval"
},
{
"allValue": null,
"current": {
"text": "swh.loader.git.tasks.UpdateGitRepository",
"value": "swh.loader.git.tasks.UpdateGitRepository"
},
"datasource": "Prometheus",
"definition": "label_values(swh_task_called_count, task)",
"hide": 0,
"includeAll": true,
"label": "Task name",
"multi": false,
"name": "task",
"options": [],
"query": "label_values(swh_task_called_count, task)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "All",
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(swh_task_called_count{task=~\"$task\"}, worker)",
"hide": 0,
"includeAll": true,
"label": "Worker",
"multi": true,
"name": "worker",
"options": [],
"query": "label_values(swh_task_called_count{task=~\"$task\"}, worker)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Worker task processing",
"uid": "b_xh3f9ik",
"version": 8
}
View Options

swh/web/tests/common/test_archive.py

  • This file was added.
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 2,
"id": 18,
"iteration": 1551112370226,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"fill": 1,
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 0
},
"id": 4,
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": true,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/uncaught/",
"color": "#bf1b00"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(rate(swh_task_called_count{worker=~\"$worker\"}[$interval])) by (task)",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{task}}",
"refId": "A"
},
{
"expr": "sum(rate(swh_task_failure_count{worker=~\"$worker\"}[$interval])) by (task)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{task}} uncaught exceptions",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Task counts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "short",
"label": "tasks per second",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"cards": {
"cardPadding": null,
"cardRound": null
},
"color": {
"cardColor": "#c15c17",
"colorScale": "sqrt",
"colorScheme": "interpolateViridis",
"exponent": 0.5,
"mode": "spectrum"
},
"dataFormat": "tsbuckets",
"datasource": "Prometheus",
"description": "Each square's color represents the number of tasks that completed within that duration range.",
"gridPos": {
"h": 14,
"w": 24,
"x": 0,
"y": 9
},
"heatmap": {},
"hideTimeOverride": false,
"highlightCards": true,
"id": 2,
"legend": {
"show": true
},
"links": [],
"repeat": "task",
"repeatDirection": "v",
"scopedVars": {
"task": {
"selected": true,
"text": "swh.loader.git.tasks.UpdateGitRepository",
"value": "swh.loader.git.tasks.UpdateGitRepository"
}
},
"targets": [
{
"expr": "sum(increase(swh_task_duration_seconds_bucket{task=~\"$task\",worker=~\"$worker\"}[$interval])) by (le)",
"format": "heatmap",
"instant": false,
"interval": "$interval",
"intervalFactor": 1,
"legendFormat": "{{le}}",
"refId": "A"
}
],
"title": "$task durations",
"tooltip": {
"show": true,
"showHistogram": true
},
"tooltipDecimals": null,
"transparent": false,
"type": "heatmap",
"xAxis": {
"show": true
},
"xBucketNumber": null,
"xBucketSize": null,
"yAxis": {
"decimals": 0,
"format": "s",
"logBase": 1,
"max": null,
"min": null,
"show": true,
"splitFactor": null
},
"yBucketBound": "upper",
"yBucketNumber": null,
"yBucketSize": null
}
],
"refresh": false,
"schemaVersion": 16,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"auto": true,
"auto_count": 100,
"auto_min": "2m",
"current": {
"text": "auto",
"value": "$__auto_interval_interval"
},
"hide": 0,
"label": "Interval",
"name": "interval",
"options": [
{
"selected": true,
"text": "auto",
"value": "$__auto_interval_interval"
},
{
"selected": false,
"text": "2m",
"value": "2m"
},
{
"selected": false,
"text": "5m",
"value": "5m"
},
{
"selected": false,
"text": "10m",
"value": "10m"
},
{
"selected": false,
"text": "30m",
"value": "30m"
},
{
"selected": false,
"text": "1h",
"value": "1h"
},
{
"selected": false,
"text": "6h",
"value": "6h"
},
{
"selected": false,
"text": "12h",
"value": "12h"
},
{
"selected": false,
"text": "1d",
"value": "1d"
},
{
"selected": false,
"text": "7d",
"value": "7d"
},
{
"selected": false,
"text": "14d",
"value": "14d"
},
{
"selected": false,
"text": "30d",
"value": "30d"
}
],
"query": "2m,5m,10m,30m,1h,6h,12h,1d,7d,14d,30d",
"refresh": 2,
"skipUrlSync": false,
"type": "interval"
},
{
"allValue": null,
"current": {
"text": "swh.loader.git.tasks.UpdateGitRepository",
"value": "swh.loader.git.tasks.UpdateGitRepository"
},
"datasource": "Prometheus",
"definition": "label_values(swh_task_called_count, task)",
"hide": 0,
"includeAll": true,
"label": "Task name",
"multi": false,
"name": "task",
"options": [],
"query": "label_values(swh_task_called_count, task)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "All",
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(swh_task_called_count{task=~\"$task\"}, worker)",
"hide": 0,
"includeAll": true,
"label": "Worker",
"multi": true,
"name": "worker",
"options": [],
"query": "label_values(swh_task_called_count{task=~\"$task\"}, worker)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Worker task processing",
"uid": "b_xh3f9ik",
"version": 8
}
About · Developer information · Legal