Page MenuHomeSoftware Heritage
Differential D7756

search: Allow to pass empty pattern when searching for origins
ClosedPublic
Actions

Authored by anlambert on May 6 2022, 2:11 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDWAPPS96cc635cab25: search: Allow to pass empty pattern when searching for origins
Summary

It enables to search for all origins or for all origins with a
specific visit type.

The main purpose here is to allow searching for origins with a
specific visit type without having to specify a URL pattern.

Depends on D7755

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.May 6 2022, 2:11 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 6 2022, 2:11 PM
anlambert mentioned this in D7755: origin_search: Allow to pass empty query or URL pattern parameters.May 6 2022, 2:12 PM
swh-public-ci added a comment.May 6 2022, 2:39 PM

Build has FAILED

Patch application report for D7756 (id=28063)

Rebasing onto 54e679c461...

Current branch diff-target is up to date.
Changes applied before test
commit 0b65901ff74819860477749962601da8035ef7cd
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu May 5 10:38:19 2022 +0200

    search: Allow to pass empty pattern when searching for origins
    
    It enables to search for all origins or for all origins with a
    specific visit type.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1790/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1790/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 6 2022, 2:39 PM
Harbormaster failed remote builds in B29134: Diff 28063!
anlambert updated this revision to Diff 28069.May 6 2022, 2:57 PM

Rebase

swh-public-ci added a comment.May 6 2022, 3:20 PM

Build is green

Patch application report for D7756 (id=28069)

Rebasing onto e6a8303eef...

Current branch diff-target is up to date.
Changes applied before test
commit 96cc635cab25f57cbd83689cd4b2395c2b1e4af7
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu May 5 10:38:19 2022 +0200

    search: Allow to pass empty pattern when searching for origins
    
    It enables to search for all origins or for all origins with a
    specific visit type.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1794/ for more details.

Harbormaster completed remote builds in B29140: Diff 28069.May 6 2022, 3:20 PM
anlambert requested review of this revision.May 6 2022, 3:20 PM
vlorentz accepted this revision.May 6 2022, 7:39 PM
This revision is now accepted and ready to land.May 6 2022, 7:39 PM
Closed by commit rDWAPPS96cc635cab25: search: Allow to pass empty pattern when searching for origins (authored by anlambert). · Explain WhyMay 9 2022, 10:49 AM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPS96cc635cab25: search: Allow to pass empty pattern when searching for origins.

Revision Contents
Changeset List

PathSize
assets/
src/
bundles/
browse/
17 lines
cypress/
integration/
45 lines
swh/
web/
api/
views/
2 lines
templates/
includes/
2 lines
tests/
api/
views/
43 lines

Diff 28094

View Options

assets/src/bundles/browse/origin-search.js

Show All 27 Lines def task_dict(self, origin_type, origin_url, **kwargs):
This is overridden from the lister_base as more information is This is overridden from the lister_base as more information is
needed for the ingestion task creation. needed for the ingestion task creation.
This creates tasks with args and kwargs set, for example: This creates tasks with args and kwargs set, for example:
.. code-block:: python .. code-block:: python
args: ['https://ftp.gnu.org/gnu/3dldf/'] args:
kwargs: { kwargs: {
'tarballs': [{ 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'archive': 'https://...', 'artifacts': [{
'time': 1071002600, 'url': 'https://...',
'length': 128}, 'time': '2003-12-09T21:43:20+00:00',
'length': 128,
'version': '1.0.1',
'filename': 'something-1.0.1.tar.gz',
},
... ...
]} ]
}
""" """
tarballs = self.gnu_tree.artifacts[origin_url] artifacts = self.gnu_tree.artifacts[origin_url]
return utils.create_task_dict( return utils.create_task_dict(
'load-%s' % origin_type, 'load-%s' % origin_type,
kwargs.get('policy', 'oneshot'), kwargs.get('policy', 'oneshot'),
origin_url, url=origin_url,
tarballs=tarballs) artifacts=artifacts)
def safely_issue_request(self, identifier): def safely_issue_request(self, identifier):
"""Bypass the implementation. It's now the GNUTree which deals with """Bypass the implementation. It's now the GNUTree which deals with
querying the gnu mirror. querying the gnu mirror.
As an implementation detail, we cannot change simply the base As an implementation detail, we cannot change simply the base
SimpleLister as other implementation still uses it. This shall be part SimpleLister as other implementation still uses it. This shall be part
of another refactoring pass. of another refactoring pass.
""" """
return None return None
def list_packages(self, response): def list_packages(self, response):
"""List the actual gnu origins (package name) with their name, url and """List the actual gnu origins (package name) with their name, url and
associated tarballs. associated tarballs.
Args: Args:
response: Unused response: Unused
Returns: Returns:
List of packages name, url, last modification time List of packages name, url, last modification time
.. code-block:: python .. code-block:: python
[ [
{'name': '3dldf', {'name': '3dldf',
'url': 'https://ftp.gnu.org/gnu/3dldf/', 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'time_modified': 1071002600}, 'time_modified': '2003-12-09T20:43:20+00:00'},
{'name': '8sync', {'name': '8sync',
'url': 'https://ftp.gnu.org/gnu/8sync/', 'url': 'https://ftp.gnu.org/gnu/8sync/',
'time_modified': 1480991830}, 'time_modified': '2016-12-06T02:37:10+00:00'},
... ...
] ]
""" """
return list(self.gnu_tree.projects.values()) return list(self.gnu_tree.projects.values())
def get_model_from_repo(self, repo): def get_model_from_repo(self, repo):
"""Transform from repository representation to model """Transform from repository representation to model
""" """
return { return {
'uid': repo['name'], 'uid': repo['name'],
'name': repo['name'], 'name': repo['name'],
'full_name': repo['name'], 'full_name': repo['name'],
'html_url': repo['url'], 'html_url': repo['url'],
'origin_url': repo['url'], 'origin_url': repo['url'],
'time_last_updated': int(repo['time_modified']), 'time_last_updated': repo['time_modified'],
'origin_type': 'tar', 'origin_type': 'tar',
} }
View Options

cypress/integration/origin-search.spec.js

Show All 27 Lines def task_dict(self, origin_type, origin_url, **kwargs):
This is overridden from the lister_base as more information is This is overridden from the lister_base as more information is
needed for the ingestion task creation. needed for the ingestion task creation.
This creates tasks with args and kwargs set, for example: This creates tasks with args and kwargs set, for example:
.. code-block:: python .. code-block:: python
args: ['https://ftp.gnu.org/gnu/3dldf/'] args:
kwargs: { kwargs: {
'tarballs': [{ 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'archive': 'https://...', 'artifacts': [{
'time': 1071002600, 'url': 'https://...',
'length': 128}, 'time': '2003-12-09T21:43:20+00:00',
'length': 128,
'version': '1.0.1',
'filename': 'something-1.0.1.tar.gz',
},
... ...
]} ]
}
""" """
tarballs = self.gnu_tree.artifacts[origin_url] artifacts = self.gnu_tree.artifacts[origin_url]
return utils.create_task_dict( return utils.create_task_dict(
'load-%s' % origin_type, 'load-%s' % origin_type,
kwargs.get('policy', 'oneshot'), kwargs.get('policy', 'oneshot'),
origin_url, url=origin_url,
tarballs=tarballs) artifacts=artifacts)
def safely_issue_request(self, identifier): def safely_issue_request(self, identifier):
"""Bypass the implementation. It's now the GNUTree which deals with """Bypass the implementation. It's now the GNUTree which deals with
querying the gnu mirror. querying the gnu mirror.
As an implementation detail, we cannot change simply the base As an implementation detail, we cannot change simply the base
SimpleLister as other implementation still uses it. This shall be part SimpleLister as other implementation still uses it. This shall be part
of another refactoring pass. of another refactoring pass.
""" """
return None return None
def list_packages(self, response): def list_packages(self, response):
"""List the actual gnu origins (package name) with their name, url and """List the actual gnu origins (package name) with their name, url and
associated tarballs. associated tarballs.
Args: Args:
response: Unused response: Unused
Returns: Returns:
List of packages name, url, last modification time List of packages name, url, last modification time
.. code-block:: python .. code-block:: python
[ [
{'name': '3dldf', {'name': '3dldf',
'url': 'https://ftp.gnu.org/gnu/3dldf/', 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'time_modified': 1071002600}, 'time_modified': '2003-12-09T20:43:20+00:00'},
{'name': '8sync', {'name': '8sync',
'url': 'https://ftp.gnu.org/gnu/8sync/', 'url': 'https://ftp.gnu.org/gnu/8sync/',
'time_modified': 1480991830}, 'time_modified': '2016-12-06T02:37:10+00:00'},
... ...
] ]
""" """
return list(self.gnu_tree.projects.values()) return list(self.gnu_tree.projects.values())
def get_model_from_repo(self, repo): def get_model_from_repo(self, repo):
"""Transform from repository representation to model """Transform from repository representation to model
""" """
return { return {
'uid': repo['name'], 'uid': repo['name'],
'name': repo['name'], 'name': repo['name'],
'full_name': repo['name'], 'full_name': repo['name'],
'html_url': repo['url'], 'html_url': repo['url'],
'origin_url': repo['url'], 'origin_url': repo['url'],
'time_last_updated': int(repo['time_modified']), 'time_last_updated': repo['time_modified'],
'origin_type': 'tar', 'origin_type': 'tar',
} }
View Options

swh/web/api/views/origin.py

Show All 27 Lines def task_dict(self, origin_type, origin_url, **kwargs):
This is overridden from the lister_base as more information is This is overridden from the lister_base as more information is
needed for the ingestion task creation. needed for the ingestion task creation.
This creates tasks with args and kwargs set, for example: This creates tasks with args and kwargs set, for example:
.. code-block:: python .. code-block:: python
args: ['https://ftp.gnu.org/gnu/3dldf/'] args:
kwargs: { kwargs: {
'tarballs': [{ 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'archive': 'https://...', 'artifacts': [{
'time': 1071002600, 'url': 'https://...',
'length': 128}, 'time': '2003-12-09T21:43:20+00:00',
'length': 128,
'version': '1.0.1',
'filename': 'something-1.0.1.tar.gz',
},
... ...
]} ]
}
""" """
tarballs = self.gnu_tree.artifacts[origin_url] artifacts = self.gnu_tree.artifacts[origin_url]
return utils.create_task_dict( return utils.create_task_dict(
'load-%s' % origin_type, 'load-%s' % origin_type,
kwargs.get('policy', 'oneshot'), kwargs.get('policy', 'oneshot'),
origin_url, url=origin_url,
tarballs=tarballs) artifacts=artifacts)
def safely_issue_request(self, identifier): def safely_issue_request(self, identifier):
"""Bypass the implementation. It's now the GNUTree which deals with """Bypass the implementation. It's now the GNUTree which deals with
querying the gnu mirror. querying the gnu mirror.
As an implementation detail, we cannot change simply the base As an implementation detail, we cannot change simply the base
SimpleLister as other implementation still uses it. This shall be part SimpleLister as other implementation still uses it. This shall be part
of another refactoring pass. of another refactoring pass.
""" """
return None return None
def list_packages(self, response): def list_packages(self, response):
"""List the actual gnu origins (package name) with their name, url and """List the actual gnu origins (package name) with their name, url and
associated tarballs. associated tarballs.
Args: Args:
response: Unused response: Unused
Returns: Returns:
List of packages name, url, last modification time List of packages name, url, last modification time
.. code-block:: python .. code-block:: python
[ [
{'name': '3dldf', {'name': '3dldf',
'url': 'https://ftp.gnu.org/gnu/3dldf/', 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'time_modified': 1071002600}, 'time_modified': '2003-12-09T20:43:20+00:00'},
{'name': '8sync', {'name': '8sync',
'url': 'https://ftp.gnu.org/gnu/8sync/', 'url': 'https://ftp.gnu.org/gnu/8sync/',
'time_modified': 1480991830}, 'time_modified': '2016-12-06T02:37:10+00:00'},
... ...
] ]
""" """
return list(self.gnu_tree.projects.values()) return list(self.gnu_tree.projects.values())
def get_model_from_repo(self, repo): def get_model_from_repo(self, repo):
"""Transform from repository representation to model """Transform from repository representation to model
""" """
return { return {
'uid': repo['name'], 'uid': repo['name'],
'name': repo['name'], 'name': repo['name'],
'full_name': repo['name'], 'full_name': repo['name'],
'html_url': repo['url'], 'html_url': repo['url'],
'origin_url': repo['url'], 'origin_url': repo['url'],
'time_last_updated': int(repo['time_modified']), 'time_last_updated': repo['time_modified'],
'origin_type': 'tar', 'origin_type': 'tar',
} }
View Options

swh/web/templates/includes/origin-search-form.html

Show All 27 Lines def task_dict(self, origin_type, origin_url, **kwargs):
This is overridden from the lister_base as more information is This is overridden from the lister_base as more information is
needed for the ingestion task creation. needed for the ingestion task creation.
This creates tasks with args and kwargs set, for example: This creates tasks with args and kwargs set, for example:
.. code-block:: python .. code-block:: python
args: ['https://ftp.gnu.org/gnu/3dldf/'] args:
kwargs: { kwargs: {
'tarballs': [{ 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'archive': 'https://...', 'artifacts': [{
'time': 1071002600, 'url': 'https://...',
'length': 128}, 'time': '2003-12-09T21:43:20+00:00',
'length': 128,
'version': '1.0.1',
'filename': 'something-1.0.1.tar.gz',
},
... ...
]} ]
}
""" """
tarballs = self.gnu_tree.artifacts[origin_url] artifacts = self.gnu_tree.artifacts[origin_url]
return utils.create_task_dict( return utils.create_task_dict(
'load-%s' % origin_type, 'load-%s' % origin_type,
kwargs.get('policy', 'oneshot'), kwargs.get('policy', 'oneshot'),
origin_url, url=origin_url,
tarballs=tarballs) artifacts=artifacts)
def safely_issue_request(self, identifier): def safely_issue_request(self, identifier):
"""Bypass the implementation. It's now the GNUTree which deals with """Bypass the implementation. It's now the GNUTree which deals with
querying the gnu mirror. querying the gnu mirror.
As an implementation detail, we cannot change simply the base As an implementation detail, we cannot change simply the base
SimpleLister as other implementation still uses it. This shall be part SimpleLister as other implementation still uses it. This shall be part
of another refactoring pass. of another refactoring pass.
""" """
return None return None
def list_packages(self, response): def list_packages(self, response):
"""List the actual gnu origins (package name) with their name, url and """List the actual gnu origins (package name) with their name, url and
associated tarballs. associated tarballs.
Args: Args:
response: Unused response: Unused
Returns: Returns:
List of packages name, url, last modification time List of packages name, url, last modification time
.. code-block:: python .. code-block:: python
[ [
{'name': '3dldf', {'name': '3dldf',
'url': 'https://ftp.gnu.org/gnu/3dldf/', 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'time_modified': 1071002600}, 'time_modified': '2003-12-09T20:43:20+00:00'},
{'name': '8sync', {'name': '8sync',
'url': 'https://ftp.gnu.org/gnu/8sync/', 'url': 'https://ftp.gnu.org/gnu/8sync/',
'time_modified': 1480991830}, 'time_modified': '2016-12-06T02:37:10+00:00'},
... ...
] ]
""" """
return list(self.gnu_tree.projects.values()) return list(self.gnu_tree.projects.values())
def get_model_from_repo(self, repo): def get_model_from_repo(self, repo):
"""Transform from repository representation to model """Transform from repository representation to model
""" """
return { return {
'uid': repo['name'], 'uid': repo['name'],
'name': repo['name'], 'name': repo['name'],
'full_name': repo['name'], 'full_name': repo['name'],
'html_url': repo['url'], 'html_url': repo['url'],
'origin_url': repo['url'], 'origin_url': repo['url'],
'time_last_updated': int(repo['time_modified']), 'time_last_updated': repo['time_modified'],
'origin_type': 'tar', 'origin_type': 'tar',
} }
View Options

swh/web/tests/api/views/test_origin.py

Show All 27 Lines def task_dict(self, origin_type, origin_url, **kwargs):
This is overridden from the lister_base as more information is This is overridden from the lister_base as more information is
needed for the ingestion task creation. needed for the ingestion task creation.
This creates tasks with args and kwargs set, for example: This creates tasks with args and kwargs set, for example:
.. code-block:: python .. code-block:: python
args: ['https://ftp.gnu.org/gnu/3dldf/'] args:
kwargs: { kwargs: {
'tarballs': [{ 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'archive': 'https://...', 'artifacts': [{
'time': 1071002600, 'url': 'https://...',
'length': 128}, 'time': '2003-12-09T21:43:20+00:00',
'length': 128,
'version': '1.0.1',
'filename': 'something-1.0.1.tar.gz',
},
... ...
]} ]
}
""" """
tarballs = self.gnu_tree.artifacts[origin_url] artifacts = self.gnu_tree.artifacts[origin_url]
return utils.create_task_dict( return utils.create_task_dict(
'load-%s' % origin_type, 'load-%s' % origin_type,
kwargs.get('policy', 'oneshot'), kwargs.get('policy', 'oneshot'),
origin_url, url=origin_url,
tarballs=tarballs) artifacts=artifacts)
def safely_issue_request(self, identifier): def safely_issue_request(self, identifier):
"""Bypass the implementation. It's now the GNUTree which deals with """Bypass the implementation. It's now the GNUTree which deals with
querying the gnu mirror. querying the gnu mirror.
As an implementation detail, we cannot change simply the base As an implementation detail, we cannot change simply the base
SimpleLister as other implementation still uses it. This shall be part SimpleLister as other implementation still uses it. This shall be part
of another refactoring pass. of another refactoring pass.
""" """
return None return None
def list_packages(self, response): def list_packages(self, response):
"""List the actual gnu origins (package name) with their name, url and """List the actual gnu origins (package name) with their name, url and
associated tarballs. associated tarballs.
Args: Args:
response: Unused response: Unused
Returns: Returns:
List of packages name, url, last modification time List of packages name, url, last modification time
.. code-block:: python .. code-block:: python
[ [
{'name': '3dldf', {'name': '3dldf',
'url': 'https://ftp.gnu.org/gnu/3dldf/', 'url': 'https://ftp.gnu.org/gnu/3dldf/',
'time_modified': 1071002600}, 'time_modified': '2003-12-09T20:43:20+00:00'},
{'name': '8sync', {'name': '8sync',
'url': 'https://ftp.gnu.org/gnu/8sync/', 'url': 'https://ftp.gnu.org/gnu/8sync/',
'time_modified': 1480991830}, 'time_modified': '2016-12-06T02:37:10+00:00'},
... ...
] ]
""" """
return list(self.gnu_tree.projects.values()) return list(self.gnu_tree.projects.values())
def get_model_from_repo(self, repo): def get_model_from_repo(self, repo):
"""Transform from repository representation to model """Transform from repository representation to model
""" """
return { return {
'uid': repo['name'], 'uid': repo['name'],
'name': repo['name'], 'name': repo['name'],
'full_name': repo['name'], 'full_name': repo['name'],
'html_url': repo['url'], 'html_url': repo['url'],
'origin_url': repo['url'], 'origin_url': repo['url'],
'time_last_updated': int(repo['time_modified']), 'time_last_updated': repo['time_modified'],
'origin_type': 'tar', 'origin_type': 'tar',
} }
About · Developer information · Legal