Page MenuHomeSoftware Heritage
Differential D6518

admin/deposit: Allow users with permission to list their deposits
ClosedPublic
Actions

Authored by anlambert on Oct 20 2021, 2:01 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T3174: Filter deposit-admin view by deposit client on admin (moderation) page
Commits
rDWAPPSa8cfc14b4a5c: admin/deposit: Allow users with permission to list their deposits
Summary

Users with the "swh.web.admin.list_deposits" permission (deposit clients)
will be able to access the deposits administration interface.

Only their own deposits will be displayed.

Related to T3174

Test Plan

Tests for links and deposits admin view access have been added.

Deposits admin view access for deposit clients have also been successfully tested in docker.

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Oct 20 2021, 2:01 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 20 2021, 2:01 PM
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 20 2021, 2:29 PM
Harbormaster failed remote builds in B24577: Diff 23688!
ardumont added a subscriber: ardumont.EditedOct 20 2021, 2:31 PM

build was not running, i stopped and triggered another one...

nvm that, that's a web build not a deposit one ¯\_(ツ)_/¯

swh-public-ci added a comment.Oct 20 2021, 2:40 PM

Build was aborted

Patch application report for D6518 (id=23688)

Rebasing onto ba1801812e...

Current branch diff-target is up to date.
Changes applied before test
commit a8cfc14b4a5cae4373f9a67725de3e6bd4cad77d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Oct 19 15:49:03 2021 +0200

    admin/deposit: Allow users with permission to list their deposits
    
    Users with the "swh.web.admin.list_deposits" permission (deposit clients)
    will be able to access the deposits administration interface.
    
    Only their own deposits will be displayed.
    
    Related to T3174

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1197/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1197/console

swh-public-ci added a comment.Oct 20 2021, 2:46 PM

Build is green

Patch application report for D6518 (id=23688)

Rebasing onto ba1801812e...

Current branch diff-target is up to date.
Changes applied before test
commit a8cfc14b4a5cae4373f9a67725de3e6bd4cad77d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Oct 19 15:49:03 2021 +0200

    admin/deposit: Allow users with permission to list their deposits
    
    Users with the "swh.web.admin.list_deposits" permission (deposit clients)
    will be able to access the deposits administration interface.
    
    Only their own deposits will be displayed.
    
    Related to T3174

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1198/ for more details.

swh-public-ci added a comment.Oct 20 2021, 2:57 PM

Build is green

Patch application report for D6518 (id=23688)

Rebasing onto ba1801812e...

Current branch diff-target is up to date.
Changes applied before test
commit a8cfc14b4a5cae4373f9a67725de3e6bd4cad77d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Oct 19 15:49:03 2021 +0200

    admin/deposit: Allow users with permission to list their deposits
    
    Users with the "swh.web.admin.list_deposits" permission (deposit clients)
    will be able to access the deposits administration interface.
    
    Only their own deposits will be displayed.
    
    Related to T3174

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1199/ for more details.

Harbormaster completed remote builds in B24577: Diff 23688.Oct 20 2021, 2:57 PM
anlambert requested review of this revision.Oct 20 2021, 2:57 PM
ardumont accepted this revision.Oct 20 2021, 3:01 PM

lgtm

swh/web/admin/deposit.py
24

did not know that decorator, how neat ;)

This revision is now accepted and ready to land.Oct 20 2021, 3:01 PM
Closed by commit rDWAPPSa8cfc14b4a5c: admin/deposit: Allow users with permission to list their deposits (authored by anlambert). · Explain WhyOct 20 2021, 4:05 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPSa8cfc14b4a5c: admin/deposit: Allow users with permission to list their deposits.

Revision Contents
Changeset List

PathSize
assets/
src/
bundles/
admin/
7 lines
swh/
web/
admin/
13 lines
auth/
1 line
common/
32 lines
templates/
admin/
4 lines
16 lines
tests/
admin/
37 lines
30 lines

Diff 23700

View Options

assets/src/bundles/admin/deposit.js

  • This file was added.
# Copyright (C) 2019 the Software Heritage developers
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import subprocess
import json
import logging
from swh.lister.rcran.models import RCRANModel
from swh.scheduler import utils
from swh.lister.core.simple_lister import SimpleLister
class RCRANLister(SimpleLister):
MODEL = RCRANModel
LISTER_NAME = 'rcran'
def __init__(self, override_config=None):
SimpleLister.__init__(self, override_config=override_config)
douarddaUnsubmitted
Done
Inline Actions

why implement the overloaded init if it does nothing more than calling super()?

douardda: why implement the overloaded __init__ if it does nothing more than calling super()?
def task_dict(self, origin_type, origin_url, **kwargs):
"""(Override) Return task format dict
This is overridden from the lister_base as more information is
needed for the ingestion task creation.
"""
_type = 'origin-update-%s' % origin_type
anlambertUnsubmitted
Done
Inline Actions

FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
This should be changed to:

_type = 'load-%s' % origin_type
anlambert: FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
nahimilegaAuthorUnsubmitted
Done
Inline Actions

Thanks for updating me about the change @anlambert. I made the recommended changes.

nahimilega: Thanks for updating me about the change @anlambert. I made the recommended changes.
_policy = 'recurring'
project_name = kwargs.get('name')
project_version = kwargs.get('version')
project_metadata = kwargs.get('description')
return utils.create_task_dict(
_type, _policy, project_name, origin_url, project_version,
project_metadata=project_metadata)
douarddaUnsubmitted
Done
Inline Actions

as I said in D1497 I don't like that kwargs.get is used to extract mandatory values.

Also don't see the need for those project_xxx variables. This method should be a one statement only.

douardda: as I said in D1497 I don't like that kwargs.get is used to extract mandatory values. Also…
def r_script_request(self):
"""(Override) Runs r script which uses inbuilt API to return a json
response containing data about all the R packages
Returns:
JSON response
"""
response = subprocess.getoutput("./list_all_the_packages.R")
douarddaUnsubmitted
Done
Inline Actions

This will not work. You assume you are running the command from the package directory. And it is a rather bad idea to call the script directly (rather than calling the Rscript tool with the path to the script as argument).

Yo need to figure out where this script will be shipped once the python package is installed and ensure you get it's proper path at execution time.

I'd also rather use check_output (without a shell) here. No need for a shell in the path.

douardda: This will not work. You assume you are running the command from the package directory. And it…
return json.loads(response)
def list_packages(self, response):
"""(Override) List the actual rcran origins from the response.
"""
pass
douarddaUnsubmitted
Done
Inline Actions

?

douardda: ?
def _compute_urls(self, repo):
"""Returns a tuple (project_url, project_metadata_url)
"""
return (
'https://cran.r-project.org/src/contrib/%s_%s.tar.gz' %
(repo["Package"], repo["Version"])
)
douarddaUnsubmitted
Done
Inline Actions

why do you need a method for this one liner?
Plus I don't like having URLs hardwritten in the middle of the code: it should be at least a constant of the class or configurable parameter which defaults to this URL.

Also, use named-based formatting here:

return 'https://cran.r-project.org/src/contrib/%(Package)s_%(Version)s.tar.gz' % repo
douardda: why do you need a method for this one liner? Plus I don't like having URLs hardwritten in the…
def get_model_from_repo(self, repo):
"""(Override) Transform from repository representation to model
"""
project_url = self._compute_urls(repo)
return {
'uid': repo["Package"],
'name': repo["Package"],
'full_name': repo["Title"],
'version': repo["Version"],
'html_url': project_url,
'origin_url': project_url,
'origin_type': 'rcran',
'description': repo["Description"]
}
def transport_response_simplified(self, response):
"""(Override) Transform response to list for model manipulation
"""
return [self.get_model_from_repo(repo_name) for repo_name in response]
def ingest_data(self, identifier, checks=False):
"""(Override)Rework the base ingest_data.
Request server endpoint which gives all in one go.
Simplify and filter response list of repositories. Inject
repo information into local db. Queue loader tasks for
linked repositories.
Args:
identifier: Resource identifier (unused)
checks (bool): Additional checks required (unused)
"""
response = self.r_script_request()
if not response:
return response, []
models_list = self.transport_response_simplified(response)
models_list = self.filter_before_inject(models_list)
all_injected = []
for models in utils.grouper(models_list, n=10000):
models = list(models)
logging.debug('models: %s' % len(models))
# inject into local db
injected = self.inject_repo_data_into_db(models)
# queue workers
self.create_missing_origins_and_tasks(models, injected)
all_injected.append(injected)
# flush
self.db_session.commit()
self.db_session = self.mk_session()
return response, all_injected
douarddaUnsubmitted
Done
Inline Actions

weird. why are these (NOP-like) methods needed?

douardda: weird. why are these (NOP-like) methods needed?
nahimilegaAuthorUnsubmitted
Done
Inline Actions

SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are needed to be present or else code will throw an error. However, the response is not achieved from conventional methods. Thus, there is no need for these functions.

nahimilega: SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are…
douarddaUnsubmitted
Not Done
Inline Actions

Yeah right... this code (I mean the base classes provided for Lister implementation) really need some serious refactoring...

douardda: Yeah right... this code (I mean the base classes provided for Lister implementation) really…
View Options

swh/web/admin/deposit.py

  • This file was added.
# Copyright (C) 2019 the Software Heritage developers
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import subprocess
import json
import logging
from swh.lister.rcran.models import RCRANModel
from swh.scheduler import utils
from swh.lister.core.simple_lister import SimpleLister
class RCRANLister(SimpleLister):
MODEL = RCRANModel
LISTER_NAME = 'rcran'
def __init__(self, override_config=None):
SimpleLister.__init__(self, override_config=override_config)
douarddaUnsubmitted
Done
Inline Actions

why implement the overloaded init if it does nothing more than calling super()?

douardda: why implement the overloaded __init__ if it does nothing more than calling super()?
def task_dict(self, origin_type, origin_url, **kwargs):
"""(Override) Return task format dict
This is overridden from the lister_base as more information is
needed for the ingestion task creation.
"""
_type = 'origin-update-%s' % origin_type
anlambertUnsubmitted
Done
Inline Actions

FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
This should be changed to:

_type = 'load-%s' % origin_type
anlambert: FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
nahimilegaAuthorUnsubmitted
Done
Inline Actions

Thanks for updating me about the change @anlambert. I made the recommended changes.

nahimilega: Thanks for updating me about the change @anlambert. I made the recommended changes.
_policy = 'recurring'
project_name = kwargs.get('name')
project_version = kwargs.get('version')
project_metadata = kwargs.get('description')
return utils.create_task_dict(
_type, _policy, project_name, origin_url, project_version,
project_metadata=project_metadata)
douarddaUnsubmitted
Done
Inline Actions

as I said in D1497 I don't like that kwargs.get is used to extract mandatory values.

Also don't see the need for those project_xxx variables. This method should be a one statement only.

douardda: as I said in D1497 I don't like that kwargs.get is used to extract mandatory values. Also…
def r_script_request(self):
"""(Override) Runs r script which uses inbuilt API to return a json
response containing data about all the R packages
Returns:
JSON response
"""
response = subprocess.getoutput("./list_all_the_packages.R")
douarddaUnsubmitted
Done
Inline Actions

This will not work. You assume you are running the command from the package directory. And it is a rather bad idea to call the script directly (rather than calling the Rscript tool with the path to the script as argument).

Yo need to figure out where this script will be shipped once the python package is installed and ensure you get it's proper path at execution time.

I'd also rather use check_output (without a shell) here. No need for a shell in the path.

douardda: This will not work. You assume you are running the command from the package directory. And it…
return json.loads(response)
def list_packages(self, response):
"""(Override) List the actual rcran origins from the response.
"""
pass
douarddaUnsubmitted
Done
Inline Actions

?

douardda: ?
def _compute_urls(self, repo):
"""Returns a tuple (project_url, project_metadata_url)
"""
return (
'https://cran.r-project.org/src/contrib/%s_%s.tar.gz' %
(repo["Package"], repo["Version"])
)
douarddaUnsubmitted
Done
Inline Actions

why do you need a method for this one liner?
Plus I don't like having URLs hardwritten in the middle of the code: it should be at least a constant of the class or configurable parameter which defaults to this URL.

Also, use named-based formatting here:

return 'https://cran.r-project.org/src/contrib/%(Package)s_%(Version)s.tar.gz' % repo
douardda: why do you need a method for this one liner? Plus I don't like having URLs hardwritten in the…
def get_model_from_repo(self, repo):
"""(Override) Transform from repository representation to model
"""
project_url = self._compute_urls(repo)
return {
'uid': repo["Package"],
'name': repo["Package"],
'full_name': repo["Title"],
'version': repo["Version"],
'html_url': project_url,
'origin_url': project_url,
'origin_type': 'rcran',
'description': repo["Description"]
}
def transport_response_simplified(self, response):
"""(Override) Transform response to list for model manipulation
"""
return [self.get_model_from_repo(repo_name) for repo_name in response]
def ingest_data(self, identifier, checks=False):
"""(Override)Rework the base ingest_data.
Request server endpoint which gives all in one go.
Simplify and filter response list of repositories. Inject
repo information into local db. Queue loader tasks for
linked repositories.
Args:
identifier: Resource identifier (unused)
checks (bool): Additional checks required (unused)
"""
response = self.r_script_request()
if not response:
return response, []
models_list = self.transport_response_simplified(response)
models_list = self.filter_before_inject(models_list)
all_injected = []
for models in utils.grouper(models_list, n=10000):
models = list(models)
logging.debug('models: %s' % len(models))
# inject into local db
injected = self.inject_repo_data_into_db(models)
# queue workers
self.create_missing_origins_and_tasks(models, injected)
all_injected.append(injected)
# flush
self.db_session.commit()
self.db_session = self.mk_session()
return response, all_injected
douarddaUnsubmitted
Done
Inline Actions

weird. why are these (NOP-like) methods needed?

douardda: weird. why are these (NOP-like) methods needed?
nahimilegaAuthorUnsubmitted
Done
Inline Actions

SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are needed to be present or else code will throw an error. However, the response is not achieved from conventional methods. Thus, there is no need for these functions.

nahimilega: SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are…
douarddaUnsubmitted
Not Done
Inline Actions

Yeah right... this code (I mean the base classes provided for Lister implementation) really need some serious refactoring...

douardda: Yeah right... this code (I mean the base classes provided for Lister implementation) really…
View Options

swh/web/auth/utils.py

  • This file was added.
# Copyright (C) 2019 the Software Heritage developers
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import subprocess
import json
import logging
from swh.lister.rcran.models import RCRANModel
from swh.scheduler import utils
from swh.lister.core.simple_lister import SimpleLister
class RCRANLister(SimpleLister):
MODEL = RCRANModel
LISTER_NAME = 'rcran'
def __init__(self, override_config=None):
SimpleLister.__init__(self, override_config=override_config)
douarddaUnsubmitted
Done
Inline Actions

why implement the overloaded init if it does nothing more than calling super()?

douardda: why implement the overloaded __init__ if it does nothing more than calling super()?
def task_dict(self, origin_type, origin_url, **kwargs):
"""(Override) Return task format dict
This is overridden from the lister_base as more information is
needed for the ingestion task creation.
"""
_type = 'origin-update-%s' % origin_type
anlambertUnsubmitted
Done
Inline Actions

FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
This should be changed to:

_type = 'load-%s' % origin_type
anlambert: FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
nahimilegaAuthorUnsubmitted
Done
Inline Actions

Thanks for updating me about the change @anlambert. I made the recommended changes.

nahimilega: Thanks for updating me about the change @anlambert. I made the recommended changes.
_policy = 'recurring'
project_name = kwargs.get('name')
project_version = kwargs.get('version')
project_metadata = kwargs.get('description')
return utils.create_task_dict(
_type, _policy, project_name, origin_url, project_version,
project_metadata=project_metadata)
douarddaUnsubmitted
Done
Inline Actions

as I said in D1497 I don't like that kwargs.get is used to extract mandatory values.

Also don't see the need for those project_xxx variables. This method should be a one statement only.

douardda: as I said in D1497 I don't like that kwargs.get is used to extract mandatory values. Also…
def r_script_request(self):
"""(Override) Runs r script which uses inbuilt API to return a json
response containing data about all the R packages
Returns:
JSON response
"""
response = subprocess.getoutput("./list_all_the_packages.R")
douarddaUnsubmitted
Done
Inline Actions

This will not work. You assume you are running the command from the package directory. And it is a rather bad idea to call the script directly (rather than calling the Rscript tool with the path to the script as argument).

Yo need to figure out where this script will be shipped once the python package is installed and ensure you get it's proper path at execution time.

I'd also rather use check_output (without a shell) here. No need for a shell in the path.

douardda: This will not work. You assume you are running the command from the package directory. And it…
return json.loads(response)
def list_packages(self, response):
"""(Override) List the actual rcran origins from the response.
"""
pass
douarddaUnsubmitted
Done
Inline Actions

?

douardda: ?
def _compute_urls(self, repo):
"""Returns a tuple (project_url, project_metadata_url)
"""
return (
'https://cran.r-project.org/src/contrib/%s_%s.tar.gz' %
(repo["Package"], repo["Version"])
)
douarddaUnsubmitted
Done
Inline Actions

why do you need a method for this one liner?
Plus I don't like having URLs hardwritten in the middle of the code: it should be at least a constant of the class or configurable parameter which defaults to this URL.

Also, use named-based formatting here:

return 'https://cran.r-project.org/src/contrib/%(Package)s_%(Version)s.tar.gz' % repo
douardda: why do you need a method for this one liner? Plus I don't like having URLs hardwritten in the…
def get_model_from_repo(self, repo):
"""(Override) Transform from repository representation to model
"""
project_url = self._compute_urls(repo)
return {
'uid': repo["Package"],
'name': repo["Package"],
'full_name': repo["Title"],
'version': repo["Version"],
'html_url': project_url,
'origin_url': project_url,
'origin_type': 'rcran',
'description': repo["Description"]
}
def transport_response_simplified(self, response):
"""(Override) Transform response to list for model manipulation
"""
return [self.get_model_from_repo(repo_name) for repo_name in response]
def ingest_data(self, identifier, checks=False):
"""(Override)Rework the base ingest_data.
Request server endpoint which gives all in one go.
Simplify and filter response list of repositories. Inject
repo information into local db. Queue loader tasks for
linked repositories.
Args:
identifier: Resource identifier (unused)
checks (bool): Additional checks required (unused)
"""
response = self.r_script_request()
if not response:
return response, []
models_list = self.transport_response_simplified(response)
models_list = self.filter_before_inject(models_list)
all_injected = []
for models in utils.grouper(models_list, n=10000):
models = list(models)
logging.debug('models: %s' % len(models))
# inject into local db
injected = self.inject_repo_data_into_db(models)
# queue workers
self.create_missing_origins_and_tasks(models, injected)
all_injected.append(injected)
# flush
self.db_session.commit()
self.db_session = self.mk_session()
return response, all_injected
douarddaUnsubmitted
Done
Inline Actions

weird. why are these (NOP-like) methods needed?

douardda: weird. why are these (NOP-like) methods needed?
nahimilegaAuthorUnsubmitted
Done
Inline Actions

SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are needed to be present or else code will throw an error. However, the response is not achieved from conventional methods. Thus, there is no need for these functions.

nahimilega: SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are…
douarddaUnsubmitted
Not Done
Inline Actions

Yeah right... this code (I mean the base classes provided for Lister implementation) really need some serious refactoring...

douardda: Yeah right... this code (I mean the base classes provided for Lister implementation) really…
View Options

swh/web/common/utils.py

  • This file was added.
# Copyright (C) 2019 the Software Heritage developers
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import subprocess
import json
import logging
from swh.lister.rcran.models import RCRANModel
from swh.scheduler import utils
from swh.lister.core.simple_lister import SimpleLister
class RCRANLister(SimpleLister):
MODEL = RCRANModel
LISTER_NAME = 'rcran'
def __init__(self, override_config=None):
SimpleLister.__init__(self, override_config=override_config)
douarddaUnsubmitted
Done
Inline Actions

why implement the overloaded init if it does nothing more than calling super()?

douardda: why implement the overloaded __init__ if it does nothing more than calling super()?
def task_dict(self, origin_type, origin_url, **kwargs):
"""(Override) Return task format dict
This is overridden from the lister_base as more information is
needed for the ingestion task creation.
"""
_type = 'origin-update-%s' % origin_type
anlambertUnsubmitted
Done
Inline Actions

FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
This should be changed to:

_type = 'load-%s' % origin_type
anlambert: FYI, we are currently renaming the scheduler task names to something more rational (see T1508).
nahimilegaAuthorUnsubmitted
Done
Inline Actions

Thanks for updating me about the change @anlambert. I made the recommended changes.

nahimilega: Thanks for updating me about the change @anlambert. I made the recommended changes.
_policy = 'recurring'
project_name = kwargs.get('name')
project_version = kwargs.get('version')
project_metadata = kwargs.get('description')
return utils.create_task_dict(
_type, _policy, project_name, origin_url, project_version,
project_metadata=project_metadata)
douarddaUnsubmitted
Done
Inline Actions

as I said in D1497 I don't like that kwargs.get is used to extract mandatory values.

Also don't see the need for those project_xxx variables. This method should be a one statement only.

douardda: as I said in D1497 I don't like that kwargs.get is used to extract mandatory values. Also…
def r_script_request(self):
"""(Override) Runs r script which uses inbuilt API to return a json
response containing data about all the R packages
Returns:
JSON response
"""
response = subprocess.getoutput("./list_all_the_packages.R")
douarddaUnsubmitted
Done
Inline Actions

This will not work. You assume you are running the command from the package directory. And it is a rather bad idea to call the script directly (rather than calling the Rscript tool with the path to the script as argument).

Yo need to figure out where this script will be shipped once the python package is installed and ensure you get it's proper path at execution time.

I'd also rather use check_output (without a shell) here. No need for a shell in the path.

douardda: This will not work. You assume you are running the command from the package directory. And it…
return json.loads(response)
def list_packages(self, response):
"""(Override) List the actual rcran origins from the response.
"""
pass
douarddaUnsubmitted
Done
Inline Actions

?

douardda: ?
def _compute_urls(self, repo):
"""Returns a tuple (project_url, project_metadata_url)
"""
return (
'https://cran.r-project.org/src/contrib/%s_%s.tar.gz' %
(repo["Package"], repo["Version"])
)
douarddaUnsubmitted
Done
Inline Actions

why do you need a method for this one liner?
Plus I don't like having URLs hardwritten in the middle of the code: it should be at least a constant of the class or configurable parameter which defaults to this URL.

Also, use named-based formatting here:

return 'https://cran.r-project.org/src/contrib/%(Package)s_%(Version)s.tar.gz' % repo
douardda: why do you need a method for this one liner? Plus I don't like having URLs hardwritten in the…
def get_model_from_repo(self, repo):
"""(Override) Transform from repository representation to model
"""
project_url = self._compute_urls(repo)
return {
'uid': repo["Package"],
'name': repo["Package"],
'full_name': repo["Title"],
'version': repo["Version"],
'html_url': project_url,
'origin_url': project_url,
'origin_type': 'rcran',
'description': repo["Description"]
}
def transport_response_simplified(self, response):
"""(Override) Transform response to list for model manipulation
"""
return [self.get_model_from_repo(repo_name) for repo_name in response]
def ingest_data(self, identifier, checks=False):
"""(Override)Rework the base ingest_data.
Request server endpoint which gives all in one go.
Simplify and filter response list of repositories. Inject
repo information into local db. Queue loader tasks for
linked repositories.
Args:
identifier: Resource identifier (unused)
checks (bool): Additional checks required (unused)
"""
response = self.r_script_request()
if not response:
return response, []
models_list = self.transport_response_simplified(response)
models_list = self.filter_before_inject(models_list)
all_injected = []
for models in utils.grouper(models_list, n=10000):
models = list(models)
logging.debug('models: %s' % len(models))
# inject into local db
injected = self.inject_repo_data_into_db(models)
# queue workers
self.create_missing_origins_and_tasks(models, injected)
all_injected.append(injected)
# flush
self.db_session.commit()
self.db_session = self.mk_session()
return response, all_injected
douarddaUnsubmitted
Done
Inline Actions

weird. why are these (NOP-like) methods needed?

douardda: weird. why are these (NOP-like) methods needed?
nahimilegaAuthorUnsubmitted
Done
Inline Actions

SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are needed to be present or else code will throw an error. However, the response is not achieved from conventional methods. Thus, there is no need for these functions.

nahimilega: SimpleLister class is inherited, and these methods are @abc.abstractmethod hence, they are…
douarddaUnsubmitted
Not Done
Inline Actions

Yeah right... this code (I mean the base classes provided for Lister implementation) really need some serious refactoring...

douardda: Yeah right... this code (I mean the base classes provided for Lister implementation) really…
View Options

swh/web/templates/admin/deposit.html

Loading...
View Options

swh/web/templates/layout.html

Loading...
View Options

swh/web/tests/admin/test_deposit.py

Loading...
View Options

swh/web/tests/test_templates.py

Loading...
About · Developer information · Legal