Changeset View
Changeset View
Standalone View
Standalone View
swh/lister/gnu/lister.py
| Show All 25 Lines | class GNULister(SimpleLister): | ||||
| def task_dict(self, origin_type, origin_url, **kwargs): | def task_dict(self, origin_type, origin_url, **kwargs): | ||||
| """ | """ | ||||
| Return task format dict | Return task format dict | ||||
| This is overridden from the lister_base as more information is | This is overridden from the lister_base as more information is | ||||
| needed for the ingestion task creation. | needed for the ingestion task creation. | ||||
| """ | """ | ||||
| return utils.create_task_dict( | return utils.create_task_dict( | ||||
| 'load-%s' % origin_type, 'recurring', kwargs.get('name'), | 'load-%s' % origin_type, 'recurring', kwargs.get('name'), | ||||
| origin_url, tarballs=self.tarballs[kwargs.get('name')]) | origin_url, tarballs=self.tarballs[kwargs.get('name')]) | ||||
nahimilega: I have only changed origin_type, but if we are using tarball loader, then I suppose we have to… | |||||
Not Done Inline Actions
Yes, but there is no need for another code change In swh-lister, my understanding of the current code is that now the task-type will change to load-tar. 13:36:01 +ardumont | archit_agrawal[m: could you change the lister-gnu's origin-type from gnu to tar btw?
13:36:38 +ardumont | it will also change the scheduler's task type from load-gnu to load-tar but i'll deal with
that part later (impact is in the scheduler)
13:36:54 +ardumont | (as the output tasks do not load yet, that's not an issue)For the loader-tar in question, indeed, currently it does not support the contract. As this runs in production, I need to:
update gnu_repo set origin_type='tar' where origin_type='gnu';
From the top of my head: # insert new task-type 'load-tar' (which does not exist yet IIRC)
insert into task_type(name, ...) values ('load-tar'...); # or some such
update task set task_type='load-tar' where task_type='load-gnu';
delete from task_type where task_type='load-gnu';ardumont: > I have only changed origin_type, but if we are using tarball loader, then I suppose we have… | |||||
| def get_file(self): | def get_file(self): | ||||
| ''' | ''' | ||||
| Download and unzip tree.json.gz file and returns its content | Download and unzip tree.json.gz file and returns its content | ||||
| in JSON format | in JSON format | ||||
| Returns | Returns | ||||
| File content in dictionary format | File content in dictionary format | ||||
| ▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines | def get_model_from_repo(self, repo): | ||||
| """ | """ | ||||
| return { | return { | ||||
| 'uid': repo['name'], | 'uid': repo['name'], | ||||
| 'name': repo['name'], | 'name': repo['name'], | ||||
| 'full_name': repo['name'], | 'full_name': repo['name'], | ||||
| 'html_url': repo['url'], | 'html_url': repo['url'], | ||||
| 'origin_url': repo['url'], | 'origin_url': repo['url'], | ||||
| 'time_last_updated': repo['time_modified'], | 'time_last_updated': repo['time_modified'], | ||||
| 'origin_type': 'gnu', | 'origin_type': 'tar', | ||||
| } | } | ||||
| def transport_response_simplified(self, response): | def transport_response_simplified(self, response): | ||||
| """Transform response to list for model manipulation | """Transform response to list for model manipulation | ||||
| """ | """ | ||||
| return [self.get_model_from_repo(repo) for repo in response] | return [self.get_model_from_repo(repo) for repo in response] | ||||
| ▲ Show 20 Lines • Show All 82 Lines • Show Last 20 Lines | |||||
I have only changed origin_type, but if we are using tarball loader, then I suppose we have to change this according to that