Changeset View
Changeset View
Standalone View
Standalone View
README.md
Show All 35 Lines | |||||
More details in requirements*.txt | More details in requirements*.txt | ||||
Local deployment | Local deployment | ||||
----------- | ----------- | ||||
## lister-github | ## lister-github | ||||
1. git clone under $GHLISTER_ROOT (of your choosing) | ### Preparation steps | ||||
1. git clone under $SWH_ENVIRONMENT_HOME/swh-lister (of your choosing) | |||||
2. mkdir ~/.config/swh/ ~/.cache/swh/lister/github.com/ | 2. mkdir ~/.config/swh/ ~/.cache/swh/lister/github.com/ | ||||
3. create configuration file ~/.config/swh/lister-github.com.yml | 3. create configuration file ~/.config/swh/lister-github.com.yml | ||||
4. Bootstrap the db instance schema | 4. Bootstrap the db instance schema | ||||
``` sh | $ createdb lister-github | ||||
$ createdb lister-github.com | $ python3 -m swh.lister.cli --db-url postgres:///lister-github \ | ||||
$ bin/ghlister --db-url postgres:///lister-github.com createdb | --lister github \ | ||||
``` | --create-tables | ||||
Configuration file samples | |||||
------------------------- | |||||
## github | |||||
cat ~/.config/swh/lister-github.com.yml | ### Configuration file sample | ||||
$ cat ~/.config/swh/lister-github.com.yml | |||||
# see http://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls | # see http://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls | ||||
lister_db_url: postgres:///lister-github.com | lister_db_url: postgres:///lister-github | ||||
credentials: [] | credentials: [] | ||||
cache_responses: True | cache_responses: True | ||||
cache_dir: /home/zack/.cache/swh/lister/github.com | cache_dir: /home/zack/.cache/swh/lister/github.com | ||||
storage: | |||||
cls: remote | |||||
args: | |||||
url: http://localhost:5002/ | |||||
### Run | |||||
$ python3 | |||||
>>> import logging | |||||
>>> logging.basicConfig(level=logging.DEBUG) | |||||
>>> from swh.lister.github.tasks import RangeGitHubLister | |||||
>>> RangeGitHubLister().run(364, 365) | |||||
INFO:root:listing repos starting at 364 | |||||
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com | |||||
DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /repositories?since=364 HTTP/1.1" 200 None | |||||
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost | |||||
DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/add HTTP/1.1" 200 1 | |||||
## lister-gitlab | |||||
### preparation steps | |||||
1. git clone under $SWH_ENVIRONMENT_HOME/swh-lister (of your choosing) | |||||
2. mkdir ~/.config/swh/ ~/.cache/swh/lister/gitlab/ | |||||
3. create configuration file ~/.config/swh/lister-gitlab.yml | |||||
4. Bootstrap the db instance schema | |||||
$ createdb lister-gitlab | |||||
$ python3 -m swh.lister.cli --db-url postgres:///lister-gitlab \ | |||||
--lister gitlab \ | |||||
--create-tables | |||||
### Configuration file sample | |||||
$ cat ~/.config/swh/lister-gitlab.yml | |||||
# see http://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls | |||||
lister_db_url: postgres:///lister-gitlab | |||||
credentials: [] | |||||
cache_responses: True | |||||
cache_dir: /home/zack/.cache/swh/lister/gitlab | |||||
storage: | storage: | ||||
cls: remote | cls: remote | ||||
args: | args: | ||||
url: http://localhost:5002/ | url: http://localhost:5002/ | ||||
### Run | |||||
$ python3 | |||||
Python 3.6.6 (default, Jun 27 2018, 14:44:17) | |||||
[GCC 8.1.0] on linux | |||||
Type "help", "copyright", "credits" or "license" for more information. | |||||
>>> from swh.lister.gitlab.tasks import RangeGitLabLister; RangeGitLabLister().run_task(1, 2, | |||||
instance='salsa.debian.org', api_baseurl='https://salsa.debian.org/api/v4') | |||||
>>> from swh.lister.gitlab.tasks import RangeGitLabLister; RangeGitLabLister().run_task(1, 2, | |||||
instance='gitlab.freedesktop.org', api_baseurl='https://gitlab.freedesktop.org/api/v4') | |||||
>>> from swh.lister.gitlab.tasks import RangeGitLabLister; RangeGitLabLister().run_task(1, 2, | |||||
instance='gitlab.gnome.org', api_baseurl='https://gitlab.gnome.org/api/v4') | |||||
>>> from swh.lister.gitlab.tasks import RangeGitLabLister; RangeGitLabLister().run_task(1, 2, | |||||
instance='gitlab.inria.fr', api_baseurl='https://gitlab.inria.fr/api/v4') | |||||
>>> |