Page MenuHomeSoftware Heritage
Differential D243

swh.web.setting: Use different setup according to platform (dev/prod)
ClosedPublic
Actions

Authored by ardumont on Sep 11 2017, 7:42 PM.

Details

Reviewers
anlambert
olasd
Group Reviewers
Reviewers
Commits
R65:bb50bb311a53: swh.web.config: Fix cache_uri and limits to the right level
R65:f9a82eed6019: swh.web.throttling: Clarify config keys intent
R65:f370df3e2c5c: swh.web.config: Set config default as near as production
R65:4ee5ea05b386: swh.web.settings.common: Remove throttling inhibition in DEBUG mode
R65:6eacd12fb10a: resources: Remove unused setup file
R65:702cf9783aa6: swh.web.settings: Split setup according to platforms (dev/prod)
R65:2bbbe920199a: swh.web.settings: Setup cache uri (if any) from webapp.yml
rDWAPPS6eacd12fb10a: resources: Remove unused setup file
rDWAPPS4ee5ea05b386: swh.web.settings.common: Remove throttling inhibition in DEBUG mode
rDWAPPSf9a82eed6019: swh.web.throttling: Clarify config keys intent
rDWAPPSbb50bb311a53: swh.web.config: Fix cache_uri and limits to the right level
rDWAPPSf370df3e2c5c: swh.web.config: Set config default as near as production
rDWAPPS2bbbe920199a: swh.web.settings: Setup cache uri (if any) from webapp.yml
rDWAPPS702cf9783aa6: swh.web.settings: Split setup according to platforms (dev/prod)
Summary

swh.web.settings: Use different setup according to platform (dev/prod)
swh.web.settings: Permit to setup the cache uri from webapp.yml
swh.web.config: Clarify setup configuration key intent
swh.web.settings: Remove throttling inhitition when in debug mode
Misc: Remove unused files from git (db.sqlite3, webapp.yml)

Related T754

Diff Detail

Repository
rDWAPPS Web applications
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 998
Build 1322: Software Heritage Python tests
Build 1321: arc lint + arc unit

Event Timeline

ardumont created this revision.Sep 11 2017, 7:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 11 2017, 7:42 PM
Harbormaster failed remote builds in B994: Diff 793!Sep 11 2017, 7:42 PM
ardumont added inline comments.Sep 11 2017, 7:46 PM
resources/test/webapp.yml
7

I'm not sure we want to keep that file (it's no longer used).

pros:

  • it's with the code (might as well be in a readme though).

cons:

  • redundant with swh-site/defaults.yaml (if you know where to look).
swh/web/config.py
21–28

typo: cache_uri...

ardumont updated this revision to Diff 794.Sep 11 2017, 7:52 PM
  • swh.web.setting: Permit to setup the cache uri from webapp.yml
  • cleanup: Remove db.sqlite3 binary from git
Harbormaster failed remote builds in B995: Diff 794!Sep 11 2017, 7:53 PM
ardumont marked an inline comment as done.Sep 11 2017, 7:58 PM
Harbormaster failed remote builds in B995: Diff 794!Sep 11 2017, 7:59 PM
Harbormaster failed remote builds in B995: Diff 794!Sep 11 2017, 8:02 PM
Harbormaster failed remote builds in B995: Diff 794!
Harbormaster completed remote builds in B995: Diff 794.Sep 11 2017, 8:09 PM
ardumont added a comment.EditedSep 11 2017, 8:10 PM

Fixed CI (due to swh-web's new dependencies not installed on CI worker)

ardumont updated this revision to Diff 795.Sep 12 2017, 10:08 AM

Rebase to latest master + fix some default config property level

  • swh.web.setting: Use different setup according to platform (dev/prod)
  • swh.web.setting: Permit to setup the cache uri from webapp.yml
  • cleanup: Remove db.sqlite3 binary from git
  • swh.web.config: Fix cache_uri and limits to the right level
  • swh.web.config: Set config default as near as production
Harbormaster failed remote builds in B996: Diff 795!Sep 12 2017, 10:09 AM
ardumont updated this revision to Diff 796.Sep 12 2017, 11:00 AM

Rebased + clarify setup according to latest discussion with anlambert

  • swh.web.setting: Use different setup according to platform (dev/prod)
  • swh.web.setting: Permit to setup the cache uri from webapp.yml
  • swh.web.config: Fix cache_uri and limits to the right level
  • swh.web.config: Set config default as near as production
  • swh.web.throttling: Clarify config keys intent
Harbormaster completed remote builds in B997: Diff 796.Sep 12 2017, 11:00 AM
ardumont added inline comments.Sep 12 2017, 11:05 AM
swh/web/config.py
24

The setup keys was at the wrong level. I fixed that.

The terms were either redundant or not clear.
After discussion with anlambert, we settled on:

  • 'throttling' as main keys
  • 'scopes': for the listing of scopes setup
anlambert removed a reviewer: olasd.Sep 12 2017, 11:09 AM
ardumont updated this revision to Diff 797.Sep 12 2017, 11:23 AM
  • swh.web.setting: Use different setup according to platform (dev/prod)
  • swh.web.setting: Permit to setup the cache uri from webapp.yml
  • swh.web.config: Fix cache_uri and limits to the right level
  • swh.web.config: Set config default as near as production
  • swh.web.throttling: Clarify config keys intent
  • swh.web.settings.common: Remove throttling inhibition in DEBUG mode
  • resources: Remove unused setup file
Harbormaster completed remote builds in B998: Diff 797.Sep 12 2017, 11:23 AM
ardumont updated this revision to Diff 798.Sep 12 2017, 11:34 AM

Rewording commits appropriately

  • swh.web.settings: Split setup according to platforms (dev/prod)
  • swh.web.settings: Setup cache uri (if any) from webapp.yml
  • swh.web.config: Fix cache_uri and limits to the right level
  • swh.web.config: Set config default as near as production
  • swh.web.throttling: Clarify config keys intent
  • swh.web.settings.common: Remove throttling inhibition in DEBUG mode
  • resources: Remove unused setup file
ardumont edited the summary of this revision. (Show Details)Sep 12 2017, 11:35 AM
Harbormaster completed remote builds in B999: Diff 798.Sep 12 2017, 11:35 AM
anlambert accepted this revision.Sep 12 2017, 11:58 AM
anlambert added a reviewer: olasd.
This revision is now accepted and ready to land.Sep 12 2017, 11:58 AM
Closed by commit rDWAPPS702cf9783aa6: swh.web.settings: Split setup according to platforms (dev/prod) (authored by ardumont). · Explain WhySep 12 2017, 12:01 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
3 lines
resources/
test/
swh/
web/
common/
18 lines
16 lines
3 lines
settings/
common.py
settings.py
7 lines
settings/
7 lines
15 lines
tests/
35 lines
2 lines

Diff 797

View Options

Makefile.local

from typing import Dict, List, Optional, Union from typing import Dict, List, Optional, Union
import yaml
from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI
from .base import DictMapping, SingleFileMapping from .base import YamlMapping
class SafeLoader(yaml.SafeLoader):
yaml_implicit_resolvers = {
k: [r for r in v if r[0] != "tag:yaml.org,2002:timestamp"]
for k, v in yaml.SafeLoader.yaml_implicit_resolvers.items()
}
class CffMapping(DictMapping, SingleFileMapping): class CffMapping(YamlMapping):
"""Dedicated class for Citation (CITATION.cff) mapping and translation""" """Dedicated class for Citation (CITATION.cff) mapping and translation"""
name = "cff" name = "cff"
filename = b"CITATION.cff" filename = b"CITATION.cff"
mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"] mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"]
string_fields = ["keywords", "license", "abstract", "version", "doi"] string_fields = ["keywords", "license", "abstract", "version", "doi"]
def translate(self, raw_content: bytes) -> Optional[Dict[str, str]]:
raw_content_string: str = raw_content.decode()
try:
content_dict = yaml.load(raw_content_string, Loader=SafeLoader)
except yaml.scanner.ScannerError:
return None
if isinstance(content_dict, dict):
return self._translate_dict(content_dict)
vlorentzUnsubmitted
Done
Inline Actions

please rebase before submitting diffs; this code was simplified; so you can simplify the new base YamlMapping similarly

vlorentz: please rebase before submitting diffs; this code was simplified; so you can simplify the new…
return None
def normalize_authors(self, d: List[dict]) -> Dict[str, list]: def normalize_authors(self, d: List[dict]) -> Dict[str, list]:
result = [] result = []
for author in d: for author in d:
author_data: Dict[str, Optional[Union[str, Dict]]] = { author_data: Dict[str, Optional[Union[str, Dict]]] = {
"@type": SCHEMA_URI + "Person" "@type": SCHEMA_URI + "Person"
} }
if "orcid" in author and isinstance(author["orcid"], str): if "orcid" in author and isinstance(author["orcid"], str):
author_data["@id"] = author["orcid"] author_data["@id"] = author["orcid"]
Show All 30 Lines
View Options

resources/test/webapp.yml

from typing import Dict, List, Optional, Union from typing import Dict, List, Optional, Union
import yaml
from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI
from .base import DictMapping, SingleFileMapping from .base import YamlMapping
class SafeLoader(yaml.SafeLoader):
yaml_implicit_resolvers = {
k: [r for r in v if r[0] != "tag:yaml.org,2002:timestamp"]
for k, v in yaml.SafeLoader.yaml_implicit_resolvers.items()
}
class CffMapping(DictMapping, SingleFileMapping): class CffMapping(YamlMapping):
"""Dedicated class for Citation (CITATION.cff) mapping and translation""" """Dedicated class for Citation (CITATION.cff) mapping and translation"""
name = "cff" name = "cff"
filename = b"CITATION.cff" filename = b"CITATION.cff"
mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"] mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"]
string_fields = ["keywords", "license", "abstract", "version", "doi"] string_fields = ["keywords", "license", "abstract", "version", "doi"]
def translate(self, raw_content: bytes) -> Optional[Dict[str, str]]:
raw_content_string: str = raw_content.decode()
try:
content_dict = yaml.load(raw_content_string, Loader=SafeLoader)
except yaml.scanner.ScannerError:
return None
if isinstance(content_dict, dict):
return self._translate_dict(content_dict)
vlorentzUnsubmitted
Done
Inline Actions

please rebase before submitting diffs; this code was simplified; so you can simplify the new base YamlMapping similarly

vlorentz: please rebase before submitting diffs; this code was simplified; so you can simplify the new…
return None
def normalize_authors(self, d: List[dict]) -> Dict[str, list]: def normalize_authors(self, d: List[dict]) -> Dict[str, list]:
result = [] result = []
for author in d: for author in d:
author_data: Dict[str, Optional[Union[str, Dict]]] = { author_data: Dict[str, Optional[Union[str, Dict]]] = {
"@type": SCHEMA_URI + "Person" "@type": SCHEMA_URI + "Person"
} }
if "orcid" in author and isinstance(author["orcid"], str): if "orcid" in author and isinstance(author["orcid"], str):
author_data["@id"] = author["orcid"] author_data["@id"] = author["orcid"]
Show All 30 Lines
View Options

swh/web/common/throttling.py

from typing import Dict, List, Optional, Union from typing import Dict, List, Optional, Union
import yaml
from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI
from .base import DictMapping, SingleFileMapping from .base import YamlMapping
class SafeLoader(yaml.SafeLoader):
yaml_implicit_resolvers = {
k: [r for r in v if r[0] != "tag:yaml.org,2002:timestamp"]
for k, v in yaml.SafeLoader.yaml_implicit_resolvers.items()
}
class CffMapping(DictMapping, SingleFileMapping): class CffMapping(YamlMapping):
"""Dedicated class for Citation (CITATION.cff) mapping and translation""" """Dedicated class for Citation (CITATION.cff) mapping and translation"""
name = "cff" name = "cff"
filename = b"CITATION.cff" filename = b"CITATION.cff"
mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"] mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"]
string_fields = ["keywords", "license", "abstract", "version", "doi"] string_fields = ["keywords", "license", "abstract", "version", "doi"]
def translate(self, raw_content: bytes) -> Optional[Dict[str, str]]:
raw_content_string: str = raw_content.decode()
try:
content_dict = yaml.load(raw_content_string, Loader=SafeLoader)
except yaml.scanner.ScannerError:
return None
if isinstance(content_dict, dict):
return self._translate_dict(content_dict)
vlorentzUnsubmitted
Done
Inline Actions

please rebase before submitting diffs; this code was simplified; so you can simplify the new base YamlMapping similarly

vlorentz: please rebase before submitting diffs; this code was simplified; so you can simplify the new…
return None
def normalize_authors(self, d: List[dict]) -> Dict[str, list]: def normalize_authors(self, d: List[dict]) -> Dict[str, list]:
result = [] result = []
for author in d: for author in d:
author_data: Dict[str, Optional[Union[str, Dict]]] = { author_data: Dict[str, Optional[Union[str, Dict]]] = {
"@type": SCHEMA_URI + "Person" "@type": SCHEMA_URI + "Person"
} }
if "orcid" in author and isinstance(author["orcid"], str): if "orcid" in author and isinstance(author["orcid"], str):
author_data["@id"] = author["orcid"] author_data["@id"] = author["orcid"]
Show All 30 Lines
View Options

swh/web/config.py

from typing import Dict, List, Optional, Union from typing import Dict, List, Optional, Union
import yaml
from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI from swh.indexer.codemeta import CROSSWALK_TABLE, SCHEMA_URI
from .base import DictMapping, SingleFileMapping from .base import YamlMapping
class SafeLoader(yaml.SafeLoader):
yaml_implicit_resolvers = {
k: [r for r in v if r[0] != "tag:yaml.org,2002:timestamp"]
for k, v in yaml.SafeLoader.yaml_implicit_resolvers.items()
}
class CffMapping(DictMapping, SingleFileMapping): class CffMapping(YamlMapping):
"""Dedicated class for Citation (CITATION.cff) mapping and translation""" """Dedicated class for Citation (CITATION.cff) mapping and translation"""
name = "cff" name = "cff"
filename = b"CITATION.cff" filename = b"CITATION.cff"
mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"] mapping = CROSSWALK_TABLE["Citation File Format Core (CFF-Core) 1.0.2"]
string_fields = ["keywords", "license", "abstract", "version", "doi"] string_fields = ["keywords", "license", "abstract", "version", "doi"]
def translate(self, raw_content: bytes) -> Optional[Dict[str, str]]:
raw_content_string: str = raw_content.decode()
try:
content_dict = yaml.load(raw_content_string, Loader=SafeLoader)
except yaml.scanner.ScannerError:
return None
if isinstance(content_dict, dict):
return self._translate_dict(content_dict)
vlorentzUnsubmitted
Done
Inline Actions

please rebase before submitting diffs; this code was simplified; so you can simplify the new base YamlMapping similarly

vlorentz: please rebase before submitting diffs; this code was simplified; so you can simplify the new…
return None
def normalize_authors(self, d: List[dict]) -> Dict[str, list]: def normalize_authors(self, d: List[dict]) -> Dict[str, list]:
result = [] result = []
for author in d: for author in d:
author_data: Dict[str, Optional[Union[str, Dict]]] = { author_data: Dict[str, Optional[Union[str, Dict]]] = {
"@type": SCHEMA_URI + "Person" "@type": SCHEMA_URI + "Person"
} }
if "orcid" in author and isinstance(author["orcid"], str): if "orcid" in author and isinstance(author["orcid"], str):
author_data["@id"] = author["orcid"] author_data["@id"] = author["orcid"]
Show All 30 Lines
View Options

swh/web/manage.py

Loading...
View Options

swh/web/settings.py

Loading...
View Options

swh/web/settings/common.py

Loading...
View Options

swh/web/settings/development.py

Loading...
View Options

swh/web/settings/production.py

Loading...
View Options

swh/web/tests/__init__.py

Loading...
View Options

swh/web/wsgi.py

Loading...
About · Developer information · Legal