Great ! Could you also add that configuration to the swh-web service ? I am currently working on T2124 and I have added a couple of statsd metric to the webapp.

In D2416#57093, @anlambert wrote:

Great ! Could you also add that configuration to the swh-web service ? I am currently working on T2124 and I have added a couple of statsd metric to the webapp.

Sure. I'm also moving these env vars in a dedicated file instead of putting them directly in the docker-compose file.

Sure. I'm also moving these env vars in a dedicated file instead of putting them directly in the docker-compose file.

move statsd emv var declarations in an env/statsd.env file and use it on swh-web

so anlambert is happy :-)

Harbormaster completed remote builds in B9630: Diff 8549.Dec 9 2019, 5:06 PM

Closed by commit rDENV3007fb37f47d: docker/compose: configure the statsd endpoint for most of the containers (authored by douardda). · Explain WhyDec 9 2019, 5:11 PM

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDENV3007fb37f47d: docker/compose: configure the statsd endpoint for most of the containers.

Revision Contents
Changeset List

Path

Size

docker/

docker-compose.yml

16 lines

Diff 8544

View Options

docker/docker-compose.yml

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				import json
				from pathlib import Path
				import sqlite3

				from swh.model.identifiers import CONTENT, SWHID, parse_swhid
				from swh.web.client.client import typify


				class Cache:
				def __init__(self, cache_dir):
				seirlUnsubmitted Done Inline Actions I would like this class to be more pythonic by working as a context manager. For me, the sqlite3 connection should probably happen in an `__enter__()` function, and closing it in an `__exit__()` function. I know you cannot use a with block in the Fuse operation class (a shame...), but at least it keeps the quirky aspect of this solely in the Fuse class. seirl: I would like this class to be more pythonic by working as a context manager. For me, the…
				seirlUnsubmitted Done Inline Actions As discussed, we probably want to have two caches (they can probably both inherit some common behavior from an abstract base if that's helpful). If you do go ahead with two classes, you will be able to use getitem and setitem to enable a more pythonic way of accessing this cache, like `blob_cache[swhid] = X`. seirl: As discussed, we probably want to have two caches (they can probably both inherit some common…
				if cache_dir == ":memory:":
				self.conn = sqlite3.connect(":memory:")
				else:
				cache_path = Path(cache_dir, "cache.sqlite")
				cache_path.parents[0].mkdir(parents=True, exist_ok=True)
				self.conn = sqlite3.connect(cache_path)

				self.db = self.conn.cursor()
				self.db.execute("create table if not exists metadata_cache (swhid, metadata)")
				self.db.execute("create table if not exists blob_cache (swhid, blob)")
				self.conn.commit()
				zackUnsubmitted Done Inline Actions if the sqlite3 module is a well-behaved Python DB module, this could be better written with a contextmanager that commits at the end zack: if the sqlite3 module is a well-behaved Python DB module, this could be better written with a…

				seirlUnsubmitted Done Inline Actions I think it's weird to keep a long-lived cursor in the class, you should probably create a new one whenever you're executing something without the `conn.execute(` shortcuts. I think the class should only hold the connection as its state, not the currently active cursors. seirl:* I think it's weird to keep a long-lived cursor in the class, you should probably create a new…
				def get_metadata(self, swhid: SWHID):
				zackUnsubmitted Done Inline Actions Please add docstring to all public methods, classes and modules. No need to be very detailed for now, oneliners would do for starters. But it's much better to have them since the beginning than having to add all of them later. zack: Please add docstring to all public methods, classes and modules. No need to be very detailed…
				self.db.execute(
				"select metadata from metadata_cache where swhid=?", (str(swhid),)
				)
				cache = self.db.fetchone()
				if cache:
				metadata = json.loads(cache[0])
				return typify(metadata, swhid.object_type)

				def put_metadata(self, swhid: SWHID, metadata):
				zackUnsubmitted Done Inline Actions methods that do not return anything should be explicitly annotated with `-> None`, otherwise the default return value is `Any` (h/t @olasd ) zack: methods that do not return anything should be explicitly annotated with `-> None`, otherwise…
				self.db.execute(
				"insert into metadata_cache values (?, ?)",
				(
				str(swhid),
				json.dumps(
				metadata,
				# Converts the typified JSON to plain str version
				default=lambda x: x.object_id
				if isinstance(x, SWHID)
				else x.__dict__,
				),
				),
				)
				self.conn.commit()

				def get_metadata_swhids(self):
				self.db.execute("select swhid from metadata_cache")
				swhids = self.db.fetchall()
				swhids = [parse_swhid(x[0]) for x in swhids]
				return swhids

				def get_blob(self, swhid: SWHID):
				if swhid.object_type != CONTENT:
				zackUnsubmitted Done Inline Actions We might want to fix this in swh-web-client, making typify optional in return values, as it's really silly to typify there and goes the other way here. Can you file a dedicated task for this? zack: We might want to fix this in swh-web-client, making typify optional in return values, as it's…
				haltodeAuthorUnsubmitted Done Inline Actions Filed in: T2646. haltode: Filed in: T2646.
				raise AttributeError("Cannot retrieve blob from non-content object type")
				seirlUnsubmitted Done Inline Actions While this fix is here temporarily, make it less hard to read by using parentheses, it's not obvious what the "if" applies to here. seirl: While this fix is here temporarily, make it less hard to read by using parentheses, it's not…

				self.db.execute("select blob from blob_cache where swhid=?", (str(swhid),))
				cache = self.db.fetchone()
				if cache:
				blob = cache[0]
				return blob

				def put_blob(self, swhid: SWHID, blob):
				self.db.execute("insert into blob_cache values (?, ?)", (str(swhid), blob))
				self.conn.commit()

docker/compose: configure the statsd endpoint for most of the containersClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 8544

docker/docker-compose.yml

docker/compose: configure the statsd endpoint for most of the containers
ClosedPublic
Actions

Revision Contents
Changeset List