Page MenuHomeSoftware Heritage
Feed Advanced Search

Dec 12 2019

douardda accepted D2431: docker: Fix nginx bound port on host.

sorry for the glitch

Dec 12 2019, 11:08 AM

Dec 11 2019

douardda added inline comments to D2424: Enable Sentry for CLIs, and Gunicorn and Celery workers..
Dec 11 2019, 2:06 PM
douardda added a comment to D2427: Make load-deposit and check-deposit URL argument absolute.

tests should pass ok however no test exists for the interaction with the
scheduler, so...

By adding the swh_scheduler fixture in one test (which adds a deposit), you could either:

  • check the check_task_id, load_task_id are set
  • introspect the swh_scheduler.search_task('check-deposit') tasks to ensure there is something [1]

[1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/cgit/tests/test_lister.py$53-55

Dec 11 2019, 11:10 AM
douardda added a task to D2427: Make load-deposit and check-deposit URL argument absolute: T2118: Deposit: End to End monitoring.
Dec 11 2019, 10:57 AM
douardda added a revision to T2118: Deposit: End to End monitoring: D2427: Make load-deposit and check-deposit URL argument absolute.
Dec 11 2019, 10:57 AM · Sprint 2019/12 (Monitor and Conquer)
douardda created D2427: Make load-deposit and check-deposit URL argument absolute.
Dec 11 2019, 10:56 AM
douardda moved T2118: Deposit: End to End monitoring from Backlog to in progress on the Sprint 2019/12 (Monitor and Conquer) board.
Dec 11 2019, 10:22 AM · Sprint 2019/12 (Monitor and Conquer)

Dec 10 2019

douardda updated the diff for D2421: docker/deposit: configure nginx so the WSGI mountpoint for /deposit is ok-ish.

add X-Forwarded-Host header and configure allowed_hosts as '*'

Dec 10 2019, 2:18 PM
douardda created D2421: docker/deposit: configure nginx so the WSGI mountpoint for /deposit is ok-ish.
Dec 10 2019, 1:16 PM
douardda committed rDENV22cb6ffb25c4: docker/compose: add rabbitmq_exporter service (authored by douardda).
docker/compose: add rabbitmq_exporter service
Dec 10 2019, 1:15 PM
douardda closed D2417: docker/compose: add rabbitmq_exporter service.
Dec 10 2019, 1:15 PM
douardda added a comment to T2124: Save Code Now: monitoring of admin infra.

prometheus_client looks quite low-level to me. We already have statsd for this kind of purpose, which does not imply having to deal with one more dedicated http server or so...

Dec 10 2019, 11:12 AM · Sprint 2019/12 (Monitor and Conquer)
douardda added a comment to T2134: loader: Implement uniform loading CLI.

Sorry I'm commenting this a bit late, but wouldn't it make more sense to have something like:

Dec 10 2019, 11:00 AM · Sprint 2019/12 (Monitor and Conquer)

Dec 9 2019

douardda committed rDENV3007fb37f47d: docker/compose: configure the statsd endpoint for most of the containers (authored by douardda).
docker/compose: configure the statsd endpoint for most of the containers
Dec 9 2019, 5:11 PM
douardda closed D2416: docker/compose: configure the statsd endpoint for most of the containers.
Dec 9 2019, 5:11 PM
douardda updated the diff for D2417: docker/compose: add rabbitmq_exporter service.

rebased

Dec 9 2019, 5:06 PM
douardda updated the diff for D2416: docker/compose: configure the statsd endpoint for most of the containers.

move statsd emv var declarations in an env/statsd.env file and use it on swh-web

Dec 9 2019, 5:06 PM
douardda added a comment to D2416: docker/compose: configure the statsd endpoint for most of the containers.

Great ! Could you also add that configuration to the swh-web service ? I am currently working on T2124 and I have added a couple of statsd metric to the webapp.

Dec 9 2019, 5:03 PM
douardda updated the diff for D2417: docker/compose: add rabbitmq_exporter service.

rebase

Dec 9 2019, 4:41 PM
douardda updated the diff for D2416: docker/compose: configure the statsd endpoint for most of the containers.

rebase

Dec 9 2019, 4:41 PM
douardda committed rDENVe4162838b388: docker/Dockefile: clean the $HOME/.cache/ at the end of the build (authored by douardda).
docker/Dockefile: clean the $HOME/.cache/ at the end of the build
Dec 9 2019, 4:40 PM
douardda closed D2415: docker/Dockefile: clean the $HOME/.cache/ at the end of the build.
Dec 9 2019, 4:40 PM
douardda committed rDENVbaeb787d8d9b: docker/nginx: use standard port 80 in nginx's container (authored by douardda).
docker/nginx: use standard port 80 in nginx's container
Dec 9 2019, 4:40 PM
douardda closed D2414: docker/nginx: use standard port 80 in nginx's container.
Dec 9 2019, 4:40 PM
douardda committed rDENV57ab82a18cd2: Rename swh-scheduler-api as swh-scheduler and swh-vault-api as swh-vault (authored by douardda).
Rename swh-scheduler-api as swh-scheduler and swh-vault-api as swh-vault
Dec 9 2019, 4:40 PM
douardda closed D2413: Rename swh-scheduler-api as swh-scheduler and swh-vault-api as swh-vault.
Dec 9 2019, 4:40 PM
douardda updated the diff for D2415: docker/Dockefile: clean the $HOME/.cache/ at the end of the build.

rebased

Dec 9 2019, 4:38 PM
douardda updated the diff for D2414: docker/nginx: use standard port 80 in nginx's container.

rebased

Dec 9 2019, 4:37 PM
douardda updated the diff for D2413: Rename swh-scheduler-api as swh-scheduler and swh-vault-api as swh-vault.

forgot a few matches, as noticed by anlambert

Dec 9 2019, 4:34 PM
douardda added a comment to D2413: Rename swh-scheduler-api as swh-scheduler and swh-vault-api as swh-vault.

I totally agree on this. You miss some replacements in the tests and the README though.

Dec 9 2019, 4:29 PM
douardda created D2417: docker/compose: add rabbitmq_exporter service.
Dec 9 2019, 4:26 PM
douardda created D2416: docker/compose: configure the statsd endpoint for most of the containers.
Dec 9 2019, 4:26 PM
douardda created D2415: docker/Dockefile: clean the $HOME/.cache/ at the end of the build.
Dec 9 2019, 4:25 PM
douardda created D2414: docker/nginx: use standard port 80 in nginx's container.
Dec 9 2019, 4:25 PM
douardda created D2413: Rename swh-scheduler-api as swh-scheduler and swh-vault-api as swh-vault.
Dec 9 2019, 4:23 PM
douardda moved T2122: Lister monitoring: add statsd probe for each lister instance from in progress to Backlog on the Sprint 2019/12 (Monitor and Conquer) board.
Dec 9 2019, 1:51 PM · Sprint 2019/12 (Monitor and Conquer)
douardda added a comment to T2122: Lister monitoring: add statsd probe for each lister instance.

Okay I see no "simple" way of doing this with the current implementation of tasks using SWHTask.

Dec 9 2019, 12:32 PM · Sprint 2019/12 (Monitor and Conquer)

Dec 6 2019

douardda moved T2122: Lister monitoring: add statsd probe for each lister instance from Backlog to in progress on the Sprint 2019/12 (Monitor and Conquer) board.
Dec 6 2019, 10:36 AM · Sprint 2019/12 (Monitor and Conquer)
douardda closed T2133: Scheduler listener/runner: add statsd probes as Resolved.

Closed by D2394

Dec 6 2019, 10:22 AM · Metrics/monitoring, Scheduling utilities, Sprint 2019/12 (Monitor and Conquer)
douardda committed rDSCHa18f56205d1d: celery: add 2 statsd probes for the runner and listener (authored by douardda).
celery: add 2 statsd probes for the runner and listener
Dec 6 2019, 10:22 AM
douardda closed D2394: celery: add 2 statsd probes for the runner and listener.
Dec 6 2019, 10:22 AM

Dec 5 2019

douardda accepted D2395: package.archive: Move archive loader to its own namespace.

LGTM

Dec 5 2019, 12:04 PM · SWH command line interface, Core Loader

Dec 4 2019

douardda updated the diff for D2394: celery: add 2 statsd probes for the runner and listener.

ok, let's keep _scheduler_ in probe names

Dec 4 2019, 4:37 PM
douardda added inline comments to D2394: celery: add 2 statsd probes for the runner and listener.
Dec 4 2019, 4:20 PM
douardda updated the diff for D2394: celery: add 2 statsd probes for the runner and listener.

I said -scheduler !

Dec 4 2019, 4:18 PM
douardda updated the diff for D2394: celery: add 2 statsd probes for the runner and listener.

rebased + rename statsd probes (-scheduler/+total)

Dec 4 2019, 3:27 PM
douardda committed rDSCH08243bbc9e3e: tests: fix celery_task's test_multiping (authored by douardda).
tests: fix celery_task's test_multiping
Dec 4 2019, 3:12 PM
douardda committed rDSCHf206076232a8: celery: make SWHTask send start/end of execution statsd gauges with timestamps (authored by douardda).
celery: make SWHTask send start/end of execution statsd gauges with timestamps
Dec 4 2019, 3:12 PM
douardda closed T2119: Monitoring of workers as Resolved by committing rDSCHf206076232a8: celery: make SWHTask send start/end of execution statsd gauges with timestamps.
Dec 4 2019, 3:12 PM · Scheduling utilities, Sprint 2019/12 (Monitor and Conquer)
douardda closed D2393: celery: make SWHTask send start/end of execution statsd gauges with timestamps.
Dec 4 2019, 3:12 PM
douardda added a revision to T2133: Scheduler listener/runner: add statsd probes: D2394: celery: add 2 statsd probes for the runner and listener.
Dec 4 2019, 10:58 AM · Metrics/monitoring, Scheduling utilities, Sprint 2019/12 (Monitor and Conquer)
douardda created D2394: celery: add 2 statsd probes for the runner and listener.
Dec 4 2019, 10:58 AM
douardda created D2393: celery: make SWHTask send start/end of execution statsd gauges with timestamps.
Dec 4 2019, 10:30 AM
douardda added a revision to T2119: Monitoring of workers: D2393: celery: make SWHTask send start/end of execution statsd gauges with timestamps.
Dec 4 2019, 10:30 AM · Scheduling utilities, Sprint 2019/12 (Monitor and Conquer)

Dec 3 2019

douardda moved T2119: Monitoring of workers from Backlog to in progress on the Sprint 2019/12 (Monitor and Conquer) board.
Dec 3 2019, 11:37 AM · Scheduling utilities, Sprint 2019/12 (Monitor and Conquer)

Nov 29 2019

douardda committed rDJNL3bb1d771c4bc: replay: use tag based statsd probes (authored by douardda).
replay: use tag based statsd probes
Nov 29 2019, 3:35 PM
douardda closed D2377: replay: use tag based statsd probes.
Nov 29 2019, 3:35 PM
douardda created D2377: replay: use tag based statsd probes.
Nov 29 2019, 3:10 PM
douardda committed rDJNL3f61999d2d08: replay: add a bit of statsd probing in process_replay_objects (authored by douardda).
replay: add a bit of statsd probing in process_replay_objects
Nov 29 2019, 11:49 AM
douardda closed D2374: replay: add a bit of statsd probing in process_replay_objects.
Nov 29 2019, 11:49 AM
douardda updated the diff for D2374: replay: add a bit of statsd probing in process_replay_objects.

rebased

Nov 29 2019, 11:46 AM
douardda committed rDJNL0177a9272bbe: replay: ensure a revision is present only once in a batch (authored by douardda).
replay: ensure a revision is present only once in a batch
Nov 29 2019, 11:45 AM
douardda closed D2372: replay: ensure a revision is present only once in a batch.
Nov 29 2019, 11:45 AM
douardda accepted D2370: Hide history link for empty snapshot.

LGTM but I'm not very used to the webapp's code.

Nov 29 2019, 11:04 AM
douardda requested changes to D2371: tarball: Add support for tar.Z, tar.x, tar.lz files.
Nov 29 2019, 11:01 AM
douardda accepted D2376: docker/README: drop obsolete mentions of swh-docker-dev.

looks very good to me, thanks!

Nov 29 2019, 10:12 AM

Nov 28 2019

douardda created D2374: replay: add a bit of statsd probing in process_replay_objects.
Nov 28 2019, 5:20 PM
douardda accepted D2367: pypi: metadata -> revision: Deal with previous metadata format.
Nov 28 2019, 5:14 PM
douardda created D2372: replay: ensure a revision is present only once in a batch.
Nov 28 2019, 4:16 PM
douardda created P564 revisions in journal.
Nov 28 2019, 4:04 PM
douardda accepted D2365: npm: metadata -> revision: Deal with previous metadata format.
Nov 28 2019, 11:55 AM

Nov 27 2019

douardda accepted D2360: sphinx/Dockerfile: Add missing pre-commit dependency.

Not very fond, but will work and remains the simplest solution.

Nov 27 2019, 10:19 AM

Nov 26 2019

douardda committed rDENV12931aff1a94: Dockerfile: fix a typo in var substitution (authored by douardda).
Dockerfile: fix a typo in var substitution
Nov 26 2019, 3:42 PM
douardda committed rDENV74252ff5d8cc: Allow to set CASS_DRIVER_ env vars when building the docker image (authored by douardda).
Allow to set CASS_DRIVER_ env vars when building the docker image
Nov 26 2019, 3:37 PM
douardda committed rDENV78ee2eeb7ae2: Stop referencing swh-docker-dev in .mrconfig (authored by douardda).
Stop referencing swh-docker-dev in .mrconfig
Nov 26 2019, 1:48 PM
douardda committed rDENV357084c46082: Fix entrypoint of most services (authored by douardda).
Fix entrypoint of most services
Nov 26 2019, 1:39 PM
douardda closed T2097: Merge swh-docker-dev into swh-environment as Resolved.

Closed by 9a98e8be06817055693671cfbe6917645171993e

Nov 26 2019, 1:36 PM · Docker environment, Development environment
douardda committed rDENVc8591006d03b: Dockerfile: get rid of swh-loader-xxx for packages loaders (authored by douardda).
Dockerfile: get rid of swh-loader-xxx for packages loaders
Nov 26 2019, 1:35 PM
douardda committed rDENV9a98e8be0681: Merge 'swh-docker-dev' into master (authored by douardda).
Merge 'swh-docker-dev' into master
Nov 26 2019, 1:35 PM
douardda committed rDENV3b0c74bc80ba: Dockerfile: install cassandra-driver in a dedicated step (authored by douardda).
Dockerfile: install cassandra-driver in a dedicated step
Nov 26 2019, 1:35 PM
douardda committed rDENV7b2e570a57da: Dockerfile: use --no-install-recommends for r-cran packages (authored by douardda).
Dockerfile: use --no-install-recommends for r-cran packages
Nov 26 2019, 1:35 PM
douardda committed rDENV9b12218d59cd: entrypoints: ensure gunicorn services can handle more than one connection (authored by douardda).
entrypoints: ensure gunicorn services can handle more than one connection
Nov 26 2019, 1:35 PM
douardda committed rDENV8d48959f9643: Dockerfile: update to python 3.7 (authored by douardda).
Dockerfile: update to python 3.7
Nov 26 2019, 1:35 PM
douardda committed rDENV00d3f8f35840: README: update doc to use the new 'swh' master cli tool (authored by douardda).
README: update doc to use the new 'swh' master cli tool
Nov 26 2019, 1:35 PM
douardda committed rDENV30e9c934ef20: lister: allways run the db initialization step (authored by douardda).
lister: allways run the db initialization step
Nov 26 2019, 1:35 PM
douardda committed rDENVb7f02d8add7f: dockerfile: install python packages in a venv under the swh user (authored by douardda).
dockerfile: install python packages in a venv under the swh user
Nov 26 2019, 1:35 PM
douardda committed rDENV53fa588c7fc2: Factorise the pip install of source repos in a shell function (authored by douardda).
Factorise the pip install of source repos in a shell function
Nov 26 2019, 1:35 PM
douardda committed rDENV6d49580cd56c: replica: ensure entrypoints are defined in replica related services (authored by douardda).
replica: ensure entrypoints are defined in replica related services
Nov 26 2019, 1:35 PM
douardda committed rDENV7b0ea2803606: Remove entrypoint.sh files from the Dockerfile (authored by douardda).
Remove entrypoint.sh files from the Dockerfile
Nov 26 2019, 1:35 PM
douardda committed rDENV6558c6c8ad18: entrypoint: pgsql has been moved to ~swh/ (authored by douardda).
entrypoint: pgsql has been moved to ~swh/
Nov 26 2019, 1:35 PM
douardda committed rDENVd1f5b87ba8c7: Dockerfile: small readability refactoring (authored by douardda).
Dockerfile: small readability refactoring
Nov 26 2019, 1:35 PM
douardda committed rDENV587f9df57a80: lister: Use explicit postgresql URLs as a workaround for a sqlalchemy issue (authored by douardda).
lister: Use explicit postgresql URLs as a workaround for a sqlalchemy issue
Nov 26 2019, 1:35 PM
douardda committed rDENV9359da6dbf95: Add support for a kafka-powered replica of the graph. (authored by douardda).
Add support for a kafka-powered replica of the graph.
Nov 26 2019, 1:35 PM
douardda committed rDENV20b0733b84f3: swh-web: enforce the database file to be created in either $HOME or /tmp (authored by douardda).
swh-web: enforce the database file to be created in either $HOME or /tmp
Nov 26 2019, 1:35 PM
douardda committed rDENV8a2458ee2bf0: compose: fix inconsistent dependency (authored by douardda).
compose: fix inconsistent dependency
Nov 26 2019, 1:35 PM
douardda committed rDENVc56678130229: readme: integrate the docker-based development setup from the main's doc (authored by douardda).
readme: integrate the docker-based development setup from the main's doc
Nov 26 2019, 1:35 PM
douardda committed rDENV6417e87e38e0: readme: make it compliant with phabricator's limited remarkup format (authored by douardda).
readme: make it compliant with phabricator's limited remarkup format
Nov 26 2019, 1:35 PM
douardda committed rDENV8871e9cd32eb: swh-scheduler: add SWH_CONFIG_FILENAME and SWH_SCHEDULER_CONFIG_FILE env vars (authored by douardda).
swh-scheduler: add SWH_CONFIG_FILENAME and SWH_SCHEDULER_CONFIG_FILE env vars
Nov 26 2019, 1:35 PM
douardda committed rDENVd192c1cde3d7: docker-compose: move the swh-indexer-journal-client in the journal "section" (authored by douardda).
docker-compose: move the swh-indexer-journal-client in the journal "section"
Nov 26 2019, 1:35 PM
douardda committed rDENV0b031ba074df: web: remove djangosettings.py file and use swh.web.settings.development (authored by douardda).
web: remove djangosettings.py file and use swh.web.settings.development
Nov 26 2019, 1:35 PM