Query: Advanced Search

webapp use counters to display the history count graph

monitoring: use consistent names for exporter job

use a local storage for the registry

try to perform a warm shutdown of the lister and loaders

loaders: use a dynamic hostname

	Include stories about projects I am a member of.

Everything is released correctly and deployed on staging

I finally found why the graphs looks weird : https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/misc/urls.py$31
With a dirty patch on the server, it's way better:

fix a typo on the commit message

fix wrong usage of alias/lookup

staging: Fix the counters history url

The pipeline is deployed in staging.
It's working but it seems the graphs need some initial values in staging to make the rendering correctly:

staging: configure counters history pipeline

Add a filter to limit the metrics to the current environment

Fix history endpoint path

Let flask manage json response by itself

The cluster is configured with the default value for cluster.max_shards_per_nodes [1] so it can have 3000 shards opened (1000*3)

I temporary unblock the ingestion by closing the systemlogs indexes created before 2020-07-01:

curl -s http://$ES_NODE/_cat/indices\?s=index | grep -v close | grep systemlogs | awk '{print $3}' | grep 2020.05 | xargs -n1 -t -i{} curl -XPOST http://${ES_NODE}/{}/_close 
curl -s http://$ES_NODE/_cat/indices\?s=index | grep -v close | grep systemlogs | awk '{print $3}' | grep 2020.06 | xargs -n1 -t -i{} curl -XPOST http://${ES_NODE}/{}/_close

it seems we have reached a limit on the cluster (from the logstash logs) :

Apr 08 10:30:24 logstash0 logstash[1605158]: [2021-04-08T10:30:24,052][WARN ][logstash.outputs.elasticsearch][main][62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"systemlogs-2021.04.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x2ec8df34>], :response=>{"index"=>{"_index"=>"systemlogs-2021.04.08", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}}
Apr 08 10:30:24 logstash0 logstash[1605158]: [2021-04-08T10:30:24,052][WARN ][logstash.outputs.elasticsearch][main][62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"systemlogs-2021.04.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x26cfcf58>], :response=>{"index"=>{"_index"=>"systemlogs-2021.04.08", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}}
Apr 08 10:30:24 logstash0 logstash[1605158]: [2021-04-08T10:30:24,053][WARN ][logstash.outputs.elasticsearch][main][62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"systemlogs-2021.04.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1ddd876d>], :response=>{"index"=>{"_index"=>"systemlogs-2021.04.08", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}}

attempt to fix the stable debian build

Thanks for your tips to add the option --build-failed-commands %SBUILD_SHELL to the gbp command to get a command line on the chroot environement when a build failed.

declare the counters stack

add missing parts for counters

add scheduler journal client

Add search journal client

Add missing dockerfile for the lister's image

reorder skaffold configuration

Increase storage limit to avoid OOM

add missing point on the documentation

configure the webapp to use postgresql

Improve elasticsearch startup

Drop scheduler-runner verbosity to INFO

rabbitmq: Expose the rabbitmq manager port

Add swh-scheduler-listener service

Add swh search service

fix postgresql persistence

Add prometheus and grafana

Add elasticsearch service

Add scheduler-runner service

add directory initialisation task

allow kafka to start without being killed by an oom

webapp: configure rolling upgrade

persist zookeeper data to fix kafka restart

webapp: correctly install source via pip

Make webapp use an entrypoint and update to use postgres backend

objstorage: correctly install source via pip

Dockerfile: Clean up whitespaces

loaders: Decrease loaders verbosity

add service declaration on the /etc/hosts

add util scripts on the base image

Add rabbitmq service

README: Fix and unify host mount points creation

Reference how to build the objstorage

Add scheduler service

Readme: Update reference names

Cleanup whitespace noises

docker: stop ignoring .git directory to allow pip to perform local install

restore missing python dependencies

Specify the righ patch creation on the doc

fix zookeeper configuration

storage: Initialize database schema on startup

add limit on containers

fix database startup

make modules discoverable for swh command line

add skaffold section on the readme

split image to efficiently manage dependencies

wip - kubernetes and skaffold initialization

Advanced Search
Use Results
Edit Query
Hide Query

Apr 10 2021

Apr 9 2021

Apr 8 2021

Apr 7 2021

Advanced SearchUse ResultsEdit QueryHide Query

Apr 10 2021

Apr 9 2021

Apr 8 2021

Apr 7 2021

Advanced Search
Use Results
Edit Query
Hide Query