Query: Advanced Search

	Include stories about projects I am a member of.

The cluster is configured with the default value for cluster.max_shards_per_nodes [1] so it can have 3000 shards opened (1000*3)

I temporary unblock the ingestion by closing the systemlogs indexes created before 2020-07-01:

curl -s http://$ES_NODE/_cat/indices\?s=index | grep -v close | grep systemlogs | awk '{print $3}' | grep 2020.05 | xargs -n1 -t -i{} curl -XPOST http://${ES_NODE}/{}/_close 
curl -s http://$ES_NODE/_cat/indices\?s=index | grep -v close | grep systemlogs | awk '{print $3}' | grep 2020.06 | xargs -n1 -t -i{} curl -XPOST http://${ES_NODE}/{}/_close

it seems we have reached a limit on the cluster (from the logstash logs) :

Apr 08 10:30:24 logstash0 logstash[1605158]: [2021-04-08T10:30:24,052][WARN ][logstash.outputs.elasticsearch][main][62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"systemlogs-2021.04.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x2ec8df34>], :response=>{"index"=>{"_index"=>"systemlogs-2021.04.08", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}}
Apr 08 10:30:24 logstash0 logstash[1605158]: [2021-04-08T10:30:24,052][WARN ][logstash.outputs.elasticsearch][main][62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"systemlogs-2021.04.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x26cfcf58>], :response=>{"index"=>{"_index"=>"systemlogs-2021.04.08", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}}
Apr 08 10:30:24 logstash0 logstash[1605158]: [2021-04-08T10:30:24,053][WARN ][logstash.outputs.elasticsearch][main][62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"systemlogs-2021.04.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1ddd876d>], :response=>{"index"=>{"_index"=>"systemlogs-2021.04.08", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}}

attempt to fix the stable debian build

Thanks for your tips to add the option --build-failed-commands %SBUILD_SHELL to the gbp command to get a command line on the chroot environement when a build failed.

declare the counters stack

add missing parts for counters

add scheduler journal client

Add search journal client

Add missing dockerfile for the lister's image

reorder skaffold configuration

Increase storage limit to avoid OOM

add missing point on the documentation

configure the webapp to use postgresql

Improve elasticsearch startup

Drop scheduler-runner verbosity to INFO

rabbitmq: Expose the rabbitmq manager port

Add swh-scheduler-listener service

Add swh search service

fix postgresql persistence

Add prometheus and grafana

Add elasticsearch service

Add scheduler-runner service

add directory initialisation task

allow kafka to start without being killed by an oom

webapp: configure rolling upgrade

persist zookeeper data to fix kafka restart

webapp: correctly install source via pip

Make webapp use an entrypoint and update to use postgres backend

objstorage: correctly install source via pip

Dockerfile: Clean up whitespaces

loaders: Decrease loaders verbosity

add service declaration on the /etc/hosts

add util scripts on the base image

Add rabbitmq service

README: Fix and unify host mount points creation

Reference how to build the objstorage

Add scheduler service

Readme: Update reference names

Cleanup whitespace noises

docker: stop ignoring .git directory to allow pip to perform local install

restore missing python dependencies

Specify the righ patch creation on the doc

fix zookeeper configuration

storage: Initialize database schema on startup

add limit on containers

fix database startup

make modules discoverable for swh command line

add skaffold section on the readme

split image to efficiently manage dependencies

wip - kubernetes and skaffold initialization

Reopening as the release is not working on the stable branch

Use an intermediate temporary file to generate the historical data

Allow the webapp to retrieve the history file via a GET endpoint

Return a 404 if the requested file does not exist
Use a fixture to configure the tests to make them easier to read

add python3-requests-mock dependency

Manage and expose the historical data

update according the review's feedbacks

Add staging environment diagram

fix typos on the commit message

The requested change was make by the DSI. Everything is working well now.

Allow to use several backends with a RPCServerApp

In D5428#137737, @ardumont wrote:

Why not a single backend class for both?

because it's not the same backend implementation
so not the same concern/perimeter as mentioned in the diff description.

The wrong network profile was asked for the staging gateway so it seems it doesn't have a complete access to internet.
A mail was sent to the DSI to request an unfiltered access.

Advanced Search
Use Results
Edit Query
Hide Query

Apr 8 2021

Apr 7 2021

Apr 6 2021

Apr 2 2021

Advanced SearchUse ResultsEdit QueryHide Query

Apr 8 2021

Apr 7 2021

Apr 6 2021

Apr 2 2021

Advanced Search
Use Results
Edit Query
Hide Query