A search query to show candidates for this label: https://forge.softwareheritage.org/maniphest/query/Z8t6ZkjPI7JV/ (ie. it filters out high-priority tasks, tasks with someone already assigned to it, and sysadmin tasks)
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Dec 13 2018
Dec 13 2018
vlorentz added a project to T1438: Add labels to prometheus metrics to help queries: Metrics/monitoring.
Dec 12 2018
Dec 12 2018
Feature request from epol on IRC: being able to selectively listen only for the graph and not the object changes. (I guess this will be the case anyway, I'm just mentioning that feedback on the task.)
zack renamed T1411: reach a minimum of 80% SLOC coverage across all components from 80% SLOC coverage to at least 80% SLOC coverage in all components.
zack moved T1436: Integrate swh-storage metrics in prometheus from Backlog to deployed on the Sprint 2018 12 board.
zack moved T1422: drop swh-storage mocking everywhere from Backlog to deployed on the Sprint 2018 12 board.
ardumont moved T1439: loader-mercurial: Improve coverage from done to deployed on the Sprint 2018 12 board.
ardumont moved T1431: Add tests for the Tar loader. from done to deployed on the Sprint 2018 12 board.
ardumont moved T1439: loader-mercurial: Improve coverage from in progress to done on the Sprint 2018 12 board.
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
ardumont closed T1439: loader-mercurial: Improve coverage, a subtask of T1411: reach a minimum of 80% SLOC coverage across all components, as Resolved.
Dec 11 2018
Dec 11 2018
Added draft dashboard: https://grafana.softwareheritage.org/d/3SAW_JEmk/software-heritage-archive-counters
olasd closed T1436: Integrate swh-storage metrics in prometheus as Resolved by committing rSPSITE51b1d1c267c4: Use the proper column name for the value of the counters.
olasd closed T1436: Integrate swh-storage metrics in prometheus, a subtask of T1408: More/better Metrics, as Resolved.
As software is mostly declared in puppet, I think the main areas that could be improved would be
- hardware inventory
- network topology
- puppet reports integration
olasd added a parent task for T1436: Integrate swh-storage metrics in prometheus: T1355: Move the object counter from munin to prometheus.
Dec 10 2018
Dec 10 2018
olasd added a subtask for T1356: Kill munin: T1355: Move the object counter from munin to prometheus.
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
ardumont closed T1431: Add tests for the Tar loader., a subtask of T1411: reach a minimum of 80% SLOC coverage across all components, as Resolved.
ardumont moved T1431: Add tests for the Tar loader. from in progress to done on the Sprint 2018 12 board.
ardumont raised the priority of this task from Normal to High.
That won't capture all events, but the easiest solution is to use Jenkins' RSS: https://jenkins.softwareheritage.org/view/swh/rssAll
Dec 7 2018
Dec 7 2018
vlorentz moved T1421: drop swh-storage mocking everywhere from done to in progress on the Sprint 2018 12 board.
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
vlorentz moved T1426: Select a tool + define an architecture to run locally a full SWH instance from Backlog to in progress on the Sprint 2018 12 board.
vlorentz moved T1428: Create an inventory of useful Munin metrics from Backlog to in progress on the Sprint 2018 12 board.
vlorentz moved T1405: Make it easy to run a complete swh instance from Backlog to in progress on the Sprint 2018 12 board.
vlorentz moved T1421: drop swh-storage mocking everywhere from in progress to done on the Sprint 2018 12 board.
| Munin metric | Comment | Prometheus metric combination | Prometheus comment |
| Disk | |||
| I/Os per device | node_disk_reads_completed_total; node_disk_writes_completed_total | Add derivative to get IOPS | |
| Disk usage in percent (space) | (node_filesystem_size_bytes - node_filesystem_{avail,free}_bytes) / node_filesystem_size_bytes | avail = available to non-root, free = available to root (tune2fs -m / reserved-blocks-percentage) | |
| Disk usage in percent (inodes) | (node_filesystem_files - node_filesystem_files_free) / node_filesystem_files | ||
| Utilization per device | is this real ? it could be useful to see if a storage subsystem is overloaded | node_disk_io_time_seconds_total | total time spent in seconds doing IO on the specified device; AFAICT the derivative of this counter is what munin calls "utilization per device" |
| node_disk_io_time_weighted_seconds_total | counts the number of seconds spent doing IO multiplied by the number of concurrent IO requests; maybe more relevant ? Docs: https://www.kernel.org/doc/Documentation/iostats.txt | ||
| Disk usage in absolute human values. | percentages are meaningless if we resize filesystems | node_filesystem_size_bytes - node_filesystem_{avail,free}_bytes | avail = available to non-root, free = available to root |
| Networking | |||
| eth0 traffic | node_network_receive_bytes_total; node_network_transmit_bytes_total | derivative for bytes per second | |
| node_network_receive_packets_total; node_network_transmit_packets_total | derivative for packets per second | ||
| node_network_receive_errs_total; node_network_transmit_errs_total | alert if non-zero | ||
| Database | |||
| implemented with prometheus-sql-exporter | |||
| Postgres replication lag | sql_pg_stat_replication{col=~'(send_lag_bytes,flush_lag_bytes,replay_lag_bytes)'} | replace commas with pipes... | |
| Postgres database size | sql_pg_stat_database{col="dbsize"} | ||
| Postgres oldest transaction | sql_pg_stat_activity{col="max_tx_duration"} | ||
| Postgres oldest query | ? | ||
| Postgres scan types (sequential / indexed) | sql_pg_stat_user_tables;sql_pg_statio_user_tables | ||
| Postgres wal segments | sql_archive_ready; sql_pg_stat_archiver | use derivative of sql_pg_stat_archiver values to get archival rates | |
| Postgres nb. of transactions | sql_txid | derivative to get tps | |
| System | |||
| CPU usage | node_cpu_seconds_total | use derivative for CPU usage | |
| load average | node_load{1,5,15} | ||
| Memory usage | node_memory_* | ||
| Pending packages | XXX | needs to be implemented with the textfile collector (see /usr/share/doc/prometheus-node-exporter/examples/text_collector_examples/apt.sh) | |
| Swap in/out | node_vmstat_pswpin; node_vmstat_pswpout | unit ?? probably absolute number of pages | |
| Uptime | time() - node_boot_time_seconds | ||
| RabbitMQ | |||
| use https://github.com/kbudde/rabbitmq_exporter or https://github.com/deadtrickster/prometheus_rabbitmq_exporter | |||
| Consumers | |||
| Memory used by queue | |||
| Unacknowledged messages | |||
| Nb. of connections | |||
| Softwareheritage (prado) | |||
| Almost everything | integrate to sql-exporter configuration | ||
| Most importantly Software Heritage Objects |
Dec 6 2018
Dec 6 2018
what kind of inventory we want to do with this? hardware? software? both?
Dec 5 2018
Dec 5 2018
vlorentz moved T1421: drop swh-storage mocking everywhere from Backlog to in progress on the Sprint 2018 12 board.
vlorentz triaged T1406: Documentation/tutorial for using public datasets (Athena/AWS) as Normal priority.
vlorentz raised the priority of T1428: Create an inventory of useful Munin metrics from Normal to High.
For the loader mercurial, there is a module swh.loader.mercurial.loader_verifier which is not production code.
It's there to test the loader manually, so that could either be probably moved to the tests and transformed into it or removed altogether.
vlorentz triaged T1411: reach a minimum of 80% SLOC coverage across all components as Normal priority.
vlorentz closed T1393: swh/loader/core/loader.py has no proper test coverage, a subtask of T1411: reach a minimum of 80% SLOC coverage across all components, as Resolved.
vlorentz updated subscribers of T1426: Select a tool + define an architecture to run locally a full SWH instance.
Dec 4 2018
Dec 4 2018
ftigeot changed the status of T1428: Create an inventory of useful Munin metrics from Open to Work in Progress.
Disk
- I/Os per device
- Disk usage in percent
- Utilization per device is this real ? it could be useful to see if a storage subsystem is overloaded
- Disk usage in absolute human values. percentages are meaningless if we resize filesystems
ftigeot changed the status of T1428: Create an inventory of useful Munin metrics, a subtask of T1408: More/better Metrics, from Open to Work in Progress.
Tip: after running Tox in a repo, run coverage report -m to show which lines are not covered.
ardumont added a subtask for T1425: refactor the loader stack for package managers: T1379: npm loader.
vlorentz triaged T1426: Select a tool + define an architecture to run locally a full SWH instance as High priority.
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
vlorentz added a subtask for T1421: drop swh-storage mocking everywhere: T1307: Remove mock storages used in tests..
Duplicate of T1421
vlorentz added a subtask for T1422: drop swh-storage mocking everywhere: T1307: Remove mock storages used in tests..