Page MenuHomeSoftware Heritage
Feed All Stories

Dec 8 2018

swh-public-ci added a comment to D794: loader.tar: Add some tests and remove dead code.

Build has FAILED

Dec 8 2018, 11:28 AM
ardumont updated the diff for D794: loader.tar: Add some tests and remove dead code.

Fix docstring typo

Dec 8 2018, 11:27 AM
ardumont added a comment to D794: loader.tar: Add some tests and remove dead code.

Are you sure these tests pass when the other Diff is merged?

Dec 8 2018, 11:25 AM
vlorentz accepted D793: utils.grouper: Improve implementation.
Dec 8 2018, 11:22 AM
Harbormaster failed remote builds in B2941: Diff 2520 for D794: loader.tar: Add some tests and remove dead code!
Dec 8 2018, 11:18 AM
swh-public-ci added a comment to D794: loader.tar: Add some tests and remove dead code.

Build has FAILED

Dec 8 2018, 11:18 AM
ardumont updated the diff for D794: loader.tar: Add some tests and remove dead code.
  • utils: Simplify random_block according to latest api change
Dec 8 2018, 11:17 AM
ardumont updated the summary of D793: utils.grouper: Improve implementation.
Dec 8 2018, 11:13 AM
swh-public-ci added a comment to D793: utils.grouper: Improve implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DCORE/job/tox/98/ for more details.

Dec 8 2018, 11:13 AM
ardumont updated the diff for D793: utils.grouper: Improve implementation.
  • utils.grouper: Rename fillvalue to stop_value and fix docstring
  • utils.grouper: Avoid potential bug of input data matching stop_value
Dec 8 2018, 11:12 AM
ardumont added inline comments to D793: utils.grouper: Improve implementation.
Dec 8 2018, 11:10 AM
vlorentz added inline comments to D793: utils.grouper: Improve implementation.
Dec 8 2018, 11:00 AM
ardumont added inline comments to D793: utils.grouper: Improve implementation.
Dec 8 2018, 10:54 AM
vlorentz added a comment to D794: loader.tar: Add some tests and remove dead code.

Are you sure these tests pass when the other Diff is merged?

Dec 8 2018, 9:45 AM
vlorentz requested changes to D793: utils.grouper: Improve implementation.
Dec 8 2018, 9:30 AM

Dec 7 2018

Harbormaster failed remote builds in B2939: Diff 2518 for D794: loader.tar: Add some tests and remove dead code!
Dec 7 2018, 8:05 PM
swh-public-ci added a comment to D794: loader.tar: Add some tests and remove dead code.

Build has FAILED

Dec 7 2018, 8:05 PM
ardumont added a reviewer for D793: utils.grouper: Improve implementation: vlorentz.
Dec 7 2018, 8:04 PM
ardumont created D794: loader.tar: Add some tests and remove dead code.
Dec 7 2018, 8:04 PM
swh-public-ci added a comment to D793: utils.grouper: Improve implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DCORE/job/tox/64/ for more details.

Dec 7 2018, 7:13 PM
Herald added a reviewer for D793: utils.grouper: Improve implementation: Reviewers.
Dec 7 2018, 7:11 PM
vlorentz moved T1421: drop swh-storage mocking everywhere from done to in progress on the Sprint 2018 12 board.
Dec 7 2018, 6:37 PM · Sprint 2018 12
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
Dec 7 2018, 6:36 PM · Development environment, Sprint 2018 12
ardumont updated the task description for T1411: reach a minimum of 80% SLOC coverage across all components.
Dec 7 2018, 6:35 PM · Development environment, Sprint 2018 12
vlorentz committed rDVAUe3a72f814561: Bump required version of swh.loader.git (class rename). (authored by vlorentz).
Bump required version of swh.loader.git (class rename).
Dec 7 2018, 6:27 PM
vlorentz updated the task description for T1307: Remove mock storages used in tests..
Dec 7 2018, 6:21 PM · Storage manager
vlorentz closed T1432: Remove mock storages from the indexers, a subtask of T1307: Remove mock storages used in tests., as Resolved.
Dec 7 2018, 6:20 PM · Storage manager
vlorentz closed T1432: Remove mock storages from the indexers as Resolved.
Dec 7 2018, 6:20 PM · Storage manager
vlorentz claimed T1426: Select a tool + define an architecture to run locally a full SWH instance.
Dec 7 2018, 6:20 PM · Sprint 2018 12
vlorentz moved T1426: Select a tool + define an architecture to run locally a full SWH instance from Backlog to in progress on the Sprint 2018 12 board.
Dec 7 2018, 6:19 PM · Sprint 2018 12
vlorentz moved T1428: Create an inventory of useful Munin metrics from Backlog to in progress on the Sprint 2018 12 board.
Dec 7 2018, 6:19 PM · Metrics/monitoring, Sprint 2018 12
vlorentz moved T1405: Make it easy to run a complete swh instance from Backlog to in progress on the Sprint 2018 12 board.
Dec 7 2018, 6:19 PM · Docker environment, Sprint 2018 12
vlorentz moved T1421: drop swh-storage mocking everywhere from in progress to done on the Sprint 2018 12 board.
Dec 7 2018, 6:19 PM · Sprint 2018 12
vlorentz committed rCDFD389705973b6d: initial commit. (authored by vlorentz).
initial commit.
Dec 7 2018, 5:45 PM
swh-public-ci added a comment to D792: Kill mocks..

Build has FAILED

Dec 7 2018, 5:38 PM
Harbormaster failed remote builds in B2935: Diff 2515 for D792: Kill mocks.!
Dec 7 2018, 5:38 PM
vlorentz committed rDCIDX58d641f05165: Kill mocks. (authored by vlorentz).
Kill mocks.
Dec 7 2018, 5:37 PM
vlorentz closed D792: Kill mocks..
Dec 7 2018, 5:37 PM
vlorentz updated the diff for D792: Kill mocks..
  • rebase
Dec 7 2018, 5:37 PM
vlorentz committed rDCIDX4629e31f51d3: Remove mock storages from test_origin_metadata.py. (authored by vlorentz).
Remove mock storages from test_origin_metadata.py.
Dec 7 2018, 5:36 PM
vlorentz closed D791: Remove mock storages from test_origin_metadata.py..
Dec 7 2018, 5:36 PM
ardumont accepted D791: Remove mock storages from test_origin_metadata.py..
Dec 7 2018, 5:35 PM
ardumont accepted D792: Kill mocks..
Dec 7 2018, 5:33 PM
vlorentz added a task to D783: rm ctags mocks + add ctags to idx db + fix doc.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added revisions to T1307: Remove mock storages used in tests.: D792: Kill mocks., D791: Remove mock storages from test_origin_metadata.py., D790: Add in-mem storage for intrinsic metadata., D789: Remove languages mock., D788: Remove mocks from range tests of mimetype and license indexers., D786: Add content_fossology_license_get_range + Remove mock from non-range tests., D784: Add content_fossology_license_{get,add} to the mem idx storage., D785: Start removing mocks for the mimetype indexer., D787: Remove mocks from range tests of mimetype and license indexers., D783: rm ctags mocks + add ctags to idx db + fix doc..
Dec 7 2018, 5:13 PM · Storage manager
vlorentz added a task to D787: Remove mocks from range tests of mimetype and license indexers.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D785: Start removing mocks for the mimetype indexer.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D784: Add content_fossology_license_{get,add} to the mem idx storage.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D786: Add content_fossology_license_get_range + Remove mock from non-range tests.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D788: Remove mocks from range tests of mimetype and license indexers.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D789: Remove languages mock.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D790: Add in-mem storage for intrinsic metadata.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D791: Remove mock storages from test_origin_metadata.py.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
vlorentz added a task to D792: Kill mocks.: T1307: Remove mock storages used in tests..
Dec 7 2018, 5:13 PM
swh-public-ci added a comment to D792: Kill mocks..

Build has FAILED

Dec 7 2018, 5:12 PM
Harbormaster failed remote builds in B2933: Diff 2513 for D792: Kill mocks.!
Dec 7 2018, 5:12 PM
Herald added a reviewer for D792: Kill mocks.: Reviewers.
Dec 7 2018, 5:11 PM
swh-public-ci added a comment to D791: Remove mock storages from test_origin_metadata.py..

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/153/ for more details.

Dec 7 2018, 5:10 PM
vlorentz committed rDDOC8b2258d85b06: Document {intrinsic,extrinsic} metadata. (authored by vlorentz).
Document {intrinsic,extrinsic} metadata.
Dec 7 2018, 5:08 PM
vlorentz closed D760: Document {intrinsic,extrinsic} metadata..
Dec 7 2018, 5:08 PM
vlorentz updated the diff for D760: Document {intrinsic,extrinsic} metadata..
  • rebase
Dec 7 2018, 5:08 PM
swh-public-ci added a comment to D790: Add in-mem storage for intrinsic metadata..

Build has FAILED

Dec 7 2018, 5:07 PM
Harbormaster failed remote builds in B2929: Diff 2508 for D790: Add in-mem storage for intrinsic metadata.!
Dec 7 2018, 5:07 PM
vlorentz updated the diff for D791: Remove mock storages from test_origin_metadata.py..
  • rebase
Dec 7 2018, 5:06 PM
swh-public-ci added a comment to D789: Remove languages mock..

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/151/ for more details.

Dec 7 2018, 5:06 PM
vlorentz committed rDCIDXa8bd3809b800: Add in-mem storage for intrinsic metadata. (authored by vlorentz).
Add in-mem storage for intrinsic metadata.
Dec 7 2018, 5:06 PM
vlorentz closed D790: Add in-mem storage for intrinsic metadata..
Dec 7 2018, 5:06 PM
vlorentz updated the diff for D790: Add in-mem storage for intrinsic metadata..
  • rebase
Dec 7 2018, 5:06 PM
vlorentz committed rDCIDX2862e3a95fb1: Remove languages mock. (authored by vlorentz).
Remove languages mock.
Dec 7 2018, 5:03 PM
vlorentz closed D789: Remove languages mock..
Dec 7 2018, 5:03 PM
vlorentz updated the diff for D789: Remove languages mock..
  • rebase
Dec 7 2018, 5:03 PM
swh-public-ci added a comment to D788: Remove mocks from range tests of mimetype and license indexers..

Build has FAILED

Dec 7 2018, 5:02 PM
Harbormaster failed remote builds in B2925: Diff 2504 for D788: Remove mocks from range tests of mimetype and license indexers.!
Dec 7 2018, 5:02 PM
vlorentz committed rDCIDXf8a2201906e8: Remove mocks from range tests of mimetype and license indexers. (authored by vlorentz).
Remove mocks from range tests of mimetype and license indexers.
Dec 7 2018, 5:02 PM
vlorentz closed D788: Remove mocks from range tests of mimetype and license indexers..
Dec 7 2018, 5:02 PM
vlorentz updated the diff for D788: Remove mocks from range tests of mimetype and license indexers..
  • squash
Dec 7 2018, 5:02 PM
swh-public-ci added a comment to D791: Remove mock storages from test_origin_metadata.py..

Build has FAILED

Dec 7 2018, 5:02 PM
Harbormaster failed remote builds in B2924: Diff 2503 for D791: Remove mock storages from test_origin_metadata.py.!
Dec 7 2018, 5:02 PM
Herald added a reviewer for D791: Remove mock storages from test_origin_metadata.py.: Reviewers.
Dec 7 2018, 5:01 PM
ardumont accepted D760: Document {intrinsic,extrinsic} metadata..
Dec 7 2018, 3:59 PM
olasd added a comment to T1428: Create an inventory of useful Munin metrics.
Munin metricCommentPrometheus metric combinationPrometheus comment
Disk
I/Os per devicenode_disk_reads_completed_total; node_disk_writes_completed_totalAdd derivative to get IOPS
Disk usage in percent (space)(node_filesystem_size_bytes - node_filesystem_{avail,free}_bytes) / node_filesystem_size_bytesavail = available to non-root, free = available to root (tune2fs -m / reserved-blocks-percentage)
Disk usage in percent (inodes)(node_filesystem_files - node_filesystem_files_free) / node_filesystem_files
Utilization per deviceis this real ? it could be useful to see if a storage subsystem is overloadednode_disk_io_time_seconds_totaltotal time spent in seconds doing IO on the specified device; AFAICT the derivative of this counter is what munin calls "utilization per device"
node_disk_io_time_weighted_seconds_totalcounts the number of seconds spent doing IO multiplied by the number of concurrent IO requests; maybe more relevant ? Docs: https://www.kernel.org/doc/Documentation/iostats.txt
Disk usage in absolute human values.percentages are meaningless if we resize filesystemsnode_filesystem_size_bytes - node_filesystem_{avail,free}_bytesavail = available to non-root, free = available to root
Networking
eth0 trafficnode_network_receive_bytes_total; node_network_transmit_bytes_totalderivative for bytes per second
node_network_receive_packets_total; node_network_transmit_packets_totalderivative for packets per second
node_network_receive_errs_total; node_network_transmit_errs_totalalert if non-zero
Database
implemented with prometheus-sql-exporter
Postgres replication lagsql_pg_stat_replication{col=~'(send_lag_bytes,flush_lag_bytes,replay_lag_bytes)'}replace commas with pipes...
Postgres database sizesql_pg_stat_database{col="dbsize"}
Postgres oldest transactionsql_pg_stat_activity{col="max_tx_duration"}
Postgres oldest query?
Postgres scan types (sequential / indexed)sql_pg_stat_user_tables;sql_pg_statio_user_tables
Postgres wal segmentssql_archive_ready; sql_pg_stat_archiveruse derivative of sql_pg_stat_archiver values to get archival rates
Postgres nb. of transactionssql_txidderivative to get tps
System
CPU usagenode_cpu_seconds_totaluse derivative for CPU usage
load averagenode_load{1,5,15}
Memory usagenode_memory_*
Pending packagesXXXneeds to be implemented with the textfile collector (see /usr/share/doc/prometheus-node-exporter/examples/text_collector_examples/apt.sh)
Swap in/outnode_vmstat_pswpin; node_vmstat_pswpoutunit ?? probably absolute number of pages
Uptimetime() - node_boot_time_seconds
RabbitMQ
use https://github.com/kbudde/rabbitmq_exporter or https://github.com/deadtrickster/prometheus_rabbitmq_exporter
Consumers
Memory used by queue
Unacknowledged messages
Nb. of connections
Softwareheritage (prado)
Almost everythingintegrate to sql-exporter configuration
Most importantly Software Heritage Objects
Dec 7 2018, 3:29 PM · Metrics/monitoring, Sprint 2018 12
ardumont accepted D790: Add in-mem storage for intrinsic metadata..
Dec 7 2018, 2:49 PM
ardumont accepted D788: Remove mocks from range tests of mimetype and license indexers..
Dec 7 2018, 2:48 PM
swh-public-ci added a comment to D790: Add in-mem storage for intrinsic metadata..

Build has FAILED

Dec 7 2018, 2:48 PM
Harbormaster failed remote builds in B2923: Diff 2502 for D790: Add in-mem storage for intrinsic metadata.!
Dec 7 2018, 2:48 PM
swh-public-ci added a comment to D790: Add in-mem storage for intrinsic metadata..

Build has FAILED

Dec 7 2018, 2:47 PM
Harbormaster failed remote builds in B2922: Diff 2501 for D790: Add in-mem storage for intrinsic metadata.!
Dec 7 2018, 2:47 PM
swh-public-ci added a comment to D788: Remove mocks from range tests of mimetype and license indexers..

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/146/ for more details.

Dec 7 2018, 2:47 PM
vlorentz updated the diff for D790: Add in-mem storage for intrinsic metadata..
  • rebase
Dec 7 2018, 2:47 PM
vlorentz updated the diff for D790: Add in-mem storage for intrinsic metadata..
  • Merge lines.
Dec 7 2018, 2:46 PM
vlorentz updated the diff for D788: Remove mocks from range tests of mimetype and license indexers..
  • add TODO
Dec 7 2018, 2:44 PM
ardumont added inline comments to D790: Add in-mem storage for intrinsic metadata..
Dec 7 2018, 2:38 PM
ardumont added inline comments to D788: Remove mocks from range tests of mimetype and license indexers..
Dec 7 2018, 2:36 PM
swh-public-ci added a comment to D790: Add in-mem storage for intrinsic metadata..

Build has FAILED

Dec 7 2018, 2:29 PM
Harbormaster failed remote builds in B2920: Diff 2499 for D790: Add in-mem storage for intrinsic metadata.!
Dec 7 2018, 2:29 PM
vlorentz updated the diff for D790: Add in-mem storage for intrinsic metadata..
  • remove useless function
Dec 7 2018, 2:28 PM
ardumont accepted D789: Remove languages mock..
Dec 7 2018, 2:28 PM
swh-public-ci added a comment to D790: Add in-mem storage for intrinsic metadata..

Build has FAILED

Dec 7 2018, 2:28 PM
Harbormaster failed remote builds in B2919: Diff 2498 for D790: Add in-mem storage for intrinsic metadata.!
Dec 7 2018, 2:28 PM
vlorentz updated the diff for D790: Add in-mem storage for intrinsic metadata..
  • rebase
Dec 7 2018, 2:27 PM