Page MenuHomeSoftware Heritage
Feed All Stories

All Stories
Use Results
Edit Query

Sep 17 2021

stsp committed rDLDCVScf5cf4594d05: link revisions to their parents and add another small test (authored by stsp).
link revisions to their parents and add another small test
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS5097bf8eaab1: use an iterator to process swh revisions; reduces memory usage (authored by stsp).
use an iterator to process swh revisions; reduces memory usage
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSed460a3280fe: do not call self.storage.revision_get() more often than necessary (authored by stsp).
do not call self.storage.revision_get() more often than necessary
Sep 17 2021, 2:49 PM
stsp committed rDLDCVScdbcdcc92e61: enable check_snapshot() in test_loader_cvs_visit() (authored by stsp).
enable check_snapshot() in test_loader_cvs_visit()
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSedc28c215955: fix rsync URL processing in fetch_cvs_repo_with_rsync() (authored by stsp).
fix rsync URL processing in fetch_cvs_repo_with_rsync()
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSae77fa161841: test two consecutive visits (authored by stsp).
test two consecutive visits
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS496ff263546a: the trivial first visit test is passing now (authored by stsp).
the trivial first visit test is passing now
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS358a05d59956: add converted CVS revisions to storage (authored by stsp).
add converted CVS revisions to storage
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS6a2f6e50f61e: ignore ENOENT when removing files (authored by stsp).
ignore ENOENT when removing files
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS59a7ff18d603: document fetch_data() method (authored by stsp).
document fetch_data() method
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS49ce9a24d6f9: create subdirectories in the work tree (authored by stsp).
create subdirectories in the work tree
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSaf30f67f7403: populate a work tree with files checked out from the repository (authored by stsp).
populate a work tree with files checked out from the repository
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS799eea5a7b68: get fetch_data working (authored by stsp).
get fetch_data working
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS15252a6ce822: force our local rcsparse dependency; upstream does not yet support py3 (authored by stsp).
force our local rcsparse dependency; upstream does not yet support py3
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS605d8f5862b8: get the 'prepare' step working (authored by stsp).
get the 'prepare' step working
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS757ce82f02eb: remove bogus import (authored by stsp).
remove bogus import
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS28cad58a0f32: add stub for an initial test (authored by stsp).
add stub for an initial test
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS9dec8a2e4fdc: rcsparse: add missing allocation failure check in parsetoken() (authored by stsp).
rcsparse: add missing allocation failure check in parsetoken()
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSaaec7d33bf74: remove the 'foo' module and references to it (authored by stsp).
remove the 'foo' module and references to it
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSecf402cf6249: compile rcsparse extension (authored by stsp).
compile rcsparse extension
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS9e3170f81c2b: rcsparse python3 support patches from OpenBSD ports (authored by stsp).
rcsparse python3 support patches from OpenBSD ports
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS75fe9f5ea285: Add 'swh/loader/cvs/rcsparse/' (authored by stsp).
Add 'swh/loader/cvs/rcsparse/'
Sep 17 2021, 2:49 PM
stsp committed rDLDCVSbb4362177743: Add 'swh/loader/cvs/cvs2gitdump/' (authored by stsp).
Add 'swh/loader/cvs/cvs2gitdump/'
Sep 17 2021, 2:49 PM
stsp closed D5988: initial CVS loader stub.
Sep 17 2021, 2:49 PM
stsp committed rDLDCVS65995f14b4c6: initial CVS loader stub (authored by stsp).
initial CVS loader stub
Sep 17 2021, 2:49 PM
anlambert accepted D6287: Allow gitlab lister's name to be overriden by task arguments.
Sep 17 2021, 2:40 PM
ardumont updated the task description for T3581: List heptapod instance foss.heptapod.net.
Sep 17 2021, 2:37 PM · Archive coverage, System administration, Origin-GitLab
vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

Commits ingested before https://forge.softwareheritage.org/rDLDGfb03140e169f38f5ea0d64d81b524b4b967335f7 have an extra newline at the end of the mergetag header

Sep 17 2021, 2:37 PM · Archive content, Restricted Project
ardumont updated the task description for T3581: List heptapod instance foss.heptapod.net.
Sep 17 2021, 2:36 PM · Archive coverage, System administration, Origin-GitLab
ardumont requested review of D6287: Allow gitlab lister's name to be overriden by task arguments.
Sep 17 2021, 2:32 PM
ardumont added a revision to T3581: List heptapod instance foss.heptapod.net: D6287: Allow gitlab lister's name to be overriden by task arguments.
Sep 17 2021, 2:29 PM · Archive coverage, System administration, Origin-GitLab
vsellier committed rDSNIP41c771d59b0c: grid5000/cassandra: add statsd metrics dashboards (authored by vsellier).
grid5000/cassandra: add statsd metrics dashboards
Sep 17 2021, 1:58 PM
vsellier committed rDSNIP7ec3d29c5c19: grid5000/cassandra: allow to test scheduler db persistance locally (authored by vsellier).
grid5000/cassandra: allow to test scheduler db persistance locally
Sep 17 2021, 1:58 PM
vsellier committed rDSNIPd4b4d44f1e61: grid5000/cassandra: allow to instantiate a best_effort node without an… (authored by vsellier).
grid5000/cassandra: allow to instantiate a best_effort node without an…
Sep 17 2021, 1:58 PM
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

Now most of the time can be spent in reading the actual mapping extids -> hgnode-id [1] to filter on something we already see.
Which does not change much from actual visits which already ended up in snapshot.
However that changes a lot for visits on forks where we can bypass already done work on those forks.

Sep 17 2021, 1:56 PM · System administration, Mercurial loader
ardumont closed D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 1:37 PM
ardumont committed rDLSfdb420238cc8: gitlab: Allow ingestion of hg_git origins as hg ones (authored by ardumont).
gitlab: Allow ingestion of hg_git origins as hg ones
Sep 17 2021, 1:37 PM
anlambert accepted D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 1:26 PM
ardumont added inline comments to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 1:20 PM
anlambert added inline comments to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 1:12 PM
stsp closed D6253: add CVS loader to the swh-loader.rst index.
Sep 17 2021, 12:57 PM
stsp committed rDDOC62f9bf297b80: add CVS loader to the swh-loader.rst index (authored by stsp).
add CVS loader to the swh-loader.rst index
Sep 17 2021, 12:57 PM
stsp updated the diff for D6253: add CVS loader to the swh-loader.rst index.

rebased patch

Sep 17 2021, 12:57 PM
stsp updated the diff for D6253: add CVS loader to the swh-loader.rst index.

insert new entry in alphabetical order

Sep 17 2021, 12:54 PM
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

Actually restarted the loader_oneshot which now makes usage of the latest v2.2.0 loader mercurial.

Sep 17 2021, 12:29 PM · System administration, Mercurial loader
vlorentz closed D6281: converters: Recompute hashes and check they match the originals.
Sep 17 2021, 12:27 PM
vlorentz committed rDLDG6d7a998b1093: converters: Recompute hashes and check they match the originals (authored by vlorentz).
converters: Recompute hashes and check they match the originals
Sep 17 2021, 12:27 PM
vlorentz closed D6280: converters: Add typing.
Sep 17 2021, 12:27 PM
vlorentz committed rDLDGf413e171b6e0: converters: Add typing (authored by vlorentz).
converters: Add typing
Sep 17 2021, 12:27 PM
swh-public-ci added a comment to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.

Build is green

Sep 17 2021, 12:20 PM
ardumont added a project to T3584: loader mercurial edge case about missing mapping from revision to hgnode-id: Mercurial loader.
Sep 17 2021, 12:18 PM · Mercurial loader
ardumont added inline comments to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 12:18 PM
ardumont updated the diff for D6286: gitlab: Allow ingestion of hg_git origins as hg ones.

Drop the hg to hg mapping, it's indeed not needed. I must have cross wired my mind and
fingers when testing that.

Sep 17 2021, 12:17 PM
swh-public-ci added a comment to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.

Build is green

Sep 17 2021, 12:16 PM
ardumont added inline comments to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 12:14 PM
ardumont updated the diff for D6286: gitlab: Allow ingestion of hg_git origins as hg ones.

Adapt according to suggestions to make code clearer! Thanks.

Sep 17 2021, 12:13 PM
ardumont added inline comments to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 12:11 PM
anlambert added inline comments to D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 12:07 PM
ardumont requested review of D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 11:57 AM
ardumont added a revision to T3581: List heptapod instance foss.heptapod.net: D6286: gitlab: Allow ingestion of hg_git origins as hg ones.
Sep 17 2021, 11:51 AM · Archive coverage, System administration, Origin-GitLab
vlorentz updated the task description for T3586: Figure out what to do with 'misordered' directories in Cassandra.
Sep 17 2021, 11:38 AM · Data Model, Storage manager
vlorentz removed a project from T3586: Figure out what to do with 'misordered' directories in Cassandra: meta-task.
Sep 17 2021, 11:37 AM · Data Model, Storage manager
vlorentz placed T3586: Figure out what to do with 'misordered' directories in Cassandra up for grabs.
Sep 17 2021, 11:37 AM · Data Model, Storage manager
vlorentz triaged T3586: Figure out what to do with 'misordered' directories in Cassandra as Normal priority.
Sep 17 2021, 11:37 AM · Data Model, Storage manager
vlorentz added a subtask for T3585: Fix inconsistencies of the Cassandra backend with postgres: T3582: cassandra: Use 'git ordering' for directory entries.
Sep 17 2021, 11:35 AM · meta-task, Storage manager
vlorentz added a parent task for T3582: cassandra: Use 'git ordering' for directory entries: T3585: Fix inconsistencies of the Cassandra backend with postgres.
Sep 17 2021, 11:35 AM · Storage manager
vlorentz triaged T3585: Fix inconsistencies of the Cassandra backend with postgres as Normal priority.
Sep 17 2021, 11:35 AM · meta-task, Storage manager
ardumont updated the task description for T3581: List heptapod instance foss.heptapod.net.
Sep 17 2021, 11:34 AM · Archive coverage, System administration, Origin-GitLab
ardumont added a comment to T3581: List heptapod instance foss.heptapod.net.

Thanks for the heads up, gonna simplify stuff then.

Sep 17 2021, 11:34 AM · Archive coverage, System administration, Origin-GitLab
ardumont added a comment to T3584: loader mercurial edge case about missing mapping from revision to hgnode-id.

This one seems small enough to use as data for analyzing and possibly fixing the test:

Sep 14 04:09:01 worker17 python3[354854]: [2021-09-14 04:09:01,867: INFO/ForkPoolWorker-292] Load origin 'https://bitbucket.org/avnomad/d' with type 'hg'
Sep 14 04:09:02 worker17 python3[354854]: [2021-09-14 04:09:02,826: ERROR/ForkPoolWorker-292] Loading failure, updating to `failed` status
                                          Traceback (most recent call last):
                                            File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 339, in load
                                              self.store_data()
                                            File "/usr/lib/python3/dist-packages/swh/loader/mercurial/from_disk.py", line 397, in store_data
                                              target = self.get_revision_id_from_hg_nodeid(node_id)
                                            File "/usr/lib/python3/dist-packages/swh/loader/mercurial/from_disk.py", line 471, in get_revision_id_from_hg_nodeid
                                              assert len(from_storage) == 1, msg % (hg_nodeid.hex(), len(from_storage))
                                          AssertionError: Expected 1 match from storage for hg node '38663535643238346139643464376432313166303463626336373865396632313562333034343034', got 0
Sep 14 04:09:03 worker17 python3[354854]: [2021-09-14 04:09:03,137: INFO/ForkPoolWorker-292] Task swh.loader.mercurial.tasks.LoadMercurial[fd077098-7779-4633-805b-f0cb0ad85b2a] succeeded in 1.9395398199558258s: {'status': 'failed'}
Sep 17 2021, 11:32 AM · Mercurial loader
marmoute added a comment to T3581: List heptapod instance foss.heptapod.net.

The hg-git type are served as regular Mercurial repository. So they can be listed as Mercurial repository safely

Sep 17 2021, 11:31 AM · Archive coverage, System administration, Origin-GitLab
ardumont renamed T3584: loader mercurial edge case about missing mapping from revision to hgnode-id from loader mercurial edge case about missing to loader mercurial edge case about missing mapping from revision to hgnode-id.
Sep 17 2021, 11:21 AM · Mercurial loader
ardumont triaged T3584: loader mercurial edge case about missing mapping from revision to hgnode-id as Normal priority.
Sep 17 2021, 11:21 AM · Mercurial loader
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

Another run in on a large repository (which cannot finish, the error is independent
though) [1]

Sep 17 2021, 11:13 AM · System administration, Mercurial loader
ardumont created P1169 Large repository rounds 1st is long, 2nd is failing faster (because the error is blocking).
Sep 17 2021, 11:13 AM
ardumont updated the task description for T3581: List heptapod instance foss.heptapod.net.
Sep 17 2021, 10:57 AM · Archive coverage, System administration, Origin-GitLab
ardumont closed D6285: gitlab: Allow listing of instances providing multiple vcs_type.
Sep 17 2021, 10:50 AM
ardumont committed rDLS4e4edee478ce: gitlab: Allow listing of instances providing multiple vcs_type (authored by ardumont).
gitlab: Allow listing of instances providing multiple vcs_type
Sep 17 2021, 10:50 AM
ardumont closed T3567: storage: Allow extid reading with filter on extid version, a subtask of T3563: Analyze and make the bitbucket ingestion faster, as Resolved.
Sep 17 2021, 10:48 AM · System administration, Mercurial loader
ardumont closed T3567: storage: Allow extid reading with filter on extid version as Resolved.

Packaged within the v0.37 version.
Deployed both in staging and production.

Sep 17 2021, 10:48 AM · System administration, Mercurial loader
ardumont closed T3571: mercurial loader: Fix snapshot creation as Resolved.
Sep 17 2021, 10:45 AM · Mercurial loader
ardumont added a comment to T3571: mercurial loader: Fix snapshot creation.

Deployed on staging and everything looked good.
So deployed on production as well.

Sep 17 2021, 10:45 AM · Mercurial loader
anlambert updated the task description for T2489: Git origin without smart transfer protocol support cannot be loaded.
Sep 17 2021, 10:44 AM · Git loader
anlambert accepted D6285: gitlab: Allow listing of instances providing multiple vcs_type.

Looks good to me !

Sep 17 2021, 10:38 AM
ardumont closed D6275: Adapt extid filtering so it happens server side.
Sep 17 2021, 10:05 AM
ardumont closed D6268: mercurial: Build snapshot on visits.
Sep 17 2021, 10:05 AM
ardumont committed rDLDHG8e3b880ebcb8: Adapt extid filtering so it happens server side (authored by ardumont).
Adapt extid filtering so it happens server side
Sep 17 2021, 10:05 AM
ardumont committed rDLDHGa0f8e9e856a4: Build snapshot on visits (authored by ardumont).
Build snapshot on visits
Sep 17 2021, 10:05 AM
ardumont closed D6240: Use extids to filter out already seen revisions across hg origins.
Sep 17 2021, 10:05 AM
ardumont committed rDLDHG765cb5aefa8b: Add support to filter out already seen revisions across hg origins (authored by ardumont).
Add support to filter out already seen revisions across hg origins
Sep 17 2021, 10:05 AM
ardumont added a parent task for T3583: check icinga alert for svn save-code-now: T3458: save code now: Requests are not getting updated from time to time.
Sep 17 2021, 9:34 AM · Scheduling utilities, Save Code Now, Monitoring
ardumont added a subtask for T3458: save code now: Requests are not getting updated from time to time: T3583: check icinga alert for svn save-code-now.
Sep 17 2021, 9:34 AM · Save Code Now
ardumont updated the task description for T3583: check icinga alert for svn save-code-now.
Sep 17 2021, 9:33 AM · Scheduling utilities, Save Code Now, Monitoring
ardumont triaged T3583: check icinga alert for svn save-code-now as Normal priority.
Sep 17 2021, 9:30 AM · Scheduling utilities, Save Code Now, Monitoring

Sep 16 2021

vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

Two other sources of mismatched checksums:

Sep 16 2021, 7:22 PM · Archive content, Restricted Project
ardumont added a comment to T3468: staging: current opam loading issues.

Heads up, it seems the main issues mentioned above have subsided.
It's still ongoing but the tendency seem to go the right way.

Sep 16 2021, 6:45 PM · System administration, Opam
ardumont requested review of D6285: gitlab: Allow listing of instances providing multiple vcs_type.
Sep 16 2021, 6:41 PM
ardumont added a revision to T3581: List heptapod instance foss.heptapod.net: D6285: gitlab: Allow listing of instances providing multiple vcs_type.
Sep 16 2021, 6:38 PM · Archive coverage, System administration, Origin-GitLab
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Sep 16 2021, 6:09 PM · System administration (Component upgrades)
ardumont renamed T3579: Meta-task: upgrade infrastructure to Debian Bullseye from Meta-task: upgrade infrastructure to Debian Buster to Meta-task: upgrade infrastructure to Debian Bullseye.
Sep 16 2021, 6:07 PM · System administration (Component upgrades)
First
Prev
Next
About · Developer information · Legal