Page MenuHomeSoftware Heritage
Feed All Stories

Nov 9 2021

ardumont requested review of D6630: Clean up loader, add types and update docstrings.
Nov 9 2021, 6:21 PM
ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6630: Clean up loader, add types and update docstrings.
Nov 9 2021, 6:20 PM · SVN Loader
olasd added a comment to D6424: Perfect hashmap C implementation.

I don't think we should be storing fields defined as size_t on disk, as I believe that's architecture-dependent: I see that you have defined and use ntohq / htonq to, if I understand correctly, make sure that the endianness of the data stored on disk is always in network byte order (although currently we're just always swapping the byte orders, AFAICT). If we end up using an architecture with a 32 (or 128) bit size_t, or even rarer, a big endian architecture, we'll have a bad time.

I think we should:

store explicit uint64_t values on disk, with a specified byte order.

Done.

use off_t instead of size_t on the C side, for types that store file offsets (which should be a signed 64 bits value when compiled with -D_FILE_OFFSET_BITS=64, even on 32 bits systems). Unfortunately that means wasting a bit on offset storage, but I think we can live with that.

I'm not sure what would be the upside so I kept it as u_int64_t. Do you have a specific scenario in mind where it would create a problem?

Nov 9 2021, 5:59 PM
swh-public-ci added a comment to D6629: Package loader: Uniformize author and message.

Build is green

Nov 9 2021, 5:46 PM
swh-public-ci added a comment to D6629: Package loader: Uniformize author and message.

Build is green

Nov 9 2021, 5:45 PM
vlorentz updated the summary of D6629: Package loader: Uniformize author and message.
Nov 9 2021, 5:44 PM
vlorentz updated the diff for D6629: Package loader: Uniformize author and message.

Update commit msg

Nov 9 2021, 5:44 PM
vlorentz updated the summary of D6629: Package loader: Uniformize author and message.
Nov 9 2021, 5:44 PM
vlorentz updated the summary of D6629: Package loader: Uniformize author and message.
Nov 9 2021, 5:44 PM
vlorentz updated the test plan for D6629: Package loader: Uniformize author and message.
Nov 9 2021, 5:43 PM
vlorentz updated the diff for D6629: Package loader: Uniformize author and message.

Use empty string as author instead of None, it's actually unclear how to make the model handle None authors with non-None dates

Nov 9 2021, 5:43 PM
vlorentz abandoned D6628: Allow release authors to be None even if the date is not None.

abandonning this, as there isn't a good way to serialize manifests with a date but no author

Nov 9 2021, 5:33 PM
vlorentz abandoned D6627: Allow release authors to be None even if the date is not None.

abandonning this, as there isn't a good way to serialize manifests with a date but no author

Nov 9 2021, 5:33 PM
douardda closed D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Nov 9 2021, 5:28 PM
douardda committed rDSTO850a7553b6d5: Add support for a redis-based reporting for invalid mirrorred objects (authored by douardda).
Add support for a redis-based reporting for invalid mirrorred objects
Nov 9 2021, 5:28 PM
douardda committed rDSTO04bd15a0bca8: Refactor fixer.fix_objects() to extract the inner object_fixers dict (authored by douardda).
Refactor fixer.fix_objects() to extract the inner object_fixers dict
Nov 9 2021, 5:28 PM
douardda closed D6570: Remove now useless revision date checker in fixer.
Nov 9 2021, 5:28 PM
douardda committed rDSTOd655c8581d3c: Remove now useless fixers (authored by douardda).
Remove now useless fixers
Nov 9 2021, 5:28 PM
douardda closed D6569: Add a --type option to 'swh storage replay'.
Nov 9 2021, 5:28 PM
douardda committed rDSTO55eed77b519c: Add a --type option to 'swh storage replay' (authored by douardda).
Add a --type option to 'swh storage replay'
Nov 9 2021, 5:28 PM
olasd added a comment to T3286: Use journal clients for webapp and deposit to subscribe to events.

Moving towards event notifications and stream processing instead of polling sounds worthwhile, before the amount of polling becomes more important than managing the event notification mechanism. For the two systems you've mentioned, I think we're really, really far away from that, but it's still worth considering a way to do event-driven notifications properly, so we don't have to rush it through.

Nov 9 2021, 5:26 PM · Save Code Now, SWORD deposit, Web app
swh-public-ci added a comment to D6570: Remove now useless revision date checker in fixer.

Build is green

Nov 9 2021, 5:26 PM
swh-public-ci added a comment to D6570: Remove now useless revision date checker in fixer.

Build was aborted

Nov 9 2021, 5:18 PM
swh-public-ci added a comment to D6571: Add support for a redis-based reporting for invalid mirrorred objects.

Build is green

Nov 9 2021, 4:47 PM
Harbormaster failed remote builds in B24961: Diff 24086 for D6570: Remove now useless revision date checker in fixer!
Nov 9 2021, 4:46 PM
swh-public-ci added a comment to D6570: Remove now useless revision date checker in fixer.

Build has FAILED

Nov 9 2021, 4:46 PM
swh-public-ci added a comment to D6569: Add a --type option to 'swh storage replay'.

Build is green

Nov 9 2021, 4:45 PM
ardumont accepted D6625: Refactor package loaders to make the version part of BasePackageInfo.
Nov 9 2021, 4:45 PM
douardda accepted D6625: Refactor package loaders to make the version part of BasePackageInfo.

makes sense

Nov 9 2021, 4:44 PM
olasd added a comment to D6627: Allow release authors to be None even if the date is not None.

Can you add a test that the manifest renders properly (with the date) in that case?

Nov 9 2021, 4:39 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

rebase

Nov 9 2021, 4:38 PM
douardda updated the diff for D6570: Remove now useless revision date checker in fixer.

rebase

Nov 9 2021, 4:38 PM
douardda updated the diff for D6569: Add a --type option to 'swh storage replay'.

rebase

Nov 9 2021, 4:37 PM
ardumont accepted D6624: Refactor package loaders to remove temporary revision objects.
Nov 9 2021, 4:37 PM
ardumont added a comment to D6624: Refactor package loaders to remove temporary revision objects.

a couple of remarks inline.

Nov 9 2021, 4:36 PM
vsellier committed rSENVa02e20781f5a: Declare kibana vm (authored by vsellier).
Declare kibana vm
Nov 9 2021, 4:34 PM
vlorentz updated the summary of D6629: Package loader: Uniformize author and message.
Nov 9 2021, 4:34 PM
Harbormaster failed remote builds in B24959: Diff 24084 for D6629: Package loader: Uniformize author and message!
Nov 9 2021, 4:34 PM
swh-public-ci added a comment to D6629: Package loader: Uniformize author and message.

Build has FAILED

Nov 9 2021, 4:34 PM
vlorentz published D6629: Package loader: Uniformize author and message for review.
Nov 9 2021, 4:32 PM
ardumont accepted D6618: Document how each package loader populates fields..
Nov 9 2021, 4:29 PM
ardumont updated the summary of D6618: Document how each package loader populates fields..
Nov 9 2021, 4:29 PM
ardumont accepted D6626: cran: Pass the package name to the loader.
Nov 9 2021, 4:24 PM
swh-public-ci added a comment to D6623: add CVS commit ID support to rlog.py.

Build is green

Nov 9 2021, 4:05 PM
stsp updated the diff for D6623: add CVS commit ID support to rlog.py.

changes suggested by vlorentz

Nov 9 2021, 4:03 PM
vlorentz requested review of D6628: Allow release authors to be None even if the date is not None.
Nov 9 2021, 4:00 PM
swh-public-ci added a comment to D6527: Update extrinsic-metadata-specification.rst to match the current implementation.

Build is green

Nov 9 2021, 3:55 PM
stsp closed D6593: handle Attic-only RCS files over CVS pserver.
Nov 9 2021, 3:54 PM
stsp committed rDLDCVSd28a4b21c56a: handle Attic-only RCS files over CVS pserver (authored by stsp).
handle Attic-only RCS files over CVS pserver
Nov 9 2021, 3:54 PM
swh-public-ci added a comment to D6593: handle Attic-only RCS files over CVS pserver.

Build is green

Nov 9 2021, 3:53 PM
stsp updated the diff for D6593: handle Attic-only RCS files over CVS pserver.

rebased patch

Nov 9 2021, 3:51 PM
stsp closed D6592: improve test coverage of file additions and deletions.
Nov 9 2021, 3:50 PM
stsp committed rDLDCVSd72f15f24dbd: improve test coverage of file additions and deletions (authored by stsp).
improve test coverage of file additions and deletions
Nov 9 2021, 3:50 PM
swh-public-ci added a comment to D6592: improve test coverage of file additions and deletions.

Build is green

Nov 9 2021, 3:50 PM
stsp updated the diff for D6592: improve test coverage of file additions and deletions.

rebased patch

Nov 9 2021, 3:48 PM
vlorentz requested review of D6627: Allow release authors to be None even if the date is not None.
Nov 9 2021, 3:48 PM
vlorentz closed D6527: Update extrinsic-metadata-specification.rst to match the current implementation.
Nov 9 2021, 3:48 PM
vlorentz committed rDSTO0262f1c191cc: Update extrinsic-metadata-specification.rst to match the current implementation (authored by vlorentz).
Update extrinsic-metadata-specification.rst to match the current implementation
Nov 9 2021, 3:48 PM
stsp closed D6591: display file state in progress logging output.
Nov 9 2021, 3:47 PM
stsp committed rDLDCVSca23bc13c21b: display file state in progress logging output (authored by stsp).
display file state in progress logging output
Nov 9 2021, 3:47 PM
vlorentz updated the diff for D6527: Update extrinsic-metadata-specification.rst to match the current implementation.

rebase

Nov 9 2021, 3:47 PM
vlorentz closed D6610: Give model and swhid objects a nicer repr().
Nov 9 2021, 3:46 PM
vlorentz committed rDMOD2ffe5dba8401: Give model and swhid objects a nicer repr() (authored by vlorentz).
Give model and swhid objects a nicer repr()
Nov 9 2021, 3:45 PM
swh-public-ci added a comment to D6591: display file state in progress logging output.

Build is green

Nov 9 2021, 3:45 PM
stsp updated the diff for D6591: display file state in progress logging output.

rebased patch

Nov 9 2021, 3:43 PM
stsp closed D6590: add support for RCS keyword expansion over pserver protocol.
Nov 9 2021, 3:43 PM
stsp committed rDLDCVSf52f0e452132: add support for RCS keyword expansion over pserver protocol (authored by stsp).
add support for RCS keyword expansion over pserver protocol
Nov 9 2021, 3:43 PM
swh-public-ci added a comment to D6590: add support for RCS keyword expansion over pserver protocol.

Build is green

Nov 9 2021, 3:20 PM
stsp updated the diff for D6590: add support for RCS keyword expansion over pserver protocol.

rebased patch

Nov 9 2021, 3:19 PM
vlorentz requested review of D6626: cran: Pass the package name to the loader.
Nov 9 2021, 3:07 PM
vlorentz accepted D6593: handle Attic-only RCS files over CVS pserver.

Thanks!

Nov 9 2021, 2:27 PM
vlorentz requested changes to D6623: add CVS commit ID support to rlog.py.

Looks good

Nov 9 2021, 2:27 PM
vlorentz requested review of D6625: Refactor package loaders to make the version part of BasePackageInfo.
Nov 9 2021, 2:21 PM
ardumont added a comment to T3721: Configure access to replicabilitystamp.org user.

What's missing, provider_url and domain:

Nov 9 2021, 2:00 PM · SWORD deposit
ardumont added a comment to T3721: Configure access to replicabilitystamp.org user.

db is almost ready:

Nov 9 2021, 1:59 PM · SWORD deposit
ardumont triaged T3721: Configure access to replicabilitystamp.org user as Normal priority.
Nov 9 2021, 1:58 PM · SWORD deposit
vlorentz requested review of D6624: Refactor package loaders to remove temporary revision objects.
Nov 9 2021, 1:52 PM
vlorentz added inline comments to D6618: Document how each package loader populates fields..
Nov 9 2021, 1:49 PM
vlorentz requested review of D6618: Document how each package loader populates fields..
Nov 9 2021, 1:46 PM
stsp added a comment to T3691: Implement CVS loader.

As of D6623 the CVS loader is able to convert GNU dino correctly over both rsync and pserver access.

Nov 9 2021, 1:30 PM · CVS loader, Archive coverage
stsp requested review of D6623: add CVS commit ID support to rlog.py.
Nov 9 2021, 1:28 PM
stsp added a revision to T3691: Implement CVS loader: D6623: add CVS commit ID support to rlog.py.
Nov 9 2021, 1:27 PM · CVS loader, Archive coverage
ardumont updated the task description for T3705: Upgrade the ELK stack.
Nov 9 2021, 1:25 PM · System administration (Component upgrades)
ardumont updated the task description for T3705: Upgrade the ELK stack.
Nov 9 2021, 1:24 PM · System administration (Component upgrades)
ardumont updated the task description for T3705: Upgrade the ELK stack.
Nov 9 2021, 1:24 PM · System administration (Component upgrades)
vlorentz added a revision to T3638: Make package loaders create releases objects instead of revisions: D6618: Document how each package loader populates fields..
Nov 9 2021, 12:39 PM · Package Loader, Data Model, Archive content
ardumont closed D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 12:24 PM
ardumont committed rDLDSVNe36caff0a4f6: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted (authored by ardumont).
SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted
Nov 9 2021, 12:24 PM
swh-public-ci added a comment to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Build is green

Nov 9 2021, 12:23 PM
ardumont added inline comments to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 12:22 PM
ardumont updated the diff for D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Wrap the remove instruction in case failure to remove happens.

Nov 9 2021, 12:21 PM
ardumont added inline comments to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 12:21 PM
ardumont added inline comments to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 12:09 PM
vlorentz accepted D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 12:07 PM
ardumont accepted D6527: Update extrinsic-metadata-specification.rst to match the current implementation.
Nov 9 2021, 12:06 PM
swh-public-ci added a comment to D6593: handle Attic-only RCS files over CVS pserver.

Build is green

Nov 9 2021, 12:04 PM
stsp added inline comments to D6593: handle Attic-only RCS files over CVS pserver.
Nov 9 2021, 12:04 PM
stsp updated the diff for D6593: handle Attic-only RCS files over CVS pserver.

assert self.cvsroot_path instead of casting and squash all changes

Nov 9 2021, 12:03 PM
swh-public-ci added a comment to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Build is green

Nov 9 2021, 11:58 AM
ardumont updated the diff for D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Update

Nov 9 2021, 11:56 AM