Page MenuHomeSoftware Heritage
Create Task
Maniphest T4668

replayer: Crashes because of directories with duplicated entry names in journal
Closed, MigratedEdits Locked
Actions

Description

https://sentry.softwareheritage.org/organizations/swh/issues/104660/?referrer=phabricator_plugin

ValueError: swh:1:dir:96f959bba77e02a9161462a07a4e0f10c7fbfa3a has duplicated entry name: b'1'
(13 additional frame(s) were not displayed)
...
  File "swh/storage/replay.py", line 118, in convert
    obj = OBJECT_CONVERTERS[object_type](dict_repr)
  File "swh/model/model.py", line 1194, in from_dict
    return cls(
  File "<attrs generated init swh.model.model.Directory>", line 6, in __init__
    """
  File "attr/_make.py", line 2946, in __call__
    v(inst, attr, value)
  File "swh/model/model.py", line 1185, in check_entries
    raise ValueError(

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT2213 Storage
 Migratedgitlab-migrationT2214 Scale-out graph and database storage in production
 Migratedgitlab-migrationT1892 Cassandra as a storage backend
 Migratedgitlab-migrationT1816 Make all clients of swh-storage use origin URL as identifier and use visit types instead of origin types
 Migratedgitlab-migrationT1731 Intrinsic identifiers for origins
 Migratedgitlab-migrationT1912 Support origin pagination without origin ids
 Migratedgitlab-migrationT2033 Run Cassandra storage backend with production data
 Migratedgitlab-migrationT4668 replayer: Crashes because of directories with duplicated entry names in journal
 Migratedgitlab-migrationT2185 Make webapp0 use Cassandra as storage backend.
 Migratedgitlab-migrationT2186 Merge swh-storage-cassandra in swh-storage master
 Migratedgitlab-migrationT2602 Investigate how to upgrade the schema of the Cassandra storage
 Migratedgitlab-migrationT3314 Test swh.storage.cassandra with ScyllaDB
 Migratedgitlab-migrationT3357 Perform some tests of the cassandra storage on Grid5000
 Migratedgitlab-migrationT3394 cassandra - origin url hashing encoding issue
 Migratedgitlab-migrationT3395 cassandra - Timeouts during revision import
 Migratedgitlab-migrationT3396 cassandra - allow to configure the consistency level used by the queries
 Migratedgitlab-migrationT3464 Prepare a quote for the cassandra servers
 Migratedgitlab-migrationT3465 Test multidatacenter replication
 Migratedgitlab-migrationT3491 Origin visit ids restart from 1 even if there is previous visits
 Migratedgitlab-migrationT3493 [cassandra] Git loader performance are very bad
 Migratedgitlab-migrationT3517 [cassandra] decorate the method calls to have statsd metrics
Migratedgitlab-migrationT3573 [cassandra] directory and content read benchmarks
 Migratedgitlab-migrationT3577 Parallel loaders performances
Migratedgitlab-migrationT3683 cassandra - benchmark the vault
 Migratedgitlab-migrationT3585 Fix inconsistencies of the Cassandra backend with postgres
 Migratedgitlab-migrationT3582 cassandra: Use 'git ordering' for directory entries
 Migratedgitlab-migrationT3586 Figure out what to do with 'misordered' directories in Cassandra

Event Timeline

swh-sentry-integration assigned this task to vlorentz.Nov 2 2022, 11:17 AM
swh-sentry-integration added projects: Journal, Data Model.
swh-sentry-integration added a subscriber: swh-sentry-integration.
vlorentz removed vlorentz as the assignee of this task.Nov 2 2022, 11:17 AM
vlorentz triaged this task as Normal priority.
vlorentz added a subscriber: vlorentz.
vlorentz added a comment.Nov 2 2022, 11:20 AM

New objects with duplicated entries go through this method to be fixed: https://docs.softwareheritage.org/devel/apidoc/swh.model.model.html#swh.model.model.Directory.from_possibly_duplicated_entries (currently only used by swh.storage.backfill); but old kafka messages still have duplicate entries, causing the crash above.

vlorentz added a parent task: T2033: Run Cassandra storage backend with production data.Nov 2 2022, 11:32 AM
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:25 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal