Page MenuHomeSoftware Heritage
Paste P520

[fixed] broken ci: index failure status
ActivePublic
Actions

Authored by ardumont on Sep 5 2019, 2:55 PM.
swh/indexer/tests/test_metadata.py .................................F
================================================================================================================== FAILURES ===================================================================================================================
___________________________________________________________________________________________________ Metadata.test_revision_metadata_indexer ___________________________________________________________________________________________________
self = <swh.indexer.tests.test_metadata.Metadata testMethod=test_revision_metadata_indexer>
def test_revision_metadata_indexer(self):
metadata_indexer = RevisionMetadataIndexer(
config=REVISION_METADATA_CONFIG)
fill_obj_storage(metadata_indexer.objstorage)
fill_storage(metadata_indexer.storage)
tool = metadata_indexer.idx_storage.indexer_configuration_get(
{'tool_'+k: v for (k, v) in TRANSLATOR_TOOL.items()})
assert tool is not None
metadata_indexer.idx_storage.content_metadata_add([{
'indexer_configuration_id': tool['id'],
'id': b'cde',
'metadata': YARN_PARSER_METADATA,
}])
sha1_gits = [
hash_to_bytes('8dbb6aeb036e7fd80664eb8bfd1507881af1ba9f'),
]
print('##### sha1gits', sha1_gits)
metadata_indexer.run(sha1_gits, 'update-dups')
results = list(
metadata_indexer.idx_storage.
revision_intrinsic_metadata_get(sha1_gits))
print('#####results: ', results)
expected_results = [{
'id': hash_to_bytes('8dbb6aeb036e7fd80664eb8bfd1507881af1ba9f'),
'tool': TRANSLATOR_TOOL,
'metadata': YARN_PARSER_METADATA,
'mappings': ['npm'],
}]
for result in results:
del result['tool']['id']
# then
> self.assertEqual(expected_results, results)
E AssertionError: Lists differ: [{'id[202 chars]}, 'metadata': {'@context': 'https://doi.org/1[549 chars]m']}] != [{'id[202 chars]}, 'mappings': ['npm'], 'metadata': {'@context[46 chars]0'}}]
E
E First differing element 0:
E {'id'[201 chars]}, 'metadata': {'@context': 'https://doi.org/1[548 chars]pm']}
E {'id'[201 chars]}, 'mappings': ['npm'], 'metadata': {'@context[45 chars].0'}}
E
E [{'id': b'\x8d\xbbj\xeb\x03n\x7f\xd8\x06d\xeb\x8b\xfd\x15\x07\x88'
E b'\x1a\xf1\xba\x9f',
E 'mappings': ['npm'],
E - 'metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0',
E + 'metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0'},
E ? +
E
E - 'author': [{'name': 'Andrew Nesbitt', 'type': 'Person'}],
E - 'codeRepository': 'git+git+https://github.com/librariesio/yarn-parser.git',
E - 'description': 'Tiny web service for parsing yarn.lock files',
E - 'issueTracker': 'https://github.com/librariesio/yarn-parser/issues',
E - 'keywords': ['yarn', 'parse', 'lock', 'dependencies'],
E - 'license': 'https://spdx.org/licenses/AGPL-3.0',
E - 'name': 'yarn-parser',
E - 'type': 'SoftwareSourceCode',
E - 'url': 'https://github.com/librariesio/yarn-parser#readme',
E - 'version': '1.0.0'},
E 'tool': {'configuration': {'context': 'NpmMapping', 'type': 'local'},
E 'name': 'swh-metadata-translator',
E 'version': '0.0.2'}}]
swh/indexer/tests/test_metadata.py:1156: AssertionError
------------------------------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------------------------------
##### sha1gits [b'\x8d\xbbj\xeb\x03n\x7f\xd8\x06d\xeb\x8b\xfd\x15\x07\x88\x1a\xf1\xba\x9f']
#### rev: {'id': b'\x8d\xbbj\xeb\x03n\x7f\xd8\x06d\xeb\x8b\xfd\x15\x07\x88\x1a\xf1\xba\x9f', 'message': 'Improve search functionality', 'author': {'name': b'Andrew Nesbitt', 'email': b'andrewnez@gmail.com', 'fullname': b'Andrew Nesbitt <andrewnez@gmail.com>'}, 'committer': {'name': b'Andrew Nesbitt', 'email': b'andrewnez@gmail.com', 'fullname': b'Andrew Nesbitt <andrewnez@gmail.com>'}, 'date': {'timestamp': {'seconds': 1487596456, 'microseconds': 0}, 'offset': 0, 'negative_utc': False}, 'committer_date': {'timestamp': {'seconds': 1380883849, 'microseconds': 0}, 'offset': 120, 'negative_utc': None}, 'type': 'git', 'directory': b'10', 'synthetic': False, 'metadata': None, 'parents': []}
##### rootdir b'10'
#### dir-ls: [{'status': 'visible', 'sha1': b'abc', 'sha1_git': b'abc', 'sha256': b'abc', 'length': 1029, 'name': b'index.js', 'type': 'file', 'target': b'abc', 'perms': 33188, 'dir_id': b'10'}, {'status': 'visible', 'sha1': b'aab', 'sha1_git': b'aab', 'sha256': b'aab', 'length': 1029, 'name': b'package.json', 'type': 'file', 'target': b'aab', 'perms': 33188, 'dir_id': b'10'}, {'status': None, 'sha1': None, 'sha1_git': None, 'sha256': None, 'length': None, 'name': b'.github', 'type': 'dir', 'target': b'11', 'perms': 16384, 'dir_id': b'10'}]
##### dir-ls: [{'status': 'visible', 'sha1': b'abc', 'sha1_git': b'abc', 'sha256': b'abc', 'length': 1029, 'name': b'index.js', 'type': 'file', 'target': b'abc', 'perms': 33188, 'dir_id': b'10'}, {'status': 'visible', 'sha1': b'aab', 'sha1_git': b'aab', 'sha256': b'aab', 'length': 1029, 'name': b'package.json', 'type': 'file', 'target': b'aab', 'perms': 33188, 'dir_id': b'10'}, {'status': None, 'sha1': None, 'sha1_git': None, 'sha256': None, 'length': None, 'name': b'.github', 'type': 'dir', 'target': b'11', 'perms': 16384, 'dir_id': b'10'}]
##### files: [{'status': 'visible', 'sha1': b'abc', 'sha1_git': b'abc', 'sha256': b'abc', 'length': 1029, 'name': b'index.js', 'type': 'file', 'target': b'abc', 'perms': 33188, 'dir_id': b'10'}, {'status': 'visible', 'sha1': b'aab', 'sha1_git': b'aab', 'sha256': b'aab', 'length': 1029, 'name': b'package.json', 'type': 'file', 'target': b'aab', 'perms': 33188, 'dir_id': b'10'}]
##### detected-files {'NpmMapping': [b'aab']}
################ {'indexer_storage': {'cls': 'memory', 'args': {}}, 'objstorage': {'cls': 'memory', 'args': {}}, 'storage': {'cls': 'memory', 'args': {}}, 'tools': [{'name': 'swh-metadata-translator', 'version': '0.0.2', 'configuration': {'context': 'NpmMapping'}}]}
############### c_metadata_indexer.objstorage.state {}
#### sha1: b'aab'
#### mappings ['npm']
#### metadata {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0'}
#### index res {'id': b'\x8d\xbbj\xeb\x03n\x7f\xd8\x06d\xeb\x8b\xfd\x15\x07\x88\x1a\xf1\xba\x9f', 'indexer_configuration_id': -7210821413007573893, 'mappings': ['npm'], 'metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0'}}
#####results: [{'id': b'\x8d\xbbj\xeb\x03n\x7f\xd8\x06d\xeb\x8b\xfd\x15\x07\x88\x1a\xf1\xba\x9f', 'tool': {'id': -7210821413007573893, 'name': 'swh-metadata-translator', 'version': '0.0.2', 'configuration': {'type': 'local', 'context': 'NpmMapping'}}, 'mappings': ['npm'], 'metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0'}}]
-------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------
indexer.py 371 WARNING Content 616162 not found in objstorage

Event Timeline

ardumont created this paste.Sep 5 2019, 2:55 PM
ardumont added a comment.Sep 5 2019, 3:01 PM

current status:

Initially, there was missing target directory entries in the objstorage, that's now fixed.

Now the test is still failing because of a shortage of metadata in the result computed.

In the test, the warning is explanable.

indexer.py                 371 WARNING  Content 616162 not found in objstorage

Note: 616162 is really the content id b'aab' ("sha1"). It's an implementation detail of the log instruction which hexifies the id.

In any case, this can be explained because when instantiated, the content-metadata-indexer is bootstraped with an empty objstorage instead of the tested one (objstorage is populated at the tests' initialization phase).

As seen through the following log:

################ {'indexer_storage': {'cls': 'memory', 'args': {}}, 'objstorage': {'cls': 'memory', 'args': {}}, 'storage': {'cls': 'memory', 'args': {}}, 'tools': [{'name': 'swh-metadata-translator', 'version': '0.0.2', 'configuration': {'context': 'NpmMapping'}}]}
############### c_metadata_indexer.objstorage.state {}  # <- instead of the content setup-ed in the utils namespace...

Now, i just don't know whether that's the issue with the fact the test fails, failling because of the missing metadata in the result.

ardumont changed the title of this paste from failure in indexer to wip: broken ci: index failure status.Sep 5 2019, 3:01 PM
ardumont mentioned this in D1958: indexer: ci: broken master build: Fix various test/utils data model issues (following the new model validation delivery).
ardumont changed the title of this paste from wip: broken ci: index failure status to [fixed] broken ci: index failure status.Sep 6 2019, 9:42 AM

In the end, it was missing initialization data steps.

About · Developer information · Legal