Page MenuHomeSoftware Heritage
Paste P496

NPM and GNU Revision
ActivePublic
Actions

Authored by nahimilega on Aug 9 2019, 3:56 PM.
NPM Revision
{'synthetic': True,
'metadata': {'package_source': {'name': 'ja','version': '0.0.1', 'filename': 'ja-0.0.1.tgz', 'sha1': '31399c51d3024f6eb91c626a31a175dc30f343e5', 'date': '2014-04-07T16:02:07.453Z', 'url': 'https://registry.npmjs.org/ja/-/ja-0.0.1.tgz', 'sha256': '8101e284d5846e77f9698628da660e92f8627535c266a3aef28549b25e63a597', 'blake2s256': 'fcf5438822e12c8a6973b9ecb40df4978947edd675c4269d728d5279dc82b203'},
'package': {'author': 'Goldeneye Solutions', 'name': 'ja', 'description': 'Compose a cross-platform application from various things.',
'version': '0.0.1',
'engines': {'node': '>=0.9.9'}, 'dependencies': {'ja-lib': '0.1.x'}, 'copyright': '© 2014 Goldeneye Solutions Inc.'}},
'author': {'fullname': b'Goldeneye Solutions', 'name': b'Goldeneye Solutions', 'email': None},
'date': {'timestamp': {'seconds': 1396886527, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
'committer': {'fullname': b'Goldeneye Solutions', 'name': b'Goldeneye Solutions', 'email': None}, 'committer_date': {'timestamp': {'seconds': 1396886527, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
'message': b'0.0.1',
'directory': b'\xec\x08\xa3\x05\x8fM9?\x81\x0eE!+\x9c\xfa\n8\xe8\xc3y',
'parents': [],
'type': 'tar',
'id': b'm\xd5@6\xaa*v\xa6A\x8c\xfb\xc5Zk\x9eW\xa8\xb2lU'}
GNU Loader Revision
{'metadata': {'package': {'date': '944729610', 'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz'}},
'date': {'timestamp': {'seconds': 944729610, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
'committer_date': {'timestamp': {'seconds': 944729610, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
'author': {'name': b'Software Heritage', 'fullname': b'Software Heritage', 'email': b'robot@softwareheritage.org'},
'committer': {'name': b'Software Heritage', 'fullname': b'Software Heritage', 'email': b'robot@softwareheritage.org'},
'type': 'tar',
'message': b'swh-loader-package: synthetic revision message',
'directory': b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(',
'synthetic': True,
'parents': [],
'id': b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3'}
Docker compose override
version: '2'
services:
swh-objstorage:
volumes:
- "/home/archit/swh-environment/swh-objstorage:/src/swh-objstorage"
swh-loader:
volumes:
- "/home/archit/swh-environment/swh-loader-core:/src/swh-loader"
loader.yml
storage:
cls: remote
args:
url: http://swh-storage:5002/
celery:
task_broker: amqp://guest:guest@amqp//
task_modules:
- swh.loader.debian.tasks
- swh.loader.dir.tasks
- swh.loader.git.tasks
- swh.loader.mercurial.tasks
- swh.loader.npm.tasks
- swh.loader.pypi.tasks
- swh.loader.svn.tasks
- swh.loader.tar.tasks
- swh.loader.package.tasks
- swh.deposit.loader.tasks
task_queues:
- swh.loader.debian.tasks.LoadDebianPackage
- swh.loader.dir.tasks.LoadDirRepository
- swh.loader.git.tasks.LoadDiskGitRepository
- swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
- swh.loader.git.tasks.UpdateGitRepository
- swh.loader.mercurial.tasks.LoadArchiveMercurial
- swh.loader.mercurial.tasks.LoadMercurial
- swh.loader.npm.tasks.LoadNpm
- swh.loader.pypi.tasks.LoadPyPI
- swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
- swh.loader.svn.tasks.LoadSvnRepository
- swh.loader.svn.tasks.MountAndLoadSvnRepository
- swh.loader.tar.tasks.LoadTarRepository
- swh.deposit.loader.tasks.LoadDepositArchiveTsk
- swh.deposit.loader.tasks.ChecksDepositTsk
- swh.loader.package.tasks.LoadGNU
lister_db_url: postgresql://postgres@swh-listers-db/swh-listers
url: 'http://swh-deposit:5006'

Event Timeline

nahimilega created this paste.Aug 9 2019, 3:56 PM
nahimilega created this object in space S1 Public.
ardumont added a subscriber: ardumont.Aug 9 2019, 4:21 PM
>>> from pprint import pprint
>>> pprint(d)
{'author': {'email': None,
            'fullname': b'Goldeneye Solutions',
            'name': b'Goldeneye Solutions'},
 'committer': {'email': None,
               'fullname': b'Goldeneye Solutions',
               'name': b'Goldeneye Solutions'},
 'committer_date': {'negative_utc': False,
                    'offset': 0,
                    'timestamp': {'microseconds': 0, 'seconds': 1396886527}},
 'date': {'negative_utc': False,
          'offset': 0,
          'timestamp': {'microseconds': 0, 'seconds': 1396886527}},
 'directory': b'\xec\x08\xa3\x05\x8fM9?\x81\x0eE!+\x9c\xfa\n8\xe8\xc3y',
 'id': b'm\xd5@6\xaa*v\xa6A\x8c\xfb\xc5Zk\x9eW\xa8\xb2lU',
 'message': b'0.0.1',
 'metadata': {'package': {'author': 'Goldeneye Solutions',
                          'copyright': '© 2014 Goldeneye Solutions Inc.',
                          'dependencies': {'ja-lib': '0.1.x'},
                          'description': 'Compose a cross-platform application '
                                         'from various things.',
                          'engines': {'node': '>=0.9.9'},
                          'name': 'ja',
                          'version': '0.0.1'},
              'package_source': {'blake2s256': 'fcf5438822e12c8a6973b9ecb40df4978947edd675c4269d728d5279dc82b203',
                                 'date': '2014-04-07T16:02:07.453Z',
                                 'filename': 'ja-0.0.1.tgz',
                                 'name': 'ja',
                                 'sha1': '31399c51d3024f6eb91c626a31a175dc30f343e5',
                                 'sha256': '8101e284d5846e77f9698628da660e92f8627535c266a3aef28549b25e63a597',
                                 'url': 'https://registry.npmjs.org/ja/-/ja-0.0.1.tgz',
                                 'version': '0.0.1'}},
 'parents': [],
 'synthetic': True,
 'type': 'tar'}
>>> pprint(e)
{'author': {'email': b'robot@softwareheritage.org',
            'fullname': b'Software Heritage',
            'name': b'Software Heritage'},
 'committer': {'email': b'robot@softwareheritage.org',
               'fullname': b'Software Heritage',
               'name': b'Software Heritage'},
 'committer_date': {'negative_utc': False,
                    'offset': 0,
                    'timestamp': {'microseconds': 0, 'seconds': 944729610}},
 'date': {'negative_utc': False,
          'offset': 0,
          'timestamp': {'microseconds': 0, 'seconds': 944729610}},
 'directory': b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(',
 'id': b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3',
 'message': b'swh-loader-package: synthetic revision message',
 'metadata': {'package': {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz',
                          'date': '944729610'}},
 'parents': [],
 'synthetic': True,
 'type': 'tar'}
ardumont added a comment.EditedAug 9 2019, 4:29 PM
$ workon swh
$ cd swh-model
$ pip install -e .
$ python
>>> npm_rev = {'synthetic': True,
... 'metadata': {'package_source': {'name': 'ja','version': '0.0.1', 'filename': 'ja-0.0.1.tgz', 'sha1': '31399c51d3024f6eb91c626a31a175dc30f343e5', 'date': '2014-04-07T16:02:07.453Z', 'url': 'https://registry.npmjs.org/ja/-/ja-0.0.1.tgz', 'sha256': '8101e284d5846e77f9698628da660e92f8627535c266a3aef28549b25e63a597', 'blake2s256': 'fcf5438822e12c8a6973b9ecb40df4978947edd675c4269d728d5279dc82b203'},
... 'package': {'author': 'Goldeneye Solutions', 'name': 'ja', 'description': 'Compose a cross-platform application from various things.',
... 'version': '0.0.1',
... 'engines': {'node': '>=0.9.9'}, 'dependencies': {'ja-lib': '0.1.x'}, 'copyright': '© 2014 Goldeneye Solutions Inc.'}},
... 'author': {'fullname': b'Goldeneye Solutions', 'name': b'Goldeneye Solutions', 'email': None},
... 'date': {'timestamp': {'seconds': 1396886527, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
... 'committer': {'fullname': b'Goldeneye Solutions', 'name': b'Goldeneye Solutions', 'email': None}, 'committer_date': {'timestamp': {'seconds': 1396886527, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
... 'message': b'0.0.1',
... 'directory': b'\xec\x08\xa3\x05\x8fM9?\x81\x0eE!+\x9c\xfa\n8\xe8\xc3y',
... 'parents': [],
... 'type': 'tar',
... 'id': b'm\xd5@6\xaa*v\xa6A\x8c\xfb\xc5Zk\x9eW\xa8\xb2lU'}
>>> gnu_rev = {'metadata': {'package': {'date': '944729610', 'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz'}},
... 'date': {'timestamp': {'seconds': 944729610, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
... 'committer_date': {'timestamp': {'seconds': 944729610, 'microseconds': 0}, 'offset': 0, 'negative_utc': False},
... 'author': {'name': b'Software Heritage', 'fullname': b'Software Heritage', 'email': b'robot@softwareheritage.org'},
... 'committer': {'name': b'Software Heritage', 'fullname': b'Software Heritage', 'email': b'robot@softwareheritage.org'},
... 'type': 'tar',
... 'message': b'swh-loader-package: synthetic revision message',
... 'directory': b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(',
... 'synthetic': True,
... 'parents': [],
... 'id': b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3'}
>>> from swh.model.model import Revision
>>> npm = Revision.from_dict(npm_rev)
>>> npm
Revision(id=b'm\xd5@6\xaa*v\xa6A\x8c\xfb\xc5Zk\x9eW\xa8\xb2lU', message=b'0.0.1', author=Person(name=b'Goldeneye Solutions', email=None, fullname=b'Goldeneye Solutions'), committer=Person(name=b'Goldeneye Solutions', email=None, fullname=b'Goldeneye Solutions'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=1396886527, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=1396886527, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\xec\x08\xa3\x05\x8fM9?\x81\x0eE!+\x9c\xfa\n8\xe8\xc3y', synthetic=True, metadata={'package_source': {'name': 'ja', 'version': '0.0.1', 'filename': 'ja-0.0.1.tgz', 'sha1': '31399c51d3024f6eb91c626a31a175dc30f343e5', 'date': '2014-04-07T16:02:07.453Z', 'url': 'https://registry.npmjs.org/ja/-/ja-0.0.1.tgz', 'sha256': '8101e284d5846e77f9698628da660e92f8627535c266a3aef28549b25e63a597', 'blake2s256': 'fcf5438822e12c8a6973b9ecb40df4978947edd675c4269d728d5279dc82b203'}, 'package': {'author': 'Goldeneye Solutions', 'name': 'ja', 'description': 'Compose a cross-platform application from various things.', 'version': '0.0.1', 'engines': {'node': '>=0.9.9'}, 'dependencies': {'ja-lib': '0.1.x'}, 'copyright': '© 2014 Goldeneye Solutions Inc.'}}, parents=[])
>>> gnu = Revision.from_dict(gnu_rev)
>>> gnu
Revision(id=b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3', message=b'swh-loader-package: synthetic revision message', author=Person(name=b'Software Heritage', email=b'robot@softwareheritage.org', fullname=b'Software Heritage'), committer=Person(name=b'Software Heritage', email=b'robot@softwareheritage.org', fullname=b'Software Heritage'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=944729610, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=944729610, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(', synthetic=True, metadata={'package': {'date': '944729610', 'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz'}}, parents=[])
>>> gnu.to_dict()
{'id': b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3', 'message': b'swh-loader-package: synthetic revision message', 'author': {'name': b'Software Heritage', 'email': b'robot@softwareheritage.org', 'fullname': b'Software Heritage'}, 'committer': {'name': b'Software Heritage', 'email': b'robot@softwareheritage.org', 'fullname': b'Software Heritage'}, 'date': {'timestamp': {'seconds': 944729610, 'microseconds': 0}, 'offset': 0, 'negative_utc': False}, 'committer_date': {'timestamp': {'seconds': 944729610, 'microseconds': 0}, 'offset': 0, 'negative_utc': False}, 'type': 'tar', 'directory': b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(', 'synthetic': True, 'metadata': {'package': {'date': '944729610', 'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz'}}, 'parents': []}

>>> from pprint import pprint
>>> pprint(gnu_rev)
{'author': {'email': b'robot@softwareheritage.org',
            'fullname': b'Software Heritage',
            'name': b'Software Heritage'},
 'committer': {'email': b'robot@softwareheritage.org',
               'fullname': b'Software Heritage',
               'name': b'Software Heritage'},
 'committer_date': {'negative_utc': False,
                    'offset': 0,
                    'timestamp': {'microseconds': 0, 'seconds': 944729610}},
 'date': {'negative_utc': False,
          'offset': 0,
          'timestamp': {'microseconds': 0, 'seconds': 944729610}},
 'directory': b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(',
 'id': b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3',
 'message': b'swh-loader-package: synthetic revision message',
 'metadata': {'package': {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz',
                          'date': '944729610'}},
 'parents': [],
 'synthetic': True,
 'type': 'tar'}
>>> pprint(gnu.to_dict())
{'author': {'email': b'robot@softwareheritage.org',
            'fullname': b'Software Heritage',
            'name': b'Software Heritage'},
 'committer': {'email': b'robot@softwareheritage.org',
               'fullname': b'Software Heritage',
               'name': b'Software Heritage'},
 'committer_date': {'negative_utc': False,
                    'offset': 0,
                    'timestamp': {'microseconds': 0, 'seconds': 944729610}},
 'date': {'negative_utc': False,
          'offset': 0,
          'timestamp': {'microseconds': 0, 'seconds': 944729610}},
 'directory': b':\xeb\xc2\x9e\xd1\xfc\xccJo/ \x10\xfb\x8eW\x88$\x06\xb5(',
 'id': b'D\x184\x88\xc0wL\xe3\xc9W\xfa\x19\xbai\\\xf1\x8aJB\xb3',
 'message': b'swh-loader-package: synthetic revision message',
 'metadata': {'package': {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz',
                          'date': '944729610'}},
 'parents': [],
 'synthetic': True,
 'type': 'tar'}
ardumont added a comment.Aug 9 2019, 4:37 PM

The gnu revision sounds fine!
I don't understand the error so far.

I'm not sure about the volume mount points though.

swh-loader:
   volumes:
   - "/home/archit/swh-environment/swh-loader-core:/src/swh-loader"

I'd be inclined to write it:

swh-loader:
   volumes:
   - "/home/archit/swh-environment/swh-loader-core:/src/swh-loader-core"   # <- swh-loader-core here

only docker-compose logs could say whether it's mounted correctly or not ;)

ardumont added a comment.Aug 9 2019, 4:52 PM

I don't understand the error so far.

I think that revision is fine so it must not be the one which fails.

P497 mentions 4 other revisions, so some more debugging statements is needed to list the 4 revisions.
At least one of those must be wrong in some way.

About · Developer information · Legal