Build is green
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Mar 16 2021
Mar 15 2021
Mar 11 2021
Mar 5 2021
Feb 17 2021
Feb 16 2021
Rework commit message (aligns with diff)
In D5071#128386, @ardumont wrote:I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader
I don't think anything would break. I'm just not sure the "indirection" would be clear in terms of code readability...
I was wrong. mypy is not happy.
Some signatures would need changing, notably the load, prepare, prepare_origin_visit to drop the spurious (i think) {*args, **kwargs} we are declaring.
(I don't think they are still used any more now, aside for the extra logging arguments)But still, i'd be confortable if we go that way to do it in another diff (maybe).
In D5071#128386, @ardumont wrote:I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader
I don't think anything would break. I'm just not sure the "indirection" would be clear in terms of code readability...
I was wrong. mypy is not happy.
Some signatures would need changing, notably the load, prepare, prepare_origin_visit to drop the spurious (i think) {*args, **kwargs} we are declaring.
(I don't think they are still used any more now, aside for the extra logging arguments)But still, i'd be confortable if we go that way to do it in another diff (maybe).
$ tox -e mypy GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/setup.py ... mypy run-test: commands[0] | mypy swh swh/loader/package/loader.py:289: error: Signature of "load" incompatible with supertype "BaseLoader" swh/loader/package/loader.py:381: error: Argument "date" to "OriginVisit" has incompatible type "Optional[datetime]"; expected "datetime" swh/loader/package/loader.py:570: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime" swh/loader/package/loader.py:692: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime" swh/loader/package/loader.py:727: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime" swh/loader/package/loader.py:756: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime" swh/loader/package/pypi/loader.py:151: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" swh/loader/package/npm/loader.py:177: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" swh/loader/package/nixguix/loader.py:218: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" swh/loader/package/deposit/loader.py:208: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" swh/loader/package/deposit/loader.py:244: error: Signature of "load" incompatible with supertype "BaseLoader" swh/loader/package/debian/loader.py:237: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" swh/loader/package/cran/loader.py:129: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" swh/loader/package/archive/loader.py:170: error: Item "None" of "Optional[datetime]" has no attribute "isoformat" Found 14 errors in 8 files (checked 71 source files)
I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader
Build is green
Adapt according to review:
- Drop swh.loader.pattern and move class Loader in swh.loader.core.loader module
- Drop unneeded self.create_authorities, self.create_fetchers
@ardumont points out that the base PackageLoader doesn't inherit from BaseLoader, which explains the new (common) base class. I think the new class could just as well be next to BaseLoader, and doesn't warrant the introduction of a pattern module.
@ardumont points out that the base PackageLoader doesn't inherit from BaseLoader, which explains the new (common) base class. I think the new class could just as well be next to BaseLoader, and doesn't warrant the introduction of a pattern module.
Are the new pattern module / pattern.Loader class really needed? It looks like these methods could live in the BaseLoader class directly.
This is great, thanks!
Build is green
Add missing test on cli run edge case
Build is green
Fix unused import
Build has FAILED
- Add missing test on Deposit.from_configfile
- Drop unneeded conditional in cli
Feb 15 2021
Feb 12 2021
Feb 9 2021
Feb 8 2021
Feb 5 2021
Nov 20 2020
Nov 18 2020
Nov 3 2020
Oct 26 2020
Sep 25 2020
Our logging handler swh.core.logger.JournalHandler already knows how to pull some metadata from the celery tasks:
...
Build is green
Adapt according to suggestion
Sep 24 2020
In D4012#99590, @ardumont wrote:To be clear, my main issue today, when I try to look through our logs to
investigate or plain read what's going on (after a deployment for example), I
don't have any clues immediately...In my mind, the kibana information is not enough by itself, so i think i need
to cross information with say sentry to have some more context... It's
currently quite frustrating... up to an eventual point of, "oh well, I have
some other urgent matters somewhere else..." (sometimes I push through but
sometimes, I fail).
Build is green
Simplify to just one log statement
ok, i'll adapt
i would not be against a nudge in the right direction to actually improve the logging
In D4012#99525, @olasd wrote:I don't think the origin url and visit type should be sent in the task result; they're arguments of the task already.
If we want them logged by the worker when the task ends (which I agree would be useful), then we should improve logging on the worker/celery side to show some of the task arguments (for instance, if there's a "url" argument) instead / in addition of the task id.
I don't think the origin url and visit type should be sent in the task
result; they're arguments of the task already.
I don't think the origin url and visit type should be sent in the task result; they're arguments of the task already.
Sep 23 2020
Sep 22 2020
I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.
Aug 31 2020
I'm afraid the only way to properly solve this is to wait until we stop writing metadata to the revision table
Aug 19 2020
Jul 27 2020
Jul 20 2020
Jul 17 2020
Jul 16 2020
Jul 10 2020
Reopening as i'm still refactoring/cleaning up more modules.