Page MenuHomeSoftware Heritage

loader.core.tests: Add loader core fixtures to ease loaders' tests
ClosedPublic

Authored by ardumont on Oct 3 2018, 5:52 PM.

Details

Summary

This adds 2 mixins:

  • One to setup the loader data to ingest and check for the state after loading
  • Another one to inhibit the self.storage calls from the loader-core implementation

Related T1238

Test Plan

None for now (I focused on refactoring the modules depending on it). I
will add some later if i found out how to actually do it.

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I like it, especially after seeing the number of lines removed in your other diffs.

Some nitpicking in inline comments below.

swh/loader/core/tests/__init__.py
40 ↗(On Diff #1430)

You could rewrite the argument list as:

def setUp(self, archive_name, *, start_path, filename=None,
          resources_path='resources', prefix_tmp_folder_name=''):

It makes start_path a keyword-only but mandatory parameter.

This way you don't need to check start_path.

49 ↗(On Diff #1430)

You can drop the v argument. It shouldn't be an issue unless the test case calling it has a huge file list.

93 ↗(On Diff #1430)

You might want to add (releases, expected_releases) as third argument to provide a helpful error message in case it fails.

110 ↗(On Diff #1430)

same

140 ↗(On Diff #1430)

same

186 ↗(On Diff #1430)

What does this comment mean?

ardumont added inline comments.
swh/loader/core/tests/__init__.py
40 ↗(On Diff #1430)

\m/

93 ↗(On Diff #1430)

I don't have much else to say than the default message... :)

186 ↗(On Diff #1430)

No idea what i meant, removing!

ardumont marked 2 inline comments as done.

Adapt according review

  • tests: Make start_path a mandatory keyword argument
  • tests: Remove unneeded verbose flag for uncompression instruction
  • tests: Remove useless comments
  • tests: Move comment to its right location
ardumont added inline comments.
swh/loader/core/tests/__init__.py
191 ↗(On Diff #1444)

ah, that was what i meant... the key is the 'type'!

I like it, especially after seeing the number of lines removed in your other diffs.

It's nice to drop code! Less to maintain \m/

  • tests: Add useful comments

Just a couple nitpicks but otherwise that's great and it will be really useful!

swh/loader/core/tests/__init__.py
31 ↗(On Diff #1444)

Clearly indicating in the docstring the behavior when filename is None would be useful.

168 ↗(On Diff #1444)

I would have written "sent to storage" here

203 ↗(On Diff #1444)

I would have used "col.append(o)" here

23 ↗(On Diff #1430)

s/variable/variables/

ardumont marked 3 inline comments as done.
  • tests: Clarify BaseLoaderTest docstring
  • tests: Simplify internal implementation
swh/loader/core/tests/__init__.py
31 ↗(On Diff #1444)

Yes.

Is the following clear enough?

filename (str/None): Name of the filename/folder once the
    archive is uncompressed. When the filename is not
    provided, the archive name is used as a derivative. This
    is used both for the self.repo_url and
    self.destination_path computation (this one only when
    provided)
203 ↗(On Diff #1444)

sold!

swh/loader/core/tests/__init__.py
31 ↗(On Diff #1444)

Yes, that's pretty clear ! :-)

swh/loader/core/tests/__init__.py
31 ↗(On Diff #1444)

lgtm. Optional[str] instead of str/None if you want to write it in a PEP484-like way.

ardumont marked an inline comment as done.
  • tests: Use more compliant type description
swh/loader/core/tests/__init__.py
31 ↗(On Diff #1444)

Thank you!

  • tests: Add the attribute 'fs' on those derivative tests
  • tests: Add tests around new fixtures
  • tests: Add failing cases
zack added inline comments.
README.md
7

What's the point of having this list here? it's going to be a pain to maintain (and hence will probably get out of date soon).

I suggest to remove the list of rev deps alltogether.

ardumont added inline comments.
README.md
7

The point is people going through our loaders' docs could start by that one...
Alphabetically, it's the first.

Historically, i started that listing as a todo list.
When our loaders did not share the core's behavior.

Ok for removing it if it's not considered interesting enough.

README.md
7

I don't it's uninteresting. I'm just saying it's gonna be a pain to maintain and will become out of date quickly.
Reverse dependency information is indeed useful, but we already have a place where it is stored (and generated automatically).
I hear your point about not fooling people into thinking this is a real load; but I think the header to the core loader that I've recently added (which reads "Low-level loading utilities and helpers used by other loaders.") should be enough to dispel that risk. Feel free to use it at the beginning of the README if you see fit.

ardumont added inline comments.
README.md
7

It's a deal!
Thanks.

  • README.md: Simplify description
ardumont retitled this revision from loader.core: Add base loader core mixins to ease loader testing to loader.core.tests: Add loader core fixtures to ease loaders' tests.Oct 8 2018, 3:24 PM

Rebase

  • README.md: Reference loader-pypi
  • loader.core: Add base loader core mixins to ease loader testing
  • loader.tests: Improve mixin to support more loaders
  • tests: Make start_path a mandatory keyword argument
  • tests: Remove unneeded verbose flag for uncompression instruction
  • tests: Remove useless comments
  • tests: Move comment to its right location
  • tests: Add useful comments
  • tests: Clarify BaseLoaderTest docstring
  • tests: Simplify internal implementation
  • tests: Use more compliant type description
  • tests: Add the attribute 'fs' on those derivative tests
  • tests: Add tests around new fixtures
  • tests: Add failing cases

besides the name mangling from hell, lgtm

swh/loader/core/tests/__init__.py
184 ↗(On Diff #1499)

The thing I don't like here is the mangled attribute __objects...
(Shity) name mangling in python is a (bad) joke, for me.

This revision is now accepted and ready to land.Oct 8 2018, 3:54 PM
ardumont added inline comments.
swh/loader/core/tests/__init__.py
184 ↗(On Diff #1499)

I'd rather we fix this now then (the part you mentioned about pytest can be dealt with without impact, but this is not easily the case here). I'd like to settle on this once and for all and no longer have to worry about those ever again.

I can think of a middle-ground to reduce the impact on the other loader's tests though!

Note: The prefixed names are here because the internal loader-core uses those without all_ already.
They must have been badly named in the first place. As they are internal they should have been probably named with __ in the first place.

  • core.tests: Address badly internal name variables issue
ardumont added inline comments.
README.md
12

I'll commit later this part that cleans up this.

  • core.tests: Address badly internal name variables issue

Add coverage on new tests endpoints

  • core.tests: Allow msg to be defined in case exception is raised
  • core.tests: Do not use deprecated assertEquals method
  • core.tests: Allow to simply compute the archive's path
  • core.tests: Rename inner state to _state variable
  • core.tests: Use sphinx keyword
  • README.md: Simplify module description