Build has FAILED

Build has FAILED

Looks good to me.

Build is green

It seems not to be necessary (besides not being the proper fix, if any)

Build is green

Build is green

Update test-filter docstring

• Remove the check on hasattr as all storage implementations have the clear_buffers
• Add the clear_buffers implementations on filter proxy storage (+ tests)

If you add clear_buffers to the basic signature of Storage, I think you can skip all the hasattrs.

If you add clear_buffers to the basic signature of Storage, I think you can skip all the hasattrs.

Build is green

• Rename method clear to clear_buffers
• Add clear_buffers on all storage endpoints, passthrough for proxy storage, noop endpoint for main backend implementations
• Explicit the warning about losing data when calling the clear_buffers endpoint
• complete tests with directories and skipped_contents

This is missing tests for directories and snapshots :)

Closed by rDWAPPS62041880dd71b8242df179206431803ea8aa8d1a

This is missing tests for directories and snapshots :)

Right now, heading for 2. for now as the solution for 3. is still a pending question [2]

Build is green

Alright. I've done a build test in unstable-swh and buster-swh and both builds have succeeded (at least until the point of tests, which will need a new tag). Thanks!

Build has FAILED

If the error is caused by an invalid argument, it should raise StorageArgumentException, not ValueError.

I'll start with a general reasoning about origin visit vs. origin visit state objects in our "conceptual" data model, as it was sprinkled throughout my comment initially.

Nevertheless, a lot of tests are now failing in storage since the latest model updates.

This test did not fail with swh-model 0.0.64, see https://jenkins.softwareheritage.org/job/DSTO/job/tests/1085/consoleFull and it did not fail too when I execute it locally on current master of swh-storage.

Build is green

Build is green

Rebase

It looks you misread what I meant: I was talking about a new OriginVisitUpdate with a snapshot "inconsistent" with the previous snapshot reported by the previous OriginVisitUpdate for the same visit.

In T2310#43182, @douardda wrote:

• do we allow an OriginVisitUpdate(status='ongoing', snapshot=yyy) with the snapshot yyy not a superset of a previous update?

It doesn't make sense to have this, but I'm not sure we should care.

I think this is a rather simple check to implement so I don't see why not do it. Intrinsic robustness is always (if not over complex) a good thing add.

Indeed. Then it would make sense to also give the full path to dir_filter, instead of adding a redundant attribute.

In T2310#43131, @ardumont wrote:

Thanks for the questions. I'm unsure about some questions and i replied as best
i could.

do we allow an OriginVisitUpdate(status='ongoing', snaphost=None)? what would
be the meaning of this?

Yes. It means "loading started, so no snapshot yet".
That sounds sensible ;)

In T2310#43129, @vlorentz wrote:

We currently don't have "created" (so no "start" either), but it would make sense to create it.

Regarding this model, a few questions come to my mind:

• do we allow an OriginVisitUpdate(status='ongoing', snaphost=None)? what would be the meaning of this? or do we enforce one just after the created step to model the start transition?

This could mean these things:

1. on a first update, to mean the visit was created (but we don't need it if we have a "created" state)

Yes that's how I do it

In D2960#71669, @ardumont wrote:

I can define a dir_filter function but how could i get the values i need, like the full path generated inside from_disk, without touching the implementation?

I'm not sure i understand the question.
from_disk has a save_path parameter. If set to True, the paths are stored alongside the output result.
Isn't that enough?

Build is green

Build is green

Yes that's how I do it

I can define a dir_filter function but how could i get the values i need, like the full path generated inside from_disk, without touching the implementation?

In D2960#71656, @ardumont wrote:

I think in the end, this should be implemented with the dir_filter parameter instead.

Please, check that you can do this. If you can, this will, there is no need to change the current implementation.
And instead, either:

• not touch swh.model at all, then define your ignore_path function in swh-scanner and use it when calling from_disk.
• if you think, this can be a shared behavior (i guess it can), define a function callable here in swh-model (like accept_all_directories and the other ones next to it).

In both cases, this won't touch Directory.from_disk though.

Build is still broken even with rebasing

Build is still broken even with rebasing

Build has FAILED

Modified Fixtures

I think in the end, this should be implemented with the dir_filter parameter instead.