Page MenuHomeSoftware Heritage

Store last_eventful_visit_date
ClosedPublic

Authored by KShivendu on Jun 16 2021, 10:20 AM.

Details

Summary

last_eventful_visit_date is a good candidate for sorting options and filters.
These changes will store this value in ES when OriginVisitStatus objects are received
by the swh.search journal client.

Diff Detail

Repository
rDSEA Archive search
Branch
last-eventful-visit-date
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22062
Build 34314: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 34313: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D5878 (id=21059)

Rebasing onto 8b2c87f4e1...

Current branch diff-target is up to date.
Changes applied before test
commit 140db0a3b7cf6f041bdb4f89b6e2d4c70b6abffb
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/142/ for more details.

swh/search/elasticsearch.py
148

Is "text" the correct type for snapshot_id ? I'm not sure. Please comment.

vsellier added inline comments.
swh/search/elasticsearch.py
148

I think it should be a keyword

swh/search/tests/test_search.py
283

A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order gives the expected result
Please have a look at the equivalent on the scheduler:
https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_journal_client.py$418

  • Add test for last_eventful_visit_date with permutations
  • Use type "keyword" for "snapshot_id"

Build is green

Patch application report for D5878 (id=21073)

Rebasing onto 69364b3b0a...

First, rewinding head to replay your work on top of it...
Applying: Store last_eventful_visit_date
Applying: Add test for last_eventful_visit_date with permutations
Changes applied before test
commit ec4d81c092d605ec86152aa8b574893b408073aa
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 17:26:52 2021 +0000

    Add test for last_eventful_visit_date with permutations

commit a411dd078e5a9bceb9339d7762fca4e24d7fbb25
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/143/ for more details.

Fix scope of variables used in permutation test

Build is green

Patch application report for D5878 (id=21074)

Rebasing onto 69364b3b0a...

First, rewinding head to replay your work on top of it...
Applying: Store last_eventful_visit_date
Applying: Add test for last_eventful_visit_date with permutations
Applying: Fix scope of variables used in permutation test
Changes applied before test
commit 3ab9b753bd787ad8e87985cf0ebb7966f6714379
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Thu Jun 17 07:11:08 2021 +0000

    Fix scope of variables used in permutation test

commit 5e9dedfcd0ca38ced119fdce72321626bb8b2ebb
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 17:26:52 2021 +0000

    Add test for last_eventful_visit_date with permutations

commit 7bc4a5c80e6842c175e8a0db80dddd8ead03c758
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/144/ for more details.

vsellier added inline comments.
swh/search/tests/test_search.py
310

Is there a reason to use permutation instead of a parameterized test?
With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

@vlorentz WDYT ?

swh/search/tests/test_search.py
310

I agree

swh/search/tests/test_search.py
310

I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work-with-unittest-class-based-tests which says.

unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

Is this stackoverflow answer wrong ?

swh/search/tests/test_search.py
310

With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

Ahh, I see. My bad.

swh/search/tests/test_search.py
310

This is not a unittest.TestCase-based test; it's just a regular class.

  • Reset index at the end of permutation loops

Build is green

Patch application report for D5878 (id=21075)

Rebasing onto 69364b3b0a...

First, rewinding head to replay your work on top of it...
Applying: Store last_eventful_visit_date
Applying: Add test for last_eventful_visit_date with permutations
Applying: Fix scope of variables used in permutation test
Applying: Reset index at the end of permutation loops
Changes applied before test
commit 41672915e628e2b9fc013bd5e9078de5a884f082
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Thu Jun 17 09:04:13 2021 +0000

    Reset index at the end of permutation loops

commit 302de2d260898589b3433d87d0bd34f1d55e9733
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Thu Jun 17 07:11:08 2021 +0000

    Fix scope of variables used in permutation test

commit 0e125330b37c3d13f50766739a3ca19ea82c6934
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 17:26:52 2021 +0000

    Add test for last_eventful_visit_date with permutations

commit 7c60c8fc4c570152419a992d38c86e754b0a38c2
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/145/ for more details.

Thanks for the changes.

This revision is now accepted and ready to land.Jun 17 2021, 2:19 PM
This revision was automatically updated to reflect the committed changes.