Page MenuHomeSoftware Heritage
Differential D5878

Store last_eventful_visit_date
ClosedPublic
Actions

Authored by KShivendu on Jun 16 2021, 10:20 AM.

Details

Reviewers
vsellier
Group Reviewers
Reviewers
Commits
rDSEA35570b4284f6: Store last_eventful_visit_date
Summary

last_eventful_visit_date is a good candidate for sorting options and filters.
These changes will store this value in ES when OriginVisitStatus objects are received
by the swh.search journal client.

Diff Detail

Repository
rDSEA Archive search
Branch
last-eventful-visit-date
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22062
Build 34314: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 34313: arc lint + arc unit

Event Timeline

KShivendu created this revision.Jun 16 2021, 10:20 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 16 2021, 10:20 AM
swh-public-ci added a comment.Jun 16 2021, 10:24 AM

Build is green

Patch application report for D5878 (id=21059)

Rebasing onto 8b2c87f4e1...

Current branch diff-target is up to date.
Changes applied before test
commit 140db0a3b7cf6f041bdb4f89b6e2d4c70b6abffb
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/142/ for more details.

Harbormaster completed remote builds in B22050: Diff 21059.Jun 16 2021, 10:24 AM
KShivendu requested review of this revision.Jun 16 2021, 10:24 AM
KShivendu added inline comments.Jun 16 2021, 10:31 AM
swh/search/elasticsearch.py
148

Is "text" the correct type for snapshot_id ? I'm not sure. Please comment.

vsellier added a subscriber: vsellier.Jun 16 2021, 10:37 AM
vsellier added inline comments.
swh/search/elasticsearch.py
148

I think it should be a keyword

vsellier added inline comments.Jun 16 2021, 10:42 AM
swh/search/tests/test_search.py
283

A parameterized test could be useful to ensure having visits (eventful/uneventful) in any order gives the expected result
Please have a look at the equivalent on the scheduler:
https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_journal_client.py$418

KShivendu updated this revision to Diff 21073.Jun 16 2021, 7:30 PM
  • Add test for last_eventful_visit_date with permutations
  • Use type "keyword" for "snapshot_id"
swh-public-ci added a comment.Jun 16 2021, 7:34 PM

Build is green

Patch application report for D5878 (id=21073)

Rebasing onto 69364b3b0a...

First, rewinding head to replay your work on top of it...
Applying: Store last_eventful_visit_date
Applying: Add test for last_eventful_visit_date with permutations
Changes applied before test
commit ec4d81c092d605ec86152aa8b574893b408073aa
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 17:26:52 2021 +0000

    Add test for last_eventful_visit_date with permutations

commit a411dd078e5a9bceb9339d7762fca4e24d7fbb25
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/143/ for more details.

Harbormaster completed remote builds in B22062: Diff 21073.Jun 16 2021, 7:34 PM
KShivendu updated this revision to Diff 21074.Jun 17 2021, 9:11 AM

Fix scope of variables used in permutation test

swh-public-ci added a comment.Jun 17 2021, 9:15 AM

Build is green

Patch application report for D5878 (id=21074)

Rebasing onto 69364b3b0a...

First, rewinding head to replay your work on top of it...
Applying: Store last_eventful_visit_date
Applying: Add test for last_eventful_visit_date with permutations
Applying: Fix scope of variables used in permutation test
Changes applied before test
commit 3ab9b753bd787ad8e87985cf0ebb7966f6714379
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Thu Jun 17 07:11:08 2021 +0000

    Fix scope of variables used in permutation test

commit 5e9dedfcd0ca38ced119fdce72321626bb8b2ebb
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 17:26:52 2021 +0000

    Add test for last_eventful_visit_date with permutations

commit 7bc4a5c80e6842c175e8a0db80dddd8ead03c758
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/144/ for more details.

Harbormaster completed remote builds in B22063: Diff 21074.Jun 17 2021, 9:15 AM
vsellier added a subscriber: vlorentz.Jun 17 2021, 9:32 AM
vsellier added inline comments.
swh/search/tests/test_search.py
310

Is there a reason to use permutation instead of a parameterized test?
With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

@vlorentz WDYT ?

vlorentz added inline comments.Jun 17 2021, 9:32 AM
swh/search/tests/test_search.py
310

I agree

KShivendu added inline comments.Jun 17 2021, 10:15 AM
swh/search/tests/test_search.py
310

I read this https://stackoverflow.com/questions/18182251/does-pytest-parametrized-test-work-with-unittest-class-based-tests which says.

unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

Is this stackoverflow answer wrong ?

KShivendu added inline comments.Jun 17 2021, 10:17 AM
swh/search/tests/test_search.py
310

With this implementation the index is not reset between 2 iterations so after the first loop, the test will always be "adding a visit oldest or equal than what is in the index"

Ahh, I see. My bad.

vlorentz added inline comments.Jun 17 2021, 10:25 AM
swh/search/tests/test_search.py
310

This is not a unittest.TestCase-based test; it's just a regular class.

KShivendu updated this revision to Diff 21075.Jun 17 2021, 11:04 AM
  • Reset index at the end of permutation loops
swh-public-ci added a comment.Jun 17 2021, 11:09 AM

Build is green

Patch application report for D5878 (id=21075)

Rebasing onto 69364b3b0a...

First, rewinding head to replay your work on top of it...
Applying: Store last_eventful_visit_date
Applying: Add test for last_eventful_visit_date with permutations
Applying: Fix scope of variables used in permutation test
Applying: Reset index at the end of permutation loops
Changes applied before test
commit 41672915e628e2b9fc013bd5e9078de5a884f082
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Thu Jun 17 09:04:13 2021 +0000

    Reset index at the end of permutation loops

commit 302de2d260898589b3433d87d0bd34f1d55e9733
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Thu Jun 17 07:11:08 2021 +0000

    Fix scope of variables used in permutation test

commit 0e125330b37c3d13f50766739a3ca19ea82c6934
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 17:26:52 2021 +0000

    Add test for last_eventful_visit_date with permutations

commit 7c60c8fc4c570152419a992d38c86e754b0a38c2
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Jun 16 08:15:31 2021 +0000

    Store last_eventful_visit_date

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/145/ for more details.

Harbormaster completed remote builds in B22065: Diff 21075.Jun 17 2021, 11:09 AM
vsellier accepted this revision.Jun 17 2021, 2:19 PM

Thanks for the changes.

This revision is now accepted and ready to land.Jun 17 2021, 2:19 PM
Closed by commit rDSEA35570b4284f6: Store last_eventful_visit_date (authored by KShivendu). · Explain WhyJun 17 2021, 7:47 PM
This revision was automatically updated to reflect the committed changes.
KShivendu added a commit: rDSEA35570b4284f6: Store last_eventful_visit_date.
KShivendu mentioned this in D5883: Setup storage and store last revision/release date.Jun 21 2021, 7:58 AM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
search/
29 lines95%
Loading...
39 lines97%
Loading...
4 lines82%
Loading...
2 lines100%
Loading...
tests/
3 lines100%
Loading...
97 lines100%
Loading...

Diff 21073

View Options

swh/search/elasticsearch.py

Loading...
View Options

swh/search/in_memory.py

Loading...
View Options

swh/search/interface.py

Loading...
View Options

swh/search/journal_client.py

Loading...
View Options

swh/search/tests/test_journal_client.py

Loading...
View Options

swh/search/tests/test_search.py

Loading...
About · Developer information · Legal