swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka).
These values are good candidates for building filters and sorting. this diff provides the code to
store these values when they're recieved by the swh.search journal client
Details
- Reviewers
vlorentz vsellier - Group Reviewers
Reviewers - Commits
- rDSEA8b2c87f4e1b4: Store nb_visits and last_visit_date
Diff Detail
- Repository
- rDSEA Archive search
- Branch
- visit-date-and-counter
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 21797 Build 33893: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 33892: arc lint + arc unit
Event Timeline
Build has FAILED
Patch application report for D5824 (id=20850)
Rebasing onto c4d6fed488...
Current branch diff-target is up to date.
Changes applied before test
commit 8c9bf75efb07895e7ec821a359e8bba6136085ed Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 7 15:17:22 2021 +0530 Store visit count and last visit date
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/115/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/115/console
Build has FAILED
Patch application report for D5824 (id=20855)
Rebasing onto c4d6fed488...
Current branch diff-target is up to date.
Changes applied before test
commit 4800883d37fa7d434c4fd923046a3fa8f6331441 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 7 15:17:22 2021 +0530 Store visit count and last visit date
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/116/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/116/console
Could you also add a test in test_search.py making sure nb_visit actually does get updated?
And also add filter and tests for last_visit_date.
I believe test_origin_nb_visit_update_search does it already.
Line 211 to 213 in test_search.py are for the same.
Please let me know if you think it's not enough.
And also add filter and tests for last_visit_date.
Done
- mypy: Fix errors with release >= v0.900 (commit made by @anlambert. I'm including it so that my builds don't fail)
- Store visit count and last visit date
- Add tests for last_visit_date
swh/search/interface.py | ||
---|---|---|
62–63 ↗ | (On Diff #20893) | It's unclear from the names they will returns visits *after* the given values (rather than equal). Could you rename the parameters and document them in the docstring? |
swh/search/tests/test_search.py | ||
235–242 ↗ | (On Diff #20893) | It's hard to spot the difference. Could you make the date constants and comment that the difference is the years? |
swh/search/interface.py | ||
---|---|---|
62–63 ↗ | (On Diff #20893) | Actually, I thought of only going for equality because at the moment these two parameters are only used in tests for the atomic merges and later they'll be replaced with the query parser. In the query, users can specify < or > and Elasticsearch queries will be generated accordingly. So can I keep it as it is ?
Sure. |
Build has FAILED
Patch application report for D5824 (id=20913)
Rebasing onto 3e129a3f48...
Current branch diff-target is up to date.
Changes applied before test
commit 7eb2cccb51e69607c254d86f73df3bb2e08e1513 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 16:12:08 2021 +0530 Improve documentation + tests and fix painless script commit 6ea5d4df4e541b3f952d5f8169119a9ad916db82 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Wed Jun 9 16:18:43 2021 +0530 Add tests for last_visit_date commit 19521597d7bcdbbf59ff25c3c1b257247fc62af8 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 7 15:17:22 2021 +0530 Store visit count and last visit date
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/121/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/121/console
Build has FAILED
Patch application report for D5824 (id=20915)
Rebasing onto 3e129a3f48...
Current branch diff-target is up to date.
Changes applied before test
commit 9911e8bcee0f0bee683b5c1e52bbb73e4e573b12 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 16:12:08 2021 +0530 Improve documentation + tests and fix painless script commit 6ea5d4df4e541b3f952d5f8169119a9ad916db82 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Wed Jun 9 16:18:43 2021 +0530 Add tests for last_visit_date commit 19521597d7bcdbbf59ff25c3c1b257247fc62af8 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 7 15:17:22 2021 +0530 Store visit count and last visit date
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/122/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/122/console
Handle difference in iso formats of datetime and elasticsearch(painless) using .replace
Build has FAILED
Patch application report for D5824 (id=20916)
Rebasing onto 3e129a3f48...
Current branch diff-target is up to date.
Changes applied before test
commit b42a650dda27bc4d490d10f1e7cef7974ec15953 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 16:12:08 2021 +0530 Improve documentation + tests and fix painless script commit 6ea5d4df4e541b3f952d5f8169119a9ad916db82 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Wed Jun 9 16:18:43 2021 +0530 Add tests for last_visit_date commit 19521597d7bcdbbf59ff25c3c1b257247fc62af8 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 7 15:17:22 2021 +0530 Store visit count and last visit date
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/123/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/123/console
Build has FAILED
Patch application report for D5824 (id=20935)
Rebasing onto 3e129a3f48...
Current branch diff-target is up to date.
Changes applied before test
commit 4fa01d6af279b75ee953ff2edf96fe8cf75caa43 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 17:18:54 2021 +0000 Attempt to fix error commit 4c012e025b300f79ed463507a8da2887a347626e Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/124/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/124/console
Build has FAILED
Patch application report for D5824 (id=20937)
Rebasing onto 3e129a3f48...
Current branch diff-target is up to date.
Changes applied before test
commit bfcef57673edde2898b2450c46732332fd2456cb Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 18:00:25 2021 +0000 Add field type in es commit 4fa01d6af279b75ee953ff2edf96fe8cf75caa43 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 17:18:54 2021 +0000 Attempt to fix error commit 4c012e025b300f79ed463507a8da2887a347626e Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/126/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/126/console
Build is green
Patch application report for D5824 (id=20943)
Rebasing onto 3e129a3f48...
Current branch diff-target is up to date.
Changes applied before test
commit 1555107a272d69ab06a251929cef9e7b0b3037c8 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/127/ for more details.
swh/search/interface.py | ||
---|---|---|
62–63 ↗ | (On Diff #20893) | The point of having them here is to be able to use them even outside tests while we are working on the query language. Otherwise we would just use origin_dump in tests. |
Build has FAILED
Patch application report for D5824 (id=20997)
Rebasing onto f3739ba16d...
First, rewinding head to replay your work on top of it... Applying: journal_client: Store visit count and last visit date Using index info to reconstruct a base tree... M swh/search/elasticsearch.py M swh/search/tests/test_search.py Falling back to patching base and 3-way merge... Auto-merging swh/search/tests/test_search.py Auto-merging swh/search/elasticsearch.py CONFLICT (content): Merge conflict in swh/search/elasticsearch.py Patch failed at 0001 journal_client: Store visit count and last visit date Resolve all conflicts manually, mark them as resolved with "git add/rm <conflicted_files>", then run "git rebase --continue". You can instead skip this commit: run "git rebase --skip". To abort and get back to the state before "git rebase", run "git rebase --abort".
Rebase failed (ret=1)!
Could not rebase; Attempt merge onto f3739ba16d...
Already up to date.
Changes applied before test
commit c2b26194a888310e28f603a028003673278c6707 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 15:37:55 2021 +0000 Use gte filter instead of equality commit 1555107a272d69ab06a251929cef9e7b0b3037c8 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/132/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/132/console
Build is green
Patch application report for D5824 (id=20998)
Rebasing onto f3739ba16d...
Current branch diff-target is up to date.
Changes applied before test
commit 4bbf80152638648402941a92fe6c0a0b4b1687a7 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 15:48:09 2021 +0000 journal_client: Store visit count and last visit date
See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/133/ for more details.
Build has FAILED
Patch application report for D5824 (id=21000)
Rebasing onto f3739ba16d...
Current branch diff-target is up to date.
Changes applied before test
commit aa1b1cdc2c2f405a9277528688a71a440632b1ce Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 15:37:55 2021 +0000 Use gte filter instead of equality commit e34daab3a35e2bfa364c490461f4d939487465bd Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/134/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/134/console
Build has FAILED
Patch application report for D5824 (id=21001)
Rebasing onto f3739ba16d...
Current branch diff-target is up to date.
Changes applied before test
commit 1d181f8b335aded8afe8312c50971a9b5584b64d Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 15:37:55 2021 +0000 Use gte filter instead of equality commit e34daab3a35e2bfa364c490461f4d939487465bd Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/135/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/135/console
Build has FAILED
Patch application report for D5824 (id=21002)
Rebasing onto f3739ba16d...
Current branch diff-target is up to date.
Changes applied before test
commit bf72257d1f784e58262585d7ddd922eb7bc8f01f Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 16:40:55 2021 +0000 nb_vist -> nb_visits commit 1d181f8b335aded8afe8312c50971a9b5584b64d Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 15:37:55 2021 +0000 Use gte filter instead of equality commit e34daab3a35e2bfa364c490461f4d939487465bd Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/136/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/136/console
Build is green
Patch application report for D5824 (id=21003)
Rebasing onto f3739ba16d...
Current branch diff-target is up to date.
Changes applied before test
commit e95fbbc76ef02999ef0cb4d0c5b0d6042f3bf163 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 16:40:55 2021 +0000 nb_vist -> nb_visits commit 1d181f8b335aded8afe8312c50971a9b5584b64d Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Mon Jun 14 15:37:55 2021 +0000 Use gte filter instead of equality commit e34daab3a35e2bfa364c490461f4d939487465bd Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 journal_client: Store visit count and last visit date Summary: swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These values are good candidates for building filters and sorting. this diff provides the code to store these values when they're recieved by the swh.search journal client Reviewers: vlorentz, vsellier, #reviewers Subscribers: anlambert, ardumont Differential Revision: https://forge.softwareheritage.org/D5824
See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/137/ for more details.
One last change, then you can land this
swh/search/interface.py | ||
---|---|---|
75–78 ↗ | (On Diff #21003) |
Build is green
Patch application report for D5824 (id=21007)
Rebasing onto f3739ba16d...
Current branch diff-target is up to date.
Changes applied before test
commit 8b2c87f4e1b454a250553f95277dac2b13329ca5 Author: KShivendu <shivendu@iitbhilai.ac.in> Date: Thu Jun 10 15:25:25 2021 +0000 Store nb_visits and last_visit_date swh.storage passes visit count and visit date for each OriginVisitStatus through swh.journal(kafka). These two values are good candidates for filters and the sorting feature so this commit provides the code to store these values when they are recieved by the swh.search journal client
See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/139/ for more details.