Rebase

• origin_search: Polish code related to sort_by

• origin_search: Allow sorting with multiple fields

• interface: Maintain SORT_BY_OPTIONS list

• test_search: Improve tests for sort_by

• Improve commit body

Limit each line in commit message to 80 chars

Update commit messsage body

Can you also please also take care of updating it at the same time?

• rename the type parameter to date_type
• add .vscode to .gitignore

Is it okay if I add .vscode in .gitignore. It often gets included by mistake. swh-indexer, swh-storage and swh-web already have it in their .gitignore files.

• Improve code quality

• Improve code quality (as suggested by @anlambert)

• Changes suggested by vlorentz

• Throw error in absence of storage config

• Add test for fetch_last_revision_release_date in journal_client
• Add missing arguments and tests related to last_release_date

• Include last revision/release date only when available

• merge the fetch_last_*_date methods
• Use snapshot_get_all_branches instead of snapshot_get_branches
• Fix diff/commit description/message
• Use in-memory backend in test_cli.py

Rebase and only include new commit

• Store last_eventful_visit_date
• Add last_revision_daate and last_release_date

Squash commits

• Load swh.storage from config

• Reset index at the end of permutation loops

Fix scope of variables used in permutation test

• Add test for last_eventful_visit_date with permutations
• Use type "keyword" for "snapshot_id"

Fix origin_search() documentation and squash commits

Match origin_search signatures

Re-adjust position of "noqa" for painless script (after applying dedent)

Fix mypy issues

Use gte filter instead of equality

Pull from origin and rebase branch

• Use gte filter instead of equality

Update docstring to fix sphinx warnings

• Add field type in es

An attempt to fix error

Handle difference in iso formats of datetime and elasticsearch(painless) using .replace

Fixes for failing tests

• use compareTo function instead of <. This should fix the painless script

• Improve documentation + tests and fix painless script

• mypy: Fix errors with release >= v0.900 (commit made by @anlambert. I'm including it so that my builds don't fail)
• Store visit count and last visit date
• Add tests for last_visit_date

In D5824#148743, @vlorentz wrote:

Could you also add a test in test_search.py making sure nb_visit actually does get updated?

Include missing commit

Add tests for last_visit_date

Use noqa instead of multiple """

Use """ to overcome line length limits for writing long Painless scripts statements

Add code to test painless script for merging values atomically

Use painless script to atomically merge nb_visit and last_visit_date (max values)

In D5824#148471, @vlorentz wrote:

There is no guarantee on the order of visits, so you should explicitly prevent nb_visit and last_visit_date from being set to a lower value in origin_update.

Hi @vlorentz, I suspect this diff would have also introduced an issue similar to the one mentioned in T3209. What do you suggest?

If you face this issue, try restarting the containers using docker-compose down and docker-compose up.

Hey @vlorentz, the sentry link isn't working (or maybe isn't publically accessible).

Rebase

Use set instead of sorting

In D5618#143247, @vlorentz wrote:

In D5618#143199, @KShivendu wrote:

assert sorted(expected) == sorted(writer.objects)

These are of type ImmutableDict and hence can't be sorted directly

What is? It works for me.

• Convert ListProxy to List and compare with []
• Use sorted function and populate writer with TEST_OBJECTS

assert sorted(expected) == sorted(writer.objects)

These are of type ImmutableDict and hence can't be sorted directly

assert writer.privileged_objects == []

throws an error because writer.privileged_objects is of type ListProxy and can't be directly compared to a list

Improve time complexity using sorting