Rebase and only include new commit

• Store last_eventful_visit_date
• Add last_revision_daate and last_release_date

Squash commits

• Load swh.storage from config

• Reset index at the end of permutation loops

Fix scope of variables used in permutation test

• Add test for last_eventful_visit_date with permutations
• Use type "keyword" for "snapshot_id"

Fix origin_search() documentation and squash commits

Match origin_search signatures

Re-adjust position of "noqa" for painless script (after applying dedent)

Fix mypy issues

Use gte filter instead of equality

Pull from origin and rebase branch

• Use gte filter instead of equality

Update docstring to fix sphinx warnings

• Add field type in es

An attempt to fix error

Handle difference in iso formats of datetime and elasticsearch(painless) using .replace

Fixes for failing tests

• use compareTo function instead of <. This should fix the painless script

• Improve documentation + tests and fix painless script

• mypy: Fix errors with release >= v0.900 (commit made by @anlambert. I'm including it so that my builds don't fail)
• Store visit count and last visit date
• Add tests for last_visit_date

In D5824#148743, @vlorentz wrote:

Could you also add a test in test_search.py making sure nb_visit actually does get updated?

Include missing commit

Add tests for last_visit_date

Use noqa instead of multiple """

Use """ to overcome line length limits for writing long Painless scripts statements

Add code to test painless script for merging values atomically

Use painless script to atomically merge nb_visit and last_visit_date (max values)

In D5824#148471, @vlorentz wrote:

There is no guarantee on the order of visits, so you should explicitly prevent nb_visit and last_visit_date from being set to a lower value in origin_update.

Hi @vlorentz, I suspect this diff would have also introduced an issue similar to the one mentioned in T3209. What do you suggest?

If you face this issue, try restarting the containers using docker-compose down and docker-compose up.

Hey @vlorentz, the sentry link isn't working (or maybe isn't publically accessible).

Rebase

Use set instead of sorting

In D5618#143247, @vlorentz wrote:

In D5618#143199, @KShivendu wrote:

assert sorted(expected) == sorted(writer.objects)

These are of type ImmutableDict and hence can't be sorted directly

What is? It works for me.

• Convert ListProxy to List and compare with []
• Use sorted function and populate writer with TEST_OBJECTS

assert sorted(expected) == sorted(writer.objects)

These are of type ImmutableDict and hence can't be sorted directly

assert writer.privileged_objects == []

throws an error because writer.privileged_objects is of type ListProxy and can't be directly compared to a list

Improve time complexity using sorting

Hii @moranegg @zack I'm working on this task and I noticed that we are using pybadges for implementing swh-badges which isn't the case with shields.io. Based on what I understand, these badges have to be re-implemented as shields.io services will use swh apis to generate the dynamic parts of the badges.

Do you some more tests or this task can be declared as resolved?

Should I add the names of other contributors as well?

Should I add the names of other contributors as well?

Rebase before pushing

Add same test for regular kafka writer

Hey @hm, I don't see any typo in https://forge.softwareheritage.org/source/swh-lister/browse/master/docs/run_a_new_lister.rst$50-51. If you do, please make a revision to fix the same.

Updating D5419: Cassandra: Deduplicate lists passed to *_add endpoints

Hey @faux it's the same as T3145

Updating D5419: Cassandra: Deduplicate lists passed to *_add endpoints

Updating D5419: Cassandra: Deduplicate lists passed to *_add endpoints

we should build a single model object for the top-level dir, and either output its SWHID, or traverse it (without recomputing SWHIDs) to output all of it

Updating D5420: cli/identify: Use TerminalColor Enum and change recursive flag's description

What do you think should be done for releases?

Updating D5419: Cassandra: Fixed failing tests

These objects has an id attribute. You can use it for deduplication (eg. via a dict)

Not all objects had id so I used swhid. But some of the tests are failing.

Hi @hm, Your changes have been accepted but they aren't merged yet. Please do a git push to get your commit(s) merged :)

Updating D5419: Cassandra: Deduplicate lists passed to *_add endpoints

Updating D5419: Cassandra: Deduplicate lists passed to *_add endpoints

I just discovered that tests failed because the set's internally used hash function throws an error for if a dictionary is passed.
Do you know any other trick which can do the de-duplication in one line? Or should I just create a common function to loop over the list and find the unique ones?

I don't think you need to convert the sets back to lists

I did that and got type errors from mypy.
Imo, it's okay to leave it as list(set(..)) because anyhow it gets transformed back into a list within the next 2-3 lines.
What do you think?

Hi guys. Any pointers on where to start?

I might be wrong but, I think it has been completed. Check out these :