Build is green

^x^s...

Build is green

What about "incremental", for consistency with the listers' terminology?

typo

I'd rather name it origin_snapshot_get_all, what do you think?

I also hesitated to use that name, I used origin_snapshot_get to match naming of other methods in storage interface but I agree origin_snapshot_get_all is more explicit about what the method does, will rename then.

Build is green

Build is green

Rebase

Build is green

• maven-lister: Fix copyright in header

Ah, it's not for all origins though...
I tried with other origins which demonstrates the same issue [1] and they did not fail...

Build is green

Drop unused period parameter in cli...

Drop spurious change

Overall it looks very good to me. There is room for improvement, for sure, but let's put this to work and see how it performs.

Drop no longer defined --period 10 call for non default runner.

Finally, having to implement the computation of the key like that is a bit depressing...
(who said "SWHID everywhere!"?)

Build has FAILED

• Adapt according to suggestions
• drop unused --period flag in the cli

In D6520#170772, @vlorentz wrote:

In D6520#170756, @ardumont wrote:

What's not clear in the description that need rework so it's clearer?

What isn't clear to me is why we need to implement this, because the scheduler already does it.

In D6520#170772, @vlorentz wrote:

In D6520#170756, @ardumont wrote:

What's not clear in the description that need rework so it's clearer?

What isn't clear to me is why we need to implement this, because the scheduler already does it.

nvm, @olasd just explained to me this used to be done manually

In D6520#170756, @ardumont wrote:

What's not clear in the description that need rework so it's clearer?

Drop --period 10 which is unused.

Build has FAILED

I'd rather name it origin_snapshot_get_all, what do you think?

Document a bit more the value_deserializer and add a test for it

You checked on the whole journal, right?

I don't really understand what this is for

I don't really understand what this is for, but lgtm.

In D6395#170648, @borisbaldassari wrote:

In D6395#170281, @vlorentz wrote:

You use three similar but subtly different conditionals:
Could you somehow unify them?

No, I don't think so, or not without risking losing entries.

Basically we parse the big index file, and as we get through some entries can be yielded immediately (jars), and for others we need some post-treatment (scm), mainly for uniqueness because we have a *lot* of duplicates here. So we store them during the parsing, and *after* we start yielding the scm's.

If we get interrupted during the first part (index parsing), we need to be able to yield again jars from where we stopped (but at this point in time we still haven't yielded any scm). If we get interrupted during the second part (scm post-treatment), all jars have already been yielded and we only need to yield the scm. Interestingly if we are interrupted in-between (say we finished parsing the index, yielded all jars but not scm), we still need to re-read the index to re-build the list of unique scm's and then yield them.

So IMHO we really need 2 different counters, and the slightly different conditionals. I may be missing something obvious as I've been struggling with that for some time, do you think it could be improved or optimised?

why? What's the blocking point?

Drop unused variables

ok

"ok then"

asking for review even if tests are expected to fail because it depends on D6565 (in swh-journal)

In D6564#170522, @ardumont wrote:

why if i may ask?

One copyright header remark inline though.

Well, fine to me.
I'll let @vlorentz have the final word.

A small description to mention why it's now useless would be great.

Build is green

rebase

For this one, i had a thought about it recently and it should be enough to trigger a
run from scratch (loader has the flag for it which is tested but got never used).

So i've had a look in the end [1], it's indeed the history altered issue (2.).

From the top of my head, 2 possible issues:

1. svn external properties on the origin. That will get detected by the loader and

raise.

Build is green

• maven-lister: Fix README, 88 chars, minor typos

In D6395#170281, @vlorentz wrote:

You use three similar but subtly different conditionals:
Could you somehow unify them?

No, I don't think so, or not without risking losing entries.

In D6395#170099, @ardumont wrote:

Thanks for the great work.
This looks mostly ok to me.

Thanks, it's appreciated. :-)

In D6395#170095, @ardumont wrote:

I can't manage to test the full toolchain (maven lister - maven loader),

why? What's the blocking point?

I've found a great doc to run the lister in a docker-dev environment, and couldn't find anything for the loader.
Well, actually the real pain point is I could not finish the execution of the lister yet as it takes days.

Build is green

• maven-loader: fixes after ardumont's review in D6396