Fix timezone issue, rebase.

Rebase diff.

• maven-lister: fix datetime for time

• maven-lister: set time to be iso8601 date str

Make repo.time an iso8601 string

Try to restart the build

Amend commit, last fixes after review.

In D6395#174021, @swh-public-ci wrote:

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/417/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/417/console

Updating D6395: rebase, update commit.

In D6395#174015, @ardumont wrote:

Not totally. I meant, can you please change the commit message (first line) to

In D6395#174005, @ardumont wrote:

Can you please squash them into one?
Also the commit message should be amended.

Done.

Add Maven lister as per T1724 and D6395

• maven-loader: various fixes for review.

• maven-lister: Add maven lister with review D6133.
• maven-lister: Add statefulness
• maven-lister: Fix README, 88 chars, minor typos
• maven-lister: Split parsing and processing
• maven-lister: Fix groupId bug, add tests.
• maven-loader: fix last_update

In D6395#173428, @ardumont wrote:

Heads up, I was correct and that'd be D6666 for the fix.

Hi there,

Hum. I don't understand why the tests fail on the sourceforge lister when it's not listed in my diff. Any idea please?

• maven-lister: Add maven lister with review D6133.
• maven-lister: Add statefulness
• maven-lister: Fix README, 88 chars, minor typos
• maven-lister: Split parsing and processing
• maven-lister: Fix groupId bug, add tests.
• maven-loader: fix last_update

• maven-loader: fix tests for snapshot-append

• maven-loader: fix last_update

• maven-loader: various fixes for vlorentz' review
• maven-loader: remove snapshot_append carryover
• maven-loader: final udpate after vlorentz review

In D6396#173266, @vlorentz wrote:

We also tweaked the values of various fields to be consistent across loaders (especially the message), see: https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

Hi @vlorentz, I've fixed the code for the new release object and for the loader fields. I also updated the loaders spec documentation. Good for me.

• maven-loader: fixes after ardumont's review in D6396
• maven-loader: fix release instead of revision

• maven jar-loader: Initalise files.
• maven-loader: fixes after ardumont's review in D6396
• maven-loader: Fix copyright in header
• maven-loader: Fix extId to include url and timestamp
• maven-loader: fix release instead of revision

In D6396#173266, @vlorentz wrote:

In D6396#173260, @borisbaldassari wrote:

Arg, changes on the master now break the build, throwing "NotImplementedError: build_release".
What happened to build_release, any idea?

I replaced "build_revision" with "build_release" in the core loader. You need to rename your "build_revision" function, and change the object it returns.

We also tweaked the values of various fields to be consistent across loaders (especially the message), see: https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

Arg, changes on the master now break the build, throwing "NotImplementedError: build_release".
What happened to build_release, any idea?

In D6395#173250, @ardumont wrote:

I'd say if we are confident enough it does the right thing and it's well covered (I
think it is right now) you should be able to rebase all your diffs on latest master. And
then "land" [1] them!

Fine. I'll wait for your review and land the diff when you tell me to then.

• maven-loader: Fix extId to include url and timestamp

In D6395#171365, @borisbaldassari wrote:

I will let you know when the full chain is executed.

In D6395#171127, @borisbaldassari wrote:

Thanks for the hint. I'll first refactor the parsing loop and then run the whole process as suggested.
Will keep you posted asap.

• maven-lister: Fix groupId bug, add tests.

In D6395#171086, @ardumont wrote:

[1] scheduler: D6520
[2] docker container: D6543 (to deploy ^ in the docker world)

So those landed. So now you should rebuild the containers [1] to have the most recent
scheduler version (after the listing is done, the scheduling of the maven origins should
happen).

First, make sure to reference in the the loader configuration [2] the new maven task
[3], override your service containers (lister and loader) with the right override and
you should be good to go [4].

Now that we've made some progress (and changes) on the lister I'll re-try that, with the proper loader conf. Thanks for the hint.
I'll keep you posted (probably next week though).

• maven-lister: Split parsing and processing

In D6395#171086, @ardumont wrote:

[1] scheduler: D6520
[2] docker container: D6543 (to deploy ^ in the docker world)

So those landed. So now you should rebuild the containers [1] to have the most recent
scheduler version (after the listing is done, the scheduling of the maven origins should
happen).

Thanks for the hint. I'll first refactor the parsing loop and then run the whole process as suggested.
Will keep you posted asap.

• maven-lister: Fix copyright in header

• maven-lister: Fix README, 88 chars, minor typos

In D6395#170281, @vlorentz wrote:

You use three similar but subtly different conditionals:
Could you somehow unify them?

No, I don't think so, or not without risking losing entries.

In D6395#170099, @ardumont wrote:

Thanks for the great work.
This looks mostly ok to me.

Thanks, it's appreciated. :-)

In D6395#170095, @ardumont wrote:

I can't manage to test the full toolchain (maven lister - maven loader),

why? What's the blocking point?

I've found a great doc to run the lister in a docker-dev environment, and couldn't find anything for the loader.
Well, actually the real pain point is I could not finish the execution of the lister yet as it takes days.

• maven-loader: fixes after ardumont's review in D6396

Superseded after massive refactoring / renaming by D6396.

Superseded by https://forge.softwareheritage.org/D6395

In D6395#169902, @ardumont wrote:

Let's keep this as is now.
For next time though, please keep the old diff id around even if you think the review is
harder ;)

Sure. Sorry for the mess. :-)
One note through: the old diff at [1] has been constantly updated and is still relevant for history.

• maven-lister: Add statefulness

• maven-lister: Add maven lister with review D6133.
• maven-lister: Add statefulness

• maven-lister: initialise lister.
• maven-lister: Refactor file get loop (review D6133).
• maven-lister: update readme, set tests to use http_index
• maven-lister: Add statefulness
• maven-lister: Add incremental task and tests

In D6395#168737, @ardumont wrote:

Thanks for your time, @ardumont.

• maven-lister: Add statefulness

Thanks vlorentz. Done.

Hum. This diff is a mess after the refactoring of the main loop.
Please give me some time to fix all of the current issues, then rebase the diff to make it more readable. Hope it will clear up the lines mix.

• maven-lister: Fix various comments from review (review D6133).
• maven-lister: add README.md to describe design decisions (review D6133).
• maven-lister: Fix useless storage of file + minor fixes (review D6133).

In D6133#163254, @douardda wrote:

I'm not done yet but here is first review on my side.

Thanks for your time and interest.

• maven-lister: Fix tests (review D6133)

• maven jar-loader: cleanup unnecessary comments.

• maven-lister: Fix inaccurate comment (review D6133).

• maven-lister: add changes from review D6133.

• maven jar-loader: add extrinsic metadata.

So, what would be useful as metadata? What we have is groupId, artifactId, version.

Hiho good people,

• maven jar-loader: fix tests.

• maven jar-loader: fix tests.

• maven jar-loader: fix tests.

• maven jar-loader: fix tests.

• maven jar-loader: fix tests.

• maven jar-loader: fix minor typos + tests.

• maven jar-loader: fix minor typos + tests.

Hey good people, this failing test [1] and the snapshot_id mentioned in the code actually work well on my local setup. How can it be that snapshot_id's are different on different hosts?
More specifically: snapshot_id on jenkins is 'ae8f76fe879ea040312182c02dae324a11252f83' and on my local setup is '677b0280cbe9e650c9dfa80fbb1ab7ebb1603c86'.

• maven jar-loader: fix + add tests.

• maven jar-loader: un-change loader.py (cleanup debug statements).

• maven jar-loader: fix tests.
• maven jar-loader: fix tests.

• maven jar-loader: messing around with tests.

• maven jar-loader: fix timezone and tests, improve tests.

• maven jar-loader: fix, improve tests.

• maven jar-loader: fix tests.

• maven jar-loader: add test and restart build.

Restart loader build after successful core build.

• maven-lister: set last modif time to epoch int.