I didn't realise how much work it would involve at first, ahah. Sure it's better for documentation. Done.

The format is described in the README. Adding a link to the documentation inline.

The link is missing, can you please add it.

Decode errors, see https://forge.softwareheritage.org/D6133#inline-45664

Should I add this link to the py file as well for reference?

I have the feeling we should rather fail (throw an exception about decoding and end execution) than pass silently (adding errors='ignore' to decode could do that).

I agree with you on this. @douardda do you concur?

Please, add a fixme explaining we should raise instead and let's move on.
So we can actually see what happens when deployed.

Ok, thanks.

Added a fixme with a link to this very place.

Yes, the parsing is a bit tough, but we don't have much choice, see my comment about the origin of this text file.

I certainly could change where the yield occurs, but I did it following douardda's comment in https://forge.softwareheritage.org/D6133#inline-45665

I'm ready to change it, though. No personal strong opinion on that.

What if you built a temporary data structure? For example this:

doc 0
  field 0
    name u
    type string
    value al.aldi|sprova4j|0.1.0|sources|jar
  field 1
    name m
    type string
    value 1626111735737

would be turned into something like this:

{
    "doc 0": {
        "field 0": {
             "name": "u",
             "type": "string",
             "value": "al.aldi|sprova4j|0.1.0|sources|jar",
        },
        "field 1": {
             "name": "m",
             "type": "string",
             "value": "1626111735737",
        },
    }
}

This would allow splitting the parser from the rest of the logic

Yes, that'd be on the path of my initial suggestion about separating parsing and reading logics.

Parsing/format conversion is extracted in some easier code to read and maintain
(whether it stays in the lister or moved elsewhere after, we can discuss later, let's start here).

Then the listing focus on the incremental listing (or not) logic without having to parse anything, just
read stuff from that intermediary data.

Note that the lister speed is not that important as long as it yields origins regularly (then it's updating
regularly the listed origins in the scheduler backend).
So if that takes a bit more time because it does some more internal conversion so it's more
maintainable in the long run it's fine.

Note that, for my part, this reading improvment can be taken care of in another diff if that's easier to handle...

So. Here is my humble proposal to split the parsing to make it more readable.

Unfortunately the proposed data structure doesn't exactly work in practice because:

• The parsing required to build the proposed Dict is still quite complex, we still need the regexp's, and it would be a shame not to use the re's groups while we them at hand -- or we would need to re-apply them later on, which is costly when there is 500m lines.
• We need to somehow keep the ordering for incremental recovery.
• Uniqueness of scm entries (these have a lot of duplicates), at least over the current run.

I tried however to keep the same spirit; please allow me to propose a slightly different intermediate version of the data structure:

{
  0: {
    'type': 'maven',
    'url': 'https://repo1.maven.org/maven2/al/aldi/sprova4j/0.1.0/sprova4j-0.1.0-sources.jar',
    'time': 1626109619335,
    'gid': 'al.aldi',
    'aid': 'sprova4j',
    'version': '0.1.0'
  },
  1: {
    'type': 'scm',
    'url': 'https://repo1.maven.org/maven2/al/aldi/sprova4j/0.1.0/sprova4j-0.1.0.pom'
  },
  2: {
    'type': 'maven',
    'url': 'https://repo1.maven.org/maven2/al/aldi/sprova4j/0.1.1/sprova4j-0.1.1-sources.jar',
    'time': 1626111425534,
    'gid': 'al.aldi',
    'aid': 'sprova4j',
    'version': '0.1.1'
  },
  3: {
    'type': 'scm',
    'url': 'https://repo1.maven.org/maven2/al/aldi/sprova4j/0.1.1/sprova4j-0.1.1.pom'
  }
}

The resulting parsing code is substantially simpler. Incremental and uniqueness stuff is moved to the second part.

In the second part, since we yield jars and scms as we find them, we do not need any more the 2 counters for the incremental recovery: the last_seen_doc is enough.

I did not have to change the tests, excepted for:

• The http errors, where the ordering had an impact on what is yielded before we get interrupted.
• The presence of the last_seen_pom, which has been removed.

So it should be ok. Please let me know what you think.