If that there is an error during the process, some files may have been copied. What should be the best option? Mark them as 'present', or just ignore them and reschedule the whole batch after the 'ongoing' maximum delay is elapsed?

Please, ignore that as it is only used for debug purposes. However, that brings up a question.

The table content_archive contains a field status that may be 'missing'. Does that means that everything in the content table will be in the content_data table as 'missing' at the start?
The 'missing' status is something we can compute on the fly (in content but not in content_archive) in order to avoid this storage.
I don't realize the size it takes, though (but I want to). It may be nothing compared to the whole database and not worth the extra-effort it would require to compute the missing on the fly. Is it?

I had to do some db commits because the archiver directly uses the local storage's db. As the test class uses it's own cursor, the archiver would not have been aware of these changes otherwise.
But that don't seems very nice. Is there another way?

why the # noqa?

a list of sha1`s`
this copier has

I think it might work without needing to consume the map.

Can you refresh my memory, why do you need to strip the first 2 characters (is it '\\x')?
I think this might be due to how you test.

As a first approximation, considering nothing is done is reasonable.
(If nothing prevents later to add again some existing content)

self.server.content_add(map(lambda c: c['data'], contents))

?

You may want to avoid writing your doubts in the code and use differential for that (as you also did below).

I like to use, TODO (you'll need to think more and improve the code), FIXME (you saw something ugly but this is not the time to fix it) , HACK (you had to do something horribly ugly but did not see any other way around it) followed by some concise description about what this is all about.

Well, it depends on how you see the asynchroneous part work...

If archive-director spawns asynchroneous archive-workers which also spawns asynchroneous archive-copiers, it could be simpler to do what you propose (database access here)...
But then the abstraction around the content copy state is broken, meaning we have now multiple classes which are in charge of it (first the worker initiates something that the copier must update and close).
Or we could have a more complex design where the copier also states that woohoo i'm done (by queueing another message)... But then it's complicated...

But if the async activity stops at archive-workers meaning only archive-directory spawns async archive-workers.
Then what is now is ok (and simpler).

which one is needed as backup?
process`es`

c`o`py`

content`s`

content`s`

content`s`
w`h`ich
cont`a`ins

storage slaves

archiva`l`

Please, ignore that as it is only used for debug purposes.

Ok.
Either, you could add a comment NOT FOR PRODUCTION in the function's docstring or get this code in a bin/ folder at the repository's root (the plus side is that it will be easier to reach from cli).

Does that means that everything in the content table will be in the content_data table as 'missing' at the start?

Possibly.

I see 2 problematics:

• one is to adapt the existing code to deal with new content as soon as this module is ready. That is, when content_add is triggered, it also adds newly added contents in archival table with missing status.
• the other one being how to make the existing contents be archived (it's another task).

But that is for later.

I don't realize the size it takes, though (but I want to).

To have an approximate size (because it's dayly updated) check https://archive.softwareheritage.org/api/1/stat/counters/.
Right now, we are at 2349689600 contents ^^.

It may be nothing compared to the whole database and not worth the extra-effort it would require to compute the missing on the fly. Is it?

I think that bootstraping this and dealing with prior data are 2 separate tasks.

Indeed but we have abstracted it.
Extend swh.scheduler.task.Task instead... (and you'd need to add python3-swh.scheduler as deps in debian/control too).

should probably extend``.
content`s`

ha`s`
dictio`n`ary

content's location

archiva`l`

amou`n`t
contain``

If it's an update, call it update ^^

change its status

content id

change`s`

We hide the sql queries in db usually, it's clearer to continue that way.

Using self.conn?

Ok, discussing with Quentin, it's due to the pre-commit-hook we have installed which refuses to let the user commits if there are some pep8 violations. And, as the module is imported but in appearance not used, for flake8, it's a fail.

We need to find a way to improve on this (or not).

I know it's possible to have some setup file for flake8 to ignore some errors (this one is F401).
But ignoring this error is too much - http://flake8.readthedocs.io/en/latest/warnings.html#error-codes.
We only want to ignore it in that particular case.

Also, i found this which relates to this http://stackoverflow.com/questions/31079047/python-pep8-class-in-init-imported-but-not-used. Following the proposed solution, this'd give something like:

from .director import ArchiverDirector
from .worker import ArchiverWorker
from .copier import ArchiverCopier

__all__ = ['ArchiverDirector', 'ArchiverWorker', 'ArchiverCopier']

I don't know if this is reasonable or not but it works.
As pros, this has the merit to force us to be explicit about what's public...
As cons, it's somehow redundant and instead of 18 characters, we have 66...

@olasd @zack What do you think?

(If nothing prevents later to add again some existing content)

What i meant was, as long as you can write and overwrite existing contents again, i think it's ok.

Shouldn't create any problem (ATM) as the object storage just write files without checking if they already exists.

Just checked : RemoteObjStorage::content_add is linked to ObjStorage::add_bytes that only take a single content.

Now that I think about it , the specification is kinda blurry about what precisely is asynchronous. But I understood it as Director spawning async workers that execute copier's code.

Thats not clean, and will be in its own commit (its own diff maybe?).

Done in D25.