Change Details

As per title, currently importing a list of ~30 M SWHIDs to an sqlite database using `swh scanner db import` takes about 30 minutes (tested on granet). It can very likely be much faster. We should try the following classic approaches to make bulk import into SQLite faster: - use [[ https://sqlite.org/pragma.html#pragma_synchronous | pragma synchronous = off ]] (yes, it is unsafe, but the conversion to sqlite is all of nothing anyway, so it's not a factor here) - use [[ https://www.sqlite.org/pragma.html#pragma_journal_mode | pragma journal_mode = memory ]] (ditto) - wrap all imports in a single transaction (with sync=off ?) - verify we are using prepared statements for inserts

As per title, currently importing a list of ~30 M SWHIDs to an sqlite database using `swh scanner db import` takes about 30 minutes (tested on granet). It can very likely be much faster. We should try the following classic approaches to make bulk import into SQLite faster: - use [[ https://sqlite.org/pragma.html#pragma_synchronous | pragma synchronous = off ]] (yes, it is unsafe, but the conversion to sqlite is all of nothing anyway, so it's not a factor here) - use [[ https://www.sqlite.org/pragma.html#pragma_journal_mode | pragma journal_mode = off ]] (ditto) - wrap all imports in a single transaction - verify we are using prepared statements for inserts

As per title, currently importing a list of ~30 M SWHIDs to an sqlite database using `swh scanner db import` takes about 30 minutes (tested on granet). It can very likely be much faster. We should try the following classic approaches to make bulk import into SQLite faster: - use [[ https://sqlite.org/pragma.html#pragma_synchronous | pragma synchronous = off ]] (yes, it is unsafe, but the conversion to sqlite is all of nothing anyway, so it's not a factor here) - use [[ https://www.sqlite.org/pragma.html#pragma_journal_mode | pragma journal_mode = memoryoff ]] (ditto) - wrap all imports in a single transaction (with sync=off ?) - verify we are using prepared statements for inserts