Change Details

This diff consists in a stack of revisions that refactor the pytest_plugin so that: - the kafka_server fixture ensured topics exists on the broker beforehand, - the kafka_prefix is used as is everywhere to prevent some naughty code coupling (user of the plugin *must* also modify the kafka_prefix the same way it was done here). Included revisions are: - pytest_plugin: improve error message in consume_messages() helper function - pytest_plugin: do not crash if the list of consumed messages is empty if assert_all_objects_consumed() helper function. - pytest_plugin: modify the kafka_server fixture so topics are created beforehand Since we now check that topics exist on the kafka broker when instanciating a JournalClient, topics but be created when the mock kafka broker is started for tests to keep working (e.g. in swh.storage). This needs a small adaptation in the JournalClient code itself to ensure "empty" messages used to create topics will be silently ignored. The kafka_server() fixture is split in 2 fixtures: - kafka_server_base() that only creates the mock kafka server, - kafka_server that() uses this later and creates the topics resulting from the conjunction of the kafka_prefix() and the new object_types() fixture. This new object_types() fixture is thus used to define the list of topics (in conjunction with the kafka_prefix fixture) a (mock) kafka_server() will create at startup time. - test: replace usage of a strategy .example() by a hardcoded value to prevent a warning from hypothesis. - pytest_plugin: remove '.swh.journal.objects' from test_config fixture This induces a tigh coupling with code from other packages that use this fixture (e.g. swh.storage currently have to modify the kafka_prefix resulting from the fixture so the consumer and the writer are in sync). If someone wants a customized version of a kafka_prefix, it must be done by overriding the kafka_prefix fixture, not by modifying its returned value. Note that this commit will break swh.storage.tests.test_kafka_writer until it is updated. - pytest_plugin: use the object_types fixture in test_config instead of hardcoding a specific list there that may become out of sync with the default list of topics used elsewhere in the pytest_plugin. Depends on D3126.

This diff consists in a stack of revisions that refactor the pytest_plugin so that: - the kafka_server fixture ensures topics exist on the broker beforehand, - the kafka_prefix is used as is everywhere to prevent some naughty code coupling (user of the plugin *must* also modify the kafka_prefix the same way it was done here). Included revisions are: - pytest_plugin: improve error message in consume_messages() helper function - pytest_plugin: do not crash if the list of consumed messages is empty if assert_all_objects_consumed() helper function. - pytest_plugin: modify the kafka_server fixture so topics are created beforehand Since we now check that topics exist on the kafka broker when instanciating a JournalClient, topics but be created when the mock kafka broker is started for tests to keep working (e.g. in swh.storage). This needs a small adaptation in the JournalClient code itself to ensure "empty" messages used to create topics will be silently ignored. The kafka_server() fixture is split in 2 fixtures: - kafka_server_base() that only creates the mock kafka server, - kafka_server that() uses this later and creates the topics resulting from the conjunction of the kafka_prefix() and the new object_types() fixture. This new object_types() fixture is thus used to define the list of topics (in conjunction with the kafka_prefix fixture) a (mock) kafka_server() will create at startup time. - test: replace usage of a strategy .example() by a hardcoded value to prevent a warning from hypothesis. - pytest_plugin: remove '.swh.journal.objects' from test_config fixture This induces a tigh coupling with code from other packages that use this fixture (e.g. swh.storage currently have to modify the kafka_prefix resulting from the fixture so the consumer and the writer are in sync). If someone wants a customized version of a kafka_prefix, it must be done by overriding the kafka_prefix fixture, not by modifying its returned value. Note that this commit will break swh.storage.tests.test_kafka_writer until it is updated. - pytest_plugin: use the object_types fixture in test_config instead of hardcoding a specific list there that may become out of sync with the default list of topics used elsewhere in the pytest_plugin. Depends on D3126.

This diff consists in a stack of revisions that refactor the pytest_plugin so that: - the kafka_server fixture ensureds topics exists on the broker beforehand, - the kafka_prefix is used as is everywhere to prevent some naughty code coupling (user of the plugin *must* also modify the kafka_prefix the same way it was done here). Included revisions are: - pytest_plugin: improve error message in consume_messages() helper function - pytest_plugin: do not crash if the list of consumed messages is empty if assert_all_objects_consumed() helper function. - pytest_plugin: modify the kafka_server fixture so topics are created beforehand Since we now check that topics exist on the kafka broker when instanciating a JournalClient, topics but be created when the mock kafka broker is started for tests to keep working (e.g. in swh.storage). This needs a small adaptation in the JournalClient code itself to ensure "empty" messages used to create topics will be silently ignored. The kafka_server() fixture is split in 2 fixtures: - kafka_server_base() that only creates the mock kafka server, - kafka_server that() uses this later and creates the topics resulting from the conjunction of the kafka_prefix() and the new object_types() fixture. This new object_types() fixture is thus used to define the list of topics (in conjunction with the kafka_prefix fixture) a (mock) kafka_server() will create at startup time. - test: replace usage of a strategy .example() by a hardcoded value to prevent a warning from hypothesis. - pytest_plugin: remove '.swh.journal.objects' from test_config fixture This induces a tigh coupling with code from other packages that use this fixture (e.g. swh.storage currently have to modify the kafka_prefix resulting from the fixture so the consumer and the writer are in sync). If someone wants a customized version of a kafka_prefix, it must be done by overriding the kafka_prefix fixture, not by modifying its returned value. Note that this commit will break swh.storage.tests.test_kafka_writer until it is updated. - pytest_plugin: use the object_types fixture in test_config instead of hardcoding a specific list there that may become out of sync with the default list of topics used elsewhere in the pytest_plugin. Depends on D3126.