Page MenuHomeSoftware Heritage

Small improvements to the mirror tests
ClosedPublic

Authored by lunar on Sep 27 2022, 5:48 PM.

Details

Summary
  • Add a listing of Python dependencies needed for the tests
  • Add a canonical way to environment variables for the tests
  • Use logging to record tests progression
  • Raise timeouts for replaying, shut down and storage access
  • Use a dedicated network for Docker swarm
  • Remove an unused Docker image from the generation process
  • Add troubleshooting tips to the test documentation

Diff Detail

Repository
rCDFP Deployment tools for hosting a mirror
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lunar requested review of this revision.Sep 27 2022, 5:48 PM
ardumont added a subscriber: ardumont.

lgtm

one non-blocking suggestion inline.

tests/test_graph_replayer.py
326 ↗(On Diff #30869)

not on you. Still, would be best to let the log instruction do the formatting (instead of f-strings).

This revision is now accepted and ready to land.Sep 28 2022, 7:00 PM
tests/test_graph_replayer.py
326 ↗(On Diff #30869)

Could you explain your rationale? Is it because of extra computation costs if logging is disabled? I find f-strings more readable than other interpolation format so I favored legibility over minor gains in CPU cycles in this case.

tests/test_graph_replayer.py
326 ↗(On Diff #30869)

It's about the logging level you set yes. But also about uniformisation in the base
code. If we were to set the log level to warn in production, that'd be still computed
indeed.

If you look at any of our other modules, you'll see we do use f-string for some string
computations but not for the log ones. We let logging do its job.

Also it happens also that some objects (SWHID comes to mind but i may be wrong) used in
f-string could trigger some heavy computations too.

tests/test_graph_replayer.py
326 ↗(On Diff #30869)

Although now that i've said that and given that I re-realized it's test code, it may be fine to keep it that way.

To be clear, it's non-blocking otherwise i would have requested changes btw.

lunar marked an inline comment as not done.

Have the log instruction perform the formatting intead of using f-strings