The main question that needs solving is making sure the prometheus instance
doesn't get bogged down by long term storage of metrics. I've been meaning to
look into thanos[12] which makes some people whose judgement I trust happy,
there's only so many hours in each day.

[12] https://github.com/improbable-eng/thanos