have you made you mind on the storage model you want for this? Can we purchase a few of them any time soon?

Let me add this to the list:

https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/

Some raw benchmarks as well: https://www.ixsystems.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/

Interesting feedback on this post: https://www.ixsystems.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/page-10#post-524412

This person is using their ZFS as a small file storage with mostly static files, which is quite similar to what we'll be doing.

• their SLOG is an Intel Optane drive (P4801x)
• they use a metadata-only L2ARC cache on a consumer-grade Samsung 960 evo SSD disk

Both these drives use the m.2 form factor, for which we don't have any ports on uffizi currently.

I think we should go ahead with:

• one HHHL (PCIe card) Intel Optane P4800x (The smallest, 375GB version will be sufficient); should be around 1200 EUR;
• one PCIe card with M.2 slots (Asus Hyper M.2 x16 or equivalent); I've found prices between 100 and 400 EUR;
• two consumer-grade NVMe disks for L2ARC cache (1 or 2TB each; e.g. Samsung 970 Evo); the 1TB drives retail for 200 EUR each.

I'll try to fish for a quote with known suppliers.

In T1958#36628, @olasd wrote:

• one HHHL (PCIe card) Intel Optane P4800x (The smallest, 375GB version will be sufficient); should be around 1200 EUR;
• one PCIe card with M.2 slots (Asus Hyper M.2 x16 or equivalent); I've found prices between 100 and 400 EUR;
• two consumer-grade NVMe disks for L2ARC cache (1 or 2TB each; e.g. Samsung 970 Evo); the 1TB drives retail for 200 EUR each.

I'll try to fish for a quote with known suppliers.

The last 2 items (even together) can be directly bought asking reimbursement later—I can do it, if you need to speed things up.
The first item unfortunately is outside the maximum cost limit for that (800€).

In T1958#36629, @zack wrote:

In T1958#36628, @olasd wrote:

• one HHHL (PCIe card) Intel Optane P4800x (The smallest, 375GB version will be sufficient); should be around 1200 EUR;
• one PCIe card with M.2 slots (Asus Hyper M.2 x16 or equivalent); I've found prices between 100 and 400 EUR;
• two consumer-grade NVMe disks for L2ARC cache (1 or 2TB each; e.g. Samsung 970 Evo); the 1TB drives retail for 200 EUR each.

I'll try to fish for a quote with known suppliers.

The last 2 items (even together) can be directly bought asking reimbursement later—I can do it, if you need to speed things up.
The first item unfortunately is outside the maximum cost limit for that (800€).

I've obtained a quote for the Optane card, which has been approved by Roberto and I've sent to Vanessa for purchasing. I've also forwarded the proper references for the m.2 disks and adapter card (totalling around 700€) to Roberto, which we'll handle together on Monday.

The aforementioned cards have been received, installed and zfs has been setup to use them.

In the meantime, most of the data in /srv/storage/space has been moved to zfs.

The primary objstorage on uffizi has been rebased onto a zfs dataset, and the workers have been restarted; The legacy xfs objstorage has been remounted read-only.

I've added a "data rates" panel to the storage backends grafana dashboard : https://grafana.softwareheritage.org/d/8ywqc76mk/storage-backend-statistics, which will allow us to compare before/after performance.

In T1958#38009, @olasd wrote:

I think the bulk of the task here is done. We can open another task if we notice any specific pain point later.

Sure.

What's the bottom line in terms of performance comparison between the new ZFS based deployment and the previous one on old disk arrays?