Sync=disabled will indeed improve write performance, but at the cost of possibly losing data if your server crashes or loses power after a write.
I run my personal servers with sync=disabled because that's an acceptable risk to me. However, it won't be to everyone.
One case where sync=disabled is perfectly fine is when initially copying a large amount of data to the ZFS filesystem. You'll know for sure if the server crashed or shutdown during the copy, so you can leave sync disabled until the copy is complete.
One case where it's not okay is a large file server serving multiple users. If the server crashes right after a user thought he saved his file successfully, then the user finds out it didn't save, you will have an angry user on your hands.
- Sync writes go to in-pool ZIL (slow), async writes go to RAM
- See 1
sync=disabledwill mean that your sync writes are no longer synchronously written to disk; this opens you up to the possibility of data loss on these filesystems if there is a crash.logbias=throughputmeans the writes would have lower priority in the ZIL.
My sytstem has 2 pools:
1 is a data (non-bootable) 10-disk raidz2 pool that has a SLOG already (2 mirrored partitions from a pair of m.2 intel optane drives)
the other pool is a single nvme drive (970 pro) that holds the system's root filesystem. This is mainly using zfs (vs ext4/xfrs/btrfs) to allow sending/receiving snaphots from the raidz2 pool, so i can have a whole bunch of them that can all be used to restore the system without taking up space on the nvme ssd.
This is ZoL (OS is Fedora 29). so not a whole lot of sync writes by default.
Currently I have sync=standard on both, logbias=throughput on the nvme root pool, and the raiudz2 pool uses logbias=latency
Im fairly sure the raidz2 pool it set right, but with the root drive i really am not sure, and dont eve know if zfs uses the same behavior as with a pool, since any "fake-slog" writesto the drive would seeingly just be writing data to the drive normally, whereas with a pool i believe these will get directly written (not as a raidz striped write) which avoids the extra I/O that using parity adds.
I also could add another partition to the intel drives and use those as a mirrored (or striped) SLOG. nvme drive has higher max throughput, but not necessairy at low queue depths. THe optane drives also have a few other nice properties from a "preventing data loss" viewpoint (can be mirrored, use direct writing not a volatile flash cache, etc)
ChrisOfAllTrades
If sync writes are that light against the RAIDZ2 pool, I'd wonder if you really need the SLOG at all. If it's media storage and stuff where latency really doesn't matter, you could just remove it and see if there's a major impact.
As far as the root pool, the same comment I made above about the "double writes" when using in-pool ZIL apply, but if your sync writes are minimal (just running a regular OS on top of it) then I wouldn't be terribly concerned. It's more of an issue if you're using it to host VM images where lack of sync writes can cause corruption.
Oh i know it doesnt have a very big impact. My reasons for implementing it were more-or-less:
- it will help a little with synced writes. Even if there arent many, there are a non-zero number of them occurring.
- I saw some reports tat not having a SLOG could lead to pol fragmentation over time (or at least speed up how quick fragmentation occurs by a little bit)
- the 2x 32gb optanes set me back something like $110 bucks. Of those 64 gb, 8 gb (4 from each drive)is dedicated to the slog (mirrored, so 4 gb usable), and the other 28 gb on each driveis use is used for dual swap devices (which I neded/wanted anyways). So were talking about a $15 or so investment.
The 10x 8gb toshiba N300 drives and 1tb 970 pro set me back nearly 3 grand. So, basically if it improves things half a percent im getting my money's worth out of it.
that said, i still wasnt to set things up optimally, even if it isnt a huge difference in real-world performance.
the same comment I made above about the "double writes" when using in-pool ZIL apply,
Even with a 1-disk pool? i mean i get why youd have this in a raidz setup - you can setup the on-pool ZIL as striped (for speed) and/or mirrored (for redundancy) and avoid the I/O penalty associated with parity.
... But on a single disk, it seems like (unless im misunderstanding something) there would be literally 0 benefit by doing this over just prioritizing the sync writes, since writing to a ondisk zil wont register the writes as completed any faster than "just writing them first" but will still require re-writing them.
side question: say i partitioned off another few gb from the optane drives and made a slog for the nvme drive. Assuming a relatively small number of sync write (i,e., non-database workload), would this ever hurt the overall speed/throughput/data integrity of the root pool? (assume that there are few enough sync writes that the optane drives never hit 100% I/O capacity within a given 5 second write window).
As long as this wont hurt and might help a little i mine as well, for the same reasons i implemented it on the raidz2 pool.
Lack of SLOG + sync writes will indeed increase fragmentation due to the double writes.
Good use of the spare space on the Optane drives as swap (which you hope to tune your system never to use anyways, I assume)
Even with a 1-disk pool? i mean i get why youd have this in a raidz setup - you can setup the on-pool ZIL as striped (for speed) and/or mirrored (for redundancy) and avoid the I/O penalty associated with parity.
In-pool ZIL can't be configured, it's just "there" - you can configure your SLOG vdev(s) however you like of course, but mirrors are recommended. And yes, even on a single-disk stripe, you have an in-pool ZIL.
... But on a single disk, it seems like (unless im misunderstanding something) there would be literally 0 benefit by doing this over just prioritizing the sync writes, since writing to a ondisk zil wont register the writes as completed any faster than "just writing them first" but will still require re-writing them.
Blame the atomic nature of ZFS for this one. It still needs to order things into transaction groups (txg's) and then commit those as an unit that either succeeds or fails as a whole.
side question: say i partitioned off another few gb from the optane drives and made a slog for the nvme drive. Assuming a relatively small number of sync write (i,e., non-database workload), would this ever hurt the overall speed/throughput/data integrity of the root pool? (assume that there are few enough sync writes that the optane drives never hit 100% I/O capacity within a given 5 second write window).
Should work. Optane by design is direct-to-NAND write so you won't have to worry about competing cache flushes, and as long as you don't exceed the total bandwidth available to the underlying device it shouldn't choke the main 10-drive pool.
If we had better access to NVMe devices that supported multiple namespaces and QoS, you could actually enforce that at the hardware level. But I haven't found anything readily consumer-available with that.
##
Student
Good use of the spare space on the Optane drives as swap (which you hope to tune your system never to use anyways, I assume)
in my case its not really a matter of tuning. The system is intended for (among other things) data processing work. It has an i9-7940x that ie already maxced out with 128 gigs of fairly fast (3264 mhz 14-14-14-34) ddr4, and, well, thats just not enough.
Unfortunately, to get more from intel you have to go xeon which is a major cost hike, and going amd id lose significant performance since i can utalize avx512. plus ddr4 is stupid expensive right now.
Im hoping in the not too distant future optane + some software optimization will allow you to basically use optane drives like you would ram and use ram as effectively a L4 CPU cache. Itll never be as fdast as ram, but it doesnt need to be, it just needs to be fast enough to make sure the ram has the stuff the L3 cache is going to ask for in memory before it asks for it. And it seems like this use case is something intel is actively working on (and which would significantly increase the number of potential buyers who can actually benfit from optane), so i think it has a decent shot of actually happening
Anyways, im getting off topic.
(side note: im also rather fond of zfs since its basically the only thing i use other than MKL that utalizes avx512)
Blame the atomic nature of ZFS for this one. It still needs to order things into transaction groups (txg's) and then commit those as an unit that either succeeds or fails as a whole.
ahh, ok that sort of makes sense. Plus i resize that zfs isnt really designed with single drive pools in mind (if it were i imagine they wouldnt have labeled them as "drive pools" lol).
Should work. Optane by design is direct-to-NAND write so you won't have to worry about competing cache flushes,
this is one of the reasons I specifically got optane (vs a standard nvme) to do double duty as slog and swap. between this, low queue depth performance, and high durability, its almost like optane was designed specifically to excel as an slog / in other slog-like use cases.
Should work. Optane by design is direct-to-NAND write so you won't have to worry about competing cache flushes,
appreciate the confirmation. I cant see this ever happening unless my system, usage drastically changes. The little optane drives dont have the performance of their big brothers (like the 900p), but its performance is more than enough for the sync writes my system currently does. (performance is surprising good considering the price - id probably argue it is literally the best SLOG you can get on the planet for $20 or less, lol)
##
If you don't have an SLOG device, in-pool ZIL is extra-slow because you're writing first to the ZIL for safety, and then having to write again to the pool itself.
If you don't mind the risk of data loss to in-flight data, go ahead and set
sync=disabled but do backups to other media if you value anything there.In above quote, do you mean logbias=throughput will directly write the sync writes to the pool
As I understand it, sync writes will still hit the ZIL, but writes from datasets with
logbias=latency will be serviced first, in sort of a "pre-emptive" way.Since i dont have fast SSD for ZIL i think the disk-ZIL is use less any way
It's still going to provide safe write capacity for sync writes, it's just going to enact a significant toll on your performance while doing it.
Is there any way i can determine if the writes are sync or async or ratio of sync/async writes ?
I'm not sure if Richard Elling's
zilstat script is provided with the default Ubuntu ZFS installation, but if not, you can get it from here:
I admit I haven't tried too much with ZoL.
No comments:
Post a Comment