1y ago

Heads up for a data corruption bug in ZFS, few versions affected, might have started at 2.1.x, but many reports on 2.2.x

github.com some copied files are corrupted (chunks replaced by zeros) · Issue #15526 · openzfs/zfs

System information Type Version/Name Distribution Name Gentoo Distribution Version (rolling) Kernel Version 6.5.11 Architecture amd64 OpenZFS Version 2.2.0 Reference https://bugs.gentoo.org/917224 ...

28 comments

Is this not for OpenZFS in particular? To my knowledge OpenZFS and ZFS are separate.
- The bug was reported on OpenZFS, check the link from OP's post:
  https://github.com/openzfs/zfs/issues/15526

Ah heck I just updated my NAS VM to FreeBSD 14.
Anyone running FreeBSD 14, make sure vfs.zfs.bclone_enabled is set to 0.
- … FreeBSD 14, …
  Not only 14 …

This is why you ALWAYS need INDEPENDENT backups. You can think all day long about detecting bitrot, and how well you're protected against X drive failures but then something comes from the side and messes up your data in a different way than you've foreseen.
- something comes from the side and messes up your data in a different way than you've foreseen.
  This happened to me years ago. Naïvely thinking SnapRAID protected me against the likelihood of a drive failure. I wasn't prepared for two drives failing simultaneously due to a power supply catastrophically failing (smoke, sparks) and frying the drives as it died.
  It was an expensive lesson: I had to send one drive off for data recovery, and after I got it back I used SnapRAID to restore the remaining drive. Independent backups (and multiple parity drives) is the way.
- Wait. Are you trying to say that raid is not a backup?
- The problem here is that those independent backups would also be corrupted. As I understand from the github discussion, the issue might be a bug that causes ZFS to not recognize when a page is dirty and needs to be flushed and is somehow triggered when copying files using a new-ish optimization that has been implemented in Linux and *BSD kernels? If you trigger the bug while copying a file, the original remains kosher but the new file has swaths of bad data. Any backup made after this point would contain both the (good) original and (corrupted) copied file.
  
  The point is you'll still have the originals, which you might in the meantime have removed (for example if one would reorganize a huge collection and started by working on the reflinked copy and in the end removed the original, natural cleanup workflow, not many would think that you'd need to check the results after a reflinked nearly-instant copy, not even foresee that if there's some bitrot it'll come from THAT).
  Sure, in this case snapshots would have worked just as well, but of course there are other cases in which they wouldn't have. Independent backups cover everything, well assuming you have enough history which is another discussion (I was considering to literally keep it forever after removing some old important file by mistake, but it becomes too daunting and too tempting to remove files removed 1,2,3 years ago).
- Also independent way to verify files. I cfv everything before a big move and then after to check

modinfo zfs | grep version
To quickly get the version installed.
- zfs --version also does the trick.
  
  That did not work for me on ubuntu, but did on my debian/proxmox distribution
  proxmox:
  zfs-0.8.3-pve1
  zfs-kmod-0.8.3-pve1
  ubuntu:
  version: 0.6.5.6-0ubuntu26
  srcversion: 0968F94158D646E259D86B5
  vermagic: 4.4.0-142-generic SMP mod_unload modversions retpoline
  looks like im using an ancient version and am ok?

Makes me really glad I almost never bother to upgrade my pool flags!
(I mean seriously, I can't think of the last time I had a use for new flags!)

Can someone tell a dummy like me, if this impacts truenas core?
- It's hard to tell because I get this:
  root@truenas[~]# zpool get version poolname
  NAME PROPERTY VALUE SOURCE
  poolname version - default
- TrueNAS-13.0-U6 (the current core version)
  zfs version reports 2.1.13
  so it should be clean.
- There is a thread going on at https://www.truenas.com/community/threads/silent-corruption-with-openzfs-ongoing-discussion-and-testing.114390/
  Some users are reporting that it does affect Truenas (though it depends on the use case).

Anyone have any ideas for checking for this issue in existing backups?
- Script at #15526 can somewhat check for a hole in the first 4K bytes of the file, but gives false positives, if script produces a syntax error in the last line - replace /bin/sh with /bin/bash or whatever the location of the BASH is.
  Used it on part of my collection, found several zeroed-out files, but I strongly suspect they were full of zeroes before they hit ZFS, at least some files from 2009 were full of zeroes. Script gave multiple false positives (and one true positive on fully-zero file) on .iso files, suspect that they miss boot record.
  
  Thank you. I've been keeping an eye on the thread to see if any consensus emerges regarding any better understanding of how the corruption manifests itself. It appears there is a possibility that a portion could be zeroed out and then new data written over it, giving the impression that all is well, but where the file is obviously still corrupt. It seems the best method is to have a list of checksums from known good files, but that obviously requires previous action that may or may not have occurred (obviously, most people never anticipated this and thus have no such list).
- Not confirmed but promising - https://github.com/openzfs/zfs/issues/15526#issuecomment-1810800004 and https://github.com/openzfs/zfs/issues/15526#issuecomment-1810819382
  
  Thank you. Upon reading further, the state of block cloning seems to be the major variable as to whether any corruption has occurred. However, there appears to remain a non-zero chance that such corruption could occur regardless of block cloning and dates back to 2.1.4/2.1.5 which were released in March/June of 2022.

It's too bad file system code isn't easy to verify. It would be great if there was a file system that was formally verified (https://en.wikipedia.org/wiki/Formal\_verification).

It looks like the source of the bug is identified and fixed.
https://github.com/openzfs/zfs/pull/15579/commits/679738cc408d575289af2e31cdb1db9e311f0adf
[2.2] dnode_is_dirty: check dnode and its data for dirtiness #15579

28 comments