Comments on: ZFS gets dedup http://pl.atyp.us/wordpress/?p=2450 Making the world better, one byte at a time. Sun, 05 Sep 2010 02:56:27 -0400 http://wordpress.org/?v=2.9 hourly 1 By: Jeff Darcy http://pl.atyp.us/wordpress/?p=2450&cpage=1#comment-142209 Jeff Darcy Tue, 03 Nov 2009 01:47:03 +0000 http://pl.atyp.us/wordpress/?p=2450#comment-142209 I suspect that Jeff B's answer would be something about boiling the oceans, and with 256-bit hashes he might even have a point. At some point the probability of a correlated hardware failure exceeds that of a hash collision, and the probability of a software bug dwarfs either. Anybody who's *that* worried about a 1/2^128 chance shouldn't be using de-dup at all, even with "verify" set. It is interesting, though, that Jeff B posts no actual performance <b>numbers</b>, most notably those with "verify" turned on. So far they seem to be sticking to their guns on the safety of running without verification, but even if they're proven wrong I'd expect a round of benchmarks with it off because that makes them look better. I suspect that Jeff B’s answer would be something about boiling the oceans, and with 256-bit hashes he might even have a point. At some point the probability of a correlated hardware failure exceeds that of a hash collision, and the probability of a software bug dwarfs either. Anybody who’s *that* worried about a 1/2^128 chance shouldn’t be using de-dup at all, even with “verify” set.

It is interesting, though, that Jeff B posts no actual performance numbers, most notably those with “verify” turned on. So far they seem to be sticking to their guns on the safety of running without verification, but even if they’re proven wrong I’d expect a round of benchmarks with it off because that makes them look better.

]]> By: Jeff Garzik http://pl.atyp.us/wordpress/?p=2450&cpage=1#comment-142198 Jeff Garzik Mon, 02 Nov 2009 23:31:56 +0000 http://pl.atyp.us/wordpress/?p=2450#comment-142198 Yuck, hash collisions rear their ugly head. Being realistic, I think few people will run with the 'verify' option enabled. Which is sad. When considering truly -massive- amounts of storage, in use for years, the probability of a collision (like the probability of life on another planet) does increase. And the cost of being wrong is -very- high: silent data corruption. Yuck, hash collisions rear their ugly head.

Being realistic, I think few people will run with the ‘verify’ option enabled. Which is sad.

When considering truly -massive- amounts of storage, in use for years, the probability of a collision (like the probability of life on another planet) does increase. And the cost of being wrong is -very- high: silent data corruption.

]]>