Picking a hash function, measured
- host
- epyc host
- cpu
- AMD EPYC 7642, 48 cores, 16 NUMA nodes
- go
- go1.26.5 linux/amd64
- governor
- performance
| Benchmark | Before | After | Change | p | Pairs |
|---|---|---|---|---|---|
| xxh3 over []byte(s), 64 KiB | 9.65 µs | 10.93 µs | +13.3% slower | 0.001 | 12 |
| crc32c over []byte(s), 64 KiB | 10.93 µs | 12.64 µs | +15.7% slower | 0.000 | 12 |
| map[netip.Addr] hit, 1M entries | 86.0 ns | 99.9 ns | +16.1% slower | 0.000 | 12 |
| map[string] hit, 1M entries | 191 ns | 214 ns | +12.2% slower | 0.000 | 12 |
“Which hash should I use” usually gets answered with one name and a throughput number, which is useless if your keys are 16 bytes and the number came from hashing a megabyte. I wanted one answer per key shape and per length, on the box the code deploys to, so I benchmarked sixteen candidates.
Twelve repetitions at -benchtime=100ms, one prebuilt binary, all arms interleaved, benchstat
against a control that nothing under test touches. The control puts the noise floor at 1.5%, so
treat anything smaller as nothing. Every table below is measured output, trimmed to the rows under
discussion.
By length
| length | use | cost | runner-up |
|---|---|---|---|
| 1 to 8 B | wyhash | 2.6-2.9 ns | xxh3-64, 3.7 ns |
| 16 to 32 B | xxh3-64 | 3.3-4.0 ns | wyhash, 3.0-4.3 ns |
| 36 to 256 B | hash/maphash | 5.3-18.4 ns | xxh3-64, 5.7-19.4 ns |
| 512 B to 1 MiB | xxh3-64 | 21.8 ns to 20.8 µs | wyhash, 1.6-3.2x slower |
Below 256 bytes the leaders stay within about 1.5x of each other and the choice barely matters. Above 512 bytes it decides the answer: xxh3 pins to the fastest line and everything else fans out to between 3x and 4.8x.
hash/fnv is the one to strike off. It is byte-at-a-time, 12x slower than xxh3 at 64 bytes and 62x
at 1 MiB, and there is no length at which it is the right answer for anything larger than a word. It
no longer allocates, though. fnv.New64a() used to cost 1 alloc/op, and does not on Go 1.26.5,
where the compiler devirtualises the hash.Hash64 interface and keeps the hasher on the stack. Zero
allocs across all 216 samples.
A native string entry point is worth having
If a library only takes []byte, hashing a string means casting it first, and the cast is not
free.
ns/op, the same candidate through both entry points:
| length | 4 | 8 | 16 | 36 | 64 | 256 | 1024 | 65536 |
|---|---|---|---|---|---|---|---|---|
xxh3.HashString(s) | 3.71 | 3.71 | 3.34 | 5.73 | 5.70 | 19.3 | 33.4 | 1.3µ |
xxh3.Hash([]byte(s)) | 8.03 | 7.74 | 7.94 | 44.3 | 44.1 | 87.8 | 260 | 9.6µ |
The cast allocates from 36 bytes upward; below that the conversion stays on the stack, which is why
the penalty is a cliff rather than a slope. crc32.Checksum has no string form at all and its
argument escapes at every length, so it allocates even for a 4-byte string.
Otherwise the two paths are the same function underneath: a harness test asserts that all sixteen candidates return identical values through both entry points at ten lengths. If you hash strings on a hot path, pick a library with a native string entry point, or pay for an allocation the hash never needed.
By type
Most hashing in a real service is not “hash this arbitrary buffer”. It is “hash this uint64”, or
this IPv6 address, or this five-tuple, and a length sweep cannot answer that.
| key | use | cost | do not use |
|---|---|---|---|
uint32, uint64 | FMix64 or SplitMix64 | 1.2-1.6 ns | maphash.Comparable, 4.9 ns |
[4]byte, [8]byte | xxh3.Hash(k[:]) | 3.7-4.0 ns | |
[16], [32], [48], [64], [128]byte | maphash.Bytes(seed, k[:]) | 4.6-8.3 ns | maphash.Comparable, 1.6-2.5x slower |
[24], [36], [40]byte | maphash.Bytes(seed, k[:]) | 5.3 ns | maphash.Comparable, 4x slower |
| pointer-free struct | maphash.Bytes over an unsafe.Slice view | 5.3 ns | maphash.Comparable, 21.3 ns |
struct holding a netip.Addr | maphash.Comparable | 21.1 ns | its raw bytes: it holds a pointer |
| anything used as a map key | the map | 11.7-21.6 ns including the lookup | hashing it yourself first |
For an integer, stop reaching for a hash function: FMix64, MurmurHash3’s finaliser, is 3.7x
cheaper than maphash.Comparable and sound on every corpus I threw at it.
maphash.Comparable on a byte array has a cliff I did not expect. Zero allocations in every cell:
| array length | 4 | 8 | 16 | 24 | 32 | 36 | 40 | 48 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|---|---|
maphash.Comparable | 4.84 | 5.16 | 7.34 | 20.3 | 7.34 | 21.5 | 21.4 | 8.55 | 8.67 | 13.2 |
maphash.Bytes(k[:]) | 4.34 | 4.34 | 4.63 | 4.35 | 4.32 | 5.26 | 5.27 | 5.26 | 5.24 | 8.33 |
Lengths of 4, 8 or a multiple of 16 stay on a fast path. 24, 36 and 40 fall off it and pay about
13 ns for the trailing partial block. A JA4 fingerprint is 36 bytes, which is how I found it. I ran
the extra lengths as their own interleaved experiment with its own control rather than trusting one
data point, and did not chase it into the runtime, because the rule holds either way: maphash.Bytes(k[:]) is never slower on a byte array and is sometimes 4x faster. Keep Comparable for types that contain pointers, where it is the only correct option.
If it goes in a map, put it in the map
Cost of the whole lookup, hash included, and at 8 cores unless the column says otherwise, so these do not compare to the columns above:
| key type | 1K hit | 1M hit | 1M hit, 32 cores |
|---|---|---|---|
uint64 | 11.7 | 59.8 | 63.7 |
[16]byte | 15.8 | 71.1 | 74.7 |
netip.Addr v6 | 18.9 | 86.0 | 99.9 |
| 5-tuple struct | 21.6 | 120 | 123 |
string, 16 B | 19.0 | 191 | 214 |
At 1,000 entries a whole lookup costs 12 to 22 ns, barely more than hashing the key alone, so a
hand-rolled hash has no room to win. At a million entries the cache miss swallows everything: a string hit costs 191 ns, of which roughly 6 is hashing. That is 3% of the cost. Make the key
smaller or the table denser instead.
The last column is the one to watch. Everything that allocates got slower as core count rose, and
nothing else moved, because more GC workers over a pointer-dense heap costs more than the extra
parallelism returns. It is an argument against []byte(s) casts on a hot path, and a warning about
benchmarking allocating code on a machine whose core count does not match production.
Quality only eliminates losers
Speed alone would have recommended something terrible. The fastest thing measured was x * K,
Knuth multiplicative hashing, at 0.62 ns. On 1,048,576 keys whose entropy lives in the high bytes it
puts every one of them in a single bucket, because the low bits of a product depend only on the low
bits of the input.
Strict avalanche, largest deviation from p=0.5 over all 128x64 input-output bit pairs, noise floor about 0.027:
| hash | maxBias | meanBias |
|---|---|---|
| maphash | 0.0273 | 0.0063 |
| xxh3-64 | 0.0305 | 0.0062 |
| wyhash | 0.0347 | 0.0063 |
| mixer:fmix64 | 0.0315 | 0.0061 |
| mixer:splitmix64 | 0.0312 | 0.0061 |
| fnv1a | 0.5000 | 0.0963 |
| mixer:wymix | 0.4973 | 0.2519 |
| mixer:fibmix | 0.5000 | 0.3793 |
| crc32c-sse42 | 0.5000 | 0.5000 |
| crc64-ecma | 0.5000 | 0.5000 |
A meanBias of exactly 0.5000 means every input bit flips a fixed set of output bits, every time.
Correct for an error-detecting code, disqualifying for a hash table. CRC32-C also has no high 32
bits at all, which stays invisible until you store it in a uint64 and mask from the top. WyMix alone fails differently: no collisions, but a maximum bucket depth of 18 against an expected 36, the
signature of a near-linear map. It is fine as wyhash’s inner step and not fine as a finaliser.
siphash-2-4, sha256, blake3, highwayhash, metro64 and murmur3 are not in the table because they sit at the sampling noise floor with the top group, statistically indistinguishable from random. Quality picks no winner here. It only removes losers.
Hashing against an adversary
Everything above assumes a corpus the hasher’s author did not have to plan for. An attacker who knows your hash function is a different threat model.
| tier | candidates | guarantee | cost |
|---|---|---|---|
| unkeyed | xxh3, xxhash, wyhash, metro, murmur3, FNV, the CRCs | none against a chosen-key attacker | baseline |
| keyed | hash/maphash, siphash-2-4, highwayhash | bounds worst-case collisions, if the key is secret | maphash at baseline, siphash 3.5-7x |
| collision-resistant | SHA-256, BLAKE3 | secure against an adversary who knows everything | 15-25x for small inputs |
An unkeyed hash lets an attacker who knows the algorithm compute colliding keys offline and turn a hash table into a linked list. The keys used in this benchmark are fixed constants so that runs reproduce, which is a property of the harness and not of the algorithms: the table describes what each one supports, not how it was configured here.
For a table keyed on untrusted input, hash/maphash is the right default on this evidence. The
runtime seeds it for you, it is in the standard library, and between 36 and 256 bytes it is also the
fastest candidate measured. siphash-2-4 costs 3.5x to 7x more for the same guarantee and buys
nothing extra unless you need it portable to a process that cannot call into maphash.
The harness
Adding a candidate is one registry entry, which then feeds the byte sweep, the string sweep, the scaling sweep, the type axis and the quality analysis. A new entry is asserted for distribution soundness by default, and exempting it means writing down why.
Three tests guard the harness rather than the hashes, because each of these failures is invisible in the output. One checks that the key generators produce distinct keys: an early version built a “one million entry” map holding one hundred, and every number downstream of it was fiction. One checks that a library’s string path agrees with its byte path. One checks that no registry entry is silently missing from a sweep.