Near Copies, Near Names: Fuzzy Hashing Four Ransomware Campaigns

Text A is a short passage about this paper; Text B starts as the same passage with one letter changed (“ships” becomes “shops”). SHA-256 changes 60 of its 64 digits, as if the two passages had nothing in common. TLSH changes six of its 72 characters, and its distance score is 5, where 0 means identical. Press Swap paragraphs: SHA-256 scrambles again, and TLSH barely notices (distance 6), because the same pieces are still there. Press Unrelated text and the TLSH distance jumps to 285. The paper used 100 as the line between “close enough to group together” and “not.”

Each hexadecimal digit stands for four bits, so a 64-digit SHA-256 is 256 bits. Flipping one letter changes about half of those bits (119 of 256 here), the “avalanche” that makes SHA-256 useful for checking a download and useless for spotting a cousin. TLSH works differently: it measures what the file contains, not its exact order of bytes. That is why Swap paragraphs gives a distance of 6 while Add a sentence gives 32: the added sentence brings new byte patterns and makes the file longer. Edit Text B yourself and watch the distance climb as you change more.

The highlighted characters mark positions that differ between A and B. For the one-letter edit, the TLSH distance of 5 is one point for the checksum byte and four for the body, the 32 bytes that encode the quartile of each of 128 counters. Add a sentence costs 32: 17 in the body, 12 for a two-step change in the Q1 ratio, and one each for the length bin, the Q2 ratio, and the checksum (the next demo but one breaks it down). You can also load two of your own files; they are read in the browser and never uploaded. TLSH refuses inputs under 50 bytes and inputs that leave half or more of the 128 counters at zero, returning TNULL.

TLSH distance is not a metric in the strict sense (the quartile-ratio and length terms use a circular difference and a step penalty), and it has no fixed upper bound; the paper cites the algorithm’s authors as saying scores rarely exceed 2,000 in practice, and its own largest pairwise score was 1,112. The scaled similarity shown here, 1 − d/1,112, assumes the paper’s minimum of 0 (each sample compared with itself) and its reported maximum. Unrelated English text scores 285, or 0.744 on that scale, which already hints that scaled similarities near 0.9 are less reassuring than they look.

TLSH reads a file five bytes (here, five characters) at a time, sliding forward one byte per step. From each group of five it picks six combinations of three bytes and turns each combination into a counter number. Drag the slider to move the window and see which counters it adds to (the highlighted bars). After the whole file is read, TLSH ranks the counters and keeps only which quarter each falls in, from the faintest bars (the lowest quarter) to the most strongly colored (the highest). Those quarters, plus the file’s length, become the digest.

The six triplets always include the newest byte, paired with two of the four before it. A lookup table scrambles each triplet into a number from 0 to 255; the default digest uses only counters 0 to 127, so roughly half of the triplets land in counters it ignores (shown in gray). The dashed lines are the three quartiles: a quarter of the counters sit at or below Q1, half at or below Q2, and three quarters at or below Q3. Each counter then gets a two-bit code (0, 1, 2, or 3), and 128 counters × 2 bits = 256 bits, or 64 hexadecimal digits, the body of the digest.

The mapping is Pearson hashing: h = T[T[T[T[salt] ^ a] ^ b] ^ c] with a fixed 256-entry permutation T and six salts, one per triplet pattern. The header holds a one-byte checksum (the same Pearson hash run over consecutive byte pairs), a length bin on a roughly logarithmic scale (the 803-byte sample text falls in bin 16), and the two ratios ⌊100·Q1/Q3⌋ and ⌊100·Q2/Q3⌋, each kept modulo 16. Because only quartile membership survives, uniform growth of a file leaves the body unchanged, and small edits move only the few counters whose rank crosses a quartile boundary.

The port follows TlshImpl::update and final from the reference C++ code. Quartiles are order statistics of the 128 effective counts (positions 31, 63, and 95 after sorting), which is what the reference partition search returns, and the validity rules (at least 50 bytes, Q3 > 0, more than 64 nonzero counters) are the same. Header bytes are nibble-swapped and the body is written last counter first; on the reference regression set the port reproduces all 80 digests character for character. The configuration matters for comparing with the paper: “V1, 128 buckets” corresponds to the T1-prefixed default build with a one-byte checksum, not the 256-bucket or three-byte-checksum variants.

The grid shows the quarter of every counter for Text A (top two rows) and Text B (bottom two rows); outlined cells changed quarter. The distance adds a few points for each changed cell, more for bigger jumps, plus points if the files differ in length or in the spread of their counts. The bar underneath places the score on the paper’s scale: the shaded part, up to 100, is where the authors linked two samples together.

A counter that moves one quarter adds 1 point; two quarters add 2; jumping from the lowest quarter to the highest adds 6, so big shifts count much more than small ones. Use the slider to try distances: the paper turned each distance into a similarity between 0 and 1 by computing 1 − distance ÷ 1,112, so the cutoff of 100 equals a similarity of about 0.910. Every pair in the paper scored somewhere between 0 and 1,112.

The total is the sum of five terms from lsh_bin_totalDiff: the length term (0 or 1 for adjacent bins, otherwise 12 per bin of circular distance over 256), two quartile-ratio terms (0 or 1 for a step of one, otherwise 12 per additional step over 16), one point for a checksum mismatch, and the body term, summed over 128 two-bit codes with costs 0, 1, 2, and 6. The default build, which the paper used, keeps the length term on, so two otherwise identical files of widely different sizes score far apart. Inverting a min-max scaled distance makes TLSH read like @DisCo’s cosine-based score, where 1 means identical.

The body term is a weighted Hamming-style distance between quantized histograms, and the 6 for a three-quartile jump makes the score sensitive to counters that change rank sharply rather than to small drifts. The port reproduces all 6,480 reference regression distances (with and without the length term) and 206,570 distances between 500 generated files exactly. Scaling by a data set’s maximum makes scaled similarities depend on the most distant pair in the set: adding one unusual sample changes every scaled value, which matters when comparing averages across studies or against a fixed threshold such as @DisCo’s 0.98.

The real samples are live ransomware, so this demo builds 162 harmless stand-in files in four families of the same sizes as the paper’s. Each square of the picture compares two files: the stronger its color, the more similar the pair. The outlined squares are the families, and the bright blocks inside them are files that share most of their content. The slider sets how close two files must be to count as linked; linked files form clusters. At the paper’s setting of 100, the stand-in files form seven clusters, each from a single family. Slide to 150 and clusters from different families start to merge into one.

Think of every file as a dot and draw a line between any two dots whose TLSH distance is at most the cutoff. A cluster is a group of dots you can walk between along lines. Groups of only one or two files are left out, as in the paper. Some stand-in files are “packed”: most of their bytes were replaced with random data, the way an encrypted payload looks, and TLSH cannot link them to anything (3 Revil, 11 Darkside, and 11 WannaCry, the same counts the paper left out). Try a cutoff of 50 and of 150 and count the clusters each time.

Each stand-in “program” is a sequence of function-sized chunks drawn from a family-specific byte distribution, with shared runtime chunks in every family. Families have one or more builds (half of their chunks rewritten), and each sample is a build with light edits; Edits per sample sets that rate. With 4% edits, cutoffs from 60 to 85 give eight single-family clusters with the paper’s excluded counts, 90 to 100 give seven (the two TrickBoot builds join), and 135 is the first cutoff that merges families (two clusters, one mixing three families). The table applies the paper’s scaling with this set’s largest distance (about 276) in place of 1,112.

The stand-in set was designed to resemble Figure 2 (one Revil and one WannaCry build, four Darkside builds, two TrickBoot builds, and packed copies in the paper’s excluded counts), so the cluster counts it produces illustrate the procedure and are not evidence about real ransomware. What the demo does show is the sensitivity of single-linkage components to the cutoff: one sub-cutoff edge merges two clusters, so a cutoff chosen by eye on one sample set may not transfer to the next campaign. The scaled within-cluster means here (0.75 to 0.88 at 4% edits and a cutoff of 100) sit below the paper’s (0.85 to 0.999) mostly because the denominator is this set’s maximum, 276, rather than 1,112; the raw mean distances (38 to 77) are the comparable quantity.

For each of the ten groups @DisCo found, the paper reports how similar the samples looked to @DisCo (filled circles) and to TLSH (open circles), from 0 (nothing alike) to 1 (identical). In most groups both tools score the samples as close to identical. Group 7 stands out: @DisCo put a WannaCry sample in with TrickBoot samples, and its average drops with a wide spread. Switch to Absolute difference to see how far apart the two tools land in each group; the largest gap is about 0.14.

The lines through each circle span one standard deviation, a measure of how spread out the scores are. Six groups have an @DisCo score of exactly 1.0000 with no spread: every function in every sample matched. For those groups, the gap between the tools has to equal 1 minus the TLSH score, and it does (group 4: 1 − 0.9686 = 0.0314, as printed). Group 7’s wide spread fits one sample unlike the rest: the authors note that both the WannaCry sample and the TrickBoot samples in that group are DLLs (dynamic link libraries).

@DisCo’s score for binary A against binary B is the share of A’s functions whose best match in B has cosine similarity above 0.98, so it is directional, and it saturates at 1 when every function finds a near-identical partner. TLSH’s scaled score is symmetric. The within-cluster absolute differences (0.0006 to 0.1428) are much smaller than the 0.201245 average over the full 162 × 162 matrices, which the authors read as agreement where samples are expected to be alike, and disagreement concentrated where the two tools would draw cluster boundaries differently.

The table below is the page’s own check of the printed values. For the six rows with @DisCo exactly 1.0000 ± 0.0000, |@DisCo − TLSH| reduces to 1 − TLSH elementwise, so both the mean and the standard deviation of the difference must match the TLSH column, and all six do to the fourth decimal. The implied mean raw distances, (1 − TLSH) × 1,112, run from about two and under one (clusters 9 and 10, near-identical rebuilds) to about 160 (cluster 7), and three clusters (5, 7, and 8) average above the cutoff of 100. That is possible for single-linkage components, and also consistent with these being @DisCo clusters, which TLSH never had to link.

• Both tools grouped the 162 ransomware samples mostly by campaign: TLSH formed eight groups, each from one campaign, and @DisCo formed ten.
• @DisCo split the Revil campaign into three groups, which may reflect versions built for different victims.
• Inside the groups, the two tools gave nearly the same similarity scores, so each can check the other, and the samples they disagree on are the ones worth a closer look.

• The data set held 162 samples from MalwareBazaar: 81 Revil, 45 Darkside, 27 WannaCry, and 9 TrickBoot.
• Linking pairs with a TLSH distance of 100 or less gave eight clusters with no campaign mixing; 25 samples (11 Darkside, 11 WannaCry, and 3 Revil) were left in groups of one or two.
• @DisCo gave ten clusters and left out 20 samples (13 Darkside, three WannaCry, and four Revil); one cluster mixed a WannaCry DLL into TrickBoot.
• Averaged over all pairs, the two tools differed by about 0.2 on a 0 to 1 scale; inside clusters the difference ranged from 0.0006 to 0.1428.

• Data. 162 PE and ELF samples from four campaigns (Table I); TLSH v1 with 128 buckets and default parameters; maximum pairwise TLSH distance 1,112, min-max scaled with scikit-learn 0.24.2 and inverted.
• Full matrices. Mean absolute difference between the scaled TLSH and @DisCo matrices: 0.201245 (Figure 1).
• Clusters. TLSH at a cutoff of 100: eight single-campaign clusters (Revil 1, WannaCry 2, Darkside 3 to 5 and 8, TrickBoot 6 and 7), 25 samples excluded. @DisCo: ten clusters (Revil 1, 2, and 5; Darkside 3, 4, and 10; WannaCry 6 and 8; TrickBoot 9; TrickBoot plus one WannaCry DLL 7), 20 excluded.
• Agreement. Within the ten clusters of Table II, mean absolute difference 0.0006 to 0.1428; @DisCo means 0.8618 to 1.0000 and scaled TLSH means 0.8547 to 0.9994.

Defenders share lists of bad files so that one organization’s discovery protects many others. Those lists only work if they recognize the next copy, and ransomware authors rebuild precisely so that the old list fails. A fuzzy hash such as TLSH can still recognize a rebuilt file, and a code-level tool such as @DisCo can explain which parts match. Ultimately, the paper shows that two distinct ways of measuring similarity agree on which samples belong together, which gives defenders a way to double-check each tool and to spot the samples that need a human analyst.

The authors frame four properties of a good indicator: it narrows the search space, explains why two samples are similar, works at more than one resolution of a sample, and scales to the volume of new malware. TLSH is fast and needs no disassembly; @DisCo is slower and needs a working lift of the binary, but its matches point to specific functions. Pairing them gives a cheap first pass and an explainable second pass, and the pairs where they disagree (packed or encrypted regions that @DisCo cannot read, or shared libraries that TLSH sees only as bytes) are exactly where analyst time pays off.

The contribution is a cross-validation argument for multi-resolution indicators: byte-level and function-level similarity largely recover the same campaign structure, and their disagreements are informative. The limits are those of a single case study: 162 samples from four campaigns, one cutoff chosen visually, manual entry nodes for the cluster search, and no ground truth below the campaign level (build, packer, or victim). The authors’ stated next step is to explore other ways of combining @DisCo’s function matches into a binary-level score that stays explainable.

Notes from building this page

• Which clusters Table II describes. The text says the table averages “the clusters depicted in Figure 2” (TLSH, eight clusters), but the table has ten rows, and the appendix and Figure 3 give @DisCo ten clusters. The campaign labels used on this page assume the rows follow Figure 3’s numbering; the six rows with @DisCo exactly 1.0000 are consistent with either reading.
• Table numbering. The table is captioned “Table II” and followed by a stray “Table III” line; the text refers to it as Table III.
• How the scaler was applied. scikit-learn’s MinMaxScaler scales each column independently. Applied to the 162 × 162 distance matrix as a 2-D array, each sample’s column would be scaled by its own maximum (and minimum 0 from the diagonal), not by the global 1,112. The paper’s text implies a global scaling; the page assumes it. Under column-wise scaling the TLSH matrix would not be symmetric, and the implied raw distances in the check table would not apply.
• What “average” includes. The text says each cluster’s matrix was averaged. Whether the diagonal (self-comparisons, TLSH similarity 1 and @DisCo 1) is included changes the means and shrinks the standard deviations, most for small clusters. This page’s simulated table includes the diagonal.
• Directional @DisCo scores. @DisCo’s A-to-B score can differ from B-to-A. The absolute-difference matrix compares a symmetric TLSH score with a directional one, so its average depends on which direction fills each cell; the paper does not say whether the matrix was symmetrized.
• Excluded samples. The TLSH clusters exclude 25 samples and the @DisCo clusters 20, and each cluster set’s averages use only its own members, so the two sets of comparisons cover different samples.
• Average raw distances above the cutoff. If the scaling was global, clusters 5, 7, and 8 average more than 100 in raw TLSH distance (about 147, 162, and 107). That fits the reading that Table II describes @DisCo clusters, whose members TLSH never linked; it would be unusual, though possible, for TLSH single-linkage clusters.
• TLSH configuration. “Version one with 128 buckets” matches the reference library’s default T1 build (one checksum byte, length term on). The library’s -xlen option and the 256-bucket or three-byte-checksum builds give different distances, so the paper’s 1,112 maximum and cutoff of 100 are specific to the default.

Try this in class

1. Copy a paragraph from a textbook into Text A and Text B. Change one word in B and record the SHA-256 digits that change and the TLSH distance. Repeat with two, four, and eight changed words, and graph distance against the number of changes.
2. Shuffle the order of the sentences in B without changing any words. Predict whether the TLSH distance will be small or large before you check, and explain the result using the five-byte window.
3. In the pipeline demo, find the smallest cutoff at which two different families first join one cluster. Discuss what a security team would lose by choosing a cutoff that is too high, and what it would lose with one that is too low.