Jangseung: A Guard at the Door of a Learning Model

Each tile is a real handwritten digit carrying the wrong label. Jangseung compares it with the training examples of the label it claims and lets it in only if it fits. Try 0 and 1: a 0 looks nothing like a 1, so almost every fake is caught. Now try 4 and 9, two digits many people write alike. Far more fakes slip through, which matches the paper’s hardest pair.

The guard is an isolation forest: it builds 100 random “twenty questions” trees that try to separate one example from the rest by asking whether a single pixel is brighter or darker than a random cutoff. Unusual examples get isolated in fewer questions. A 7 labeled as a 1 sits far from every real 1, so the 1-forest isolates it fast and turns it away. Change the pair and count how many of the 100 fakes get caught.

Each trial fits one isolation forest per digit on the training digits (half of the training size each), draws 100 training digits at random, flips their labels, and scores each one with the forest of the claimed digit. A digit passes when its score is at least −0.5, scikit-learn’s default cut. The chart below the tiles places genuine digits and poisoned digits on the same score line. The paper’s attack also nudges each point along the hinge-loss gradient, but by only 0.001 grey levels (out of 255), so the tiles show each poisoned digit exactly as drawn.

The anomaly score is s(x) = −2^(−E[h(x)]/c(ψ)), where h(x) is the path length to isolate x in one tree, ψ = min(256, n) is the subsample, and c(ψ) = 2H(ψ − 1) − 2(ψ − 1)/ψ normalizes by the average unsuccessful-search length in a binary search tree. The guard passes s ≥ −0.5. The forest of the claimed class sees a flipped digit as an outlier only when the two digit distributions separate along random single-pixel cuts, which explains why 0 pairs are easy and 4 and 9 are hard. The score line also shows the cost the paper does not report: some genuine digits fall below the cut too, and a guard that screens new data would turn those away as well.

This classifier reads 30 measurements of a breast tumor from a medical image and calls it harmful or harmless. Press Run, and the page sends in poisoned records one at a time: real records with the answer swapped. The gray line has no guard. The colored line has Jangseung. With 500 poisoned records the unguarded classifier ends up worse than a coin flip, while the guarded one stays close to where it started.

The page trains two copies of the same classifier on 300 real records, then adds poisoned records one at a time and retests both copies on 100 records they never saw. The unguarded copy learns from every poisoned record; the guarded copy learns only from the ones Jangseung lets through. Watch what happens once the poisoned records outnumber the real ones: for most patients, the training data now holds more wrong answers than right ones, and the unguarded classifier starts answering backwards.

Algorithm 2 in the browser: a stratified split of the 569-record Wisconsin diagnostic set, an isolation forest per class, and two RBF support vector machines retrained after every poisoned record, all inside a Web Worker. Smaller training sets fail sooner. The paper reports unguarded averages of 90.03% to 90.72% for 50 to 250 poisoned records and 75.51% for 500; the guarded models stay above 91% and at 88.18%, respectively.

Each poisoned record is a copy of a training record with its label flipped (the gradient step is 0.001 in feature units, negligible next to features that span tens to thousands). Once the flipped copies outnumber the originals, the label majority flips locally almost everywhere, and the RBF machine learns the inverted rule, which is why final unguarded accuracy can sink far below the majority-class rate. The guard caps that by rejecting 67% to 79% of the poison in the page’s runs. Features are unscaled, as the default pipeline implies; scaling them would change both the clean accuracy and the guard.

Each square shows how many of 100 fake digits Jangseung turned away for one pair of digits. Darker squares mean a stronger guard. Every pair with a 0 is easy to defend. The paper’s hardest pair, 4 and 9, is the hardest here too, and 7 and 9 comes next in both.

The page’s map reproduces the paper’s ranking at the extremes: 0 and 1 is the best pair (99.8% here, 100% in the paper), 4 and 9 the worst (20.0% here, 36.5% in the paper), and 7 and 9 the second worst in both. The page turns away fewer poisoned digits overall than the paper reports (55.5% for 1 and 7 against 75%, 44.5% for 5 and 8 against 73%, and six pairs above 95% against ten). Pixel scaling, subsample choices, or a different forest cut in the authors’ code could each account for a uniform shift of this kind.

A classifier trained on more examples is harder to push around, so the guard matters most for small training sets. The tumor view shows this plainly: with only 50 real records, the unguarded classifier drops below 50% on average while the guarded one stays near 91%.

On the tumor data the page shows the paper’s pattern, with a larger effect at the smallest size (a 42-point gain at 50 training records here, 15.12 points in the paper) and almost none at the largest (0.4 points at 450 here, 1.03 at 500 in the paper). On MNIST the page’s unguarded models stay between 98.5% and 99.3% at every size, far above the paper’s 53.2% at 250 digits. The elliptic envelope turns away less poison than the isolation forest and loses ground past 300 attacks, the same ordering as the paper’s 89.4% and 91.3% averages.

• With Jangseung in place, the protected model beat the unprotected one in every case the authors tested.
• Handwritten-digit models trained on only 250 examples averaged 53.2% accuracy under attack without the guard and 91.1% with it.
• On tumor data, 500 poisoned records pulled the unguarded model down to an average of 75.51%; the guarded model averaged 88.18%.
• Digits that look alike, such as 4 and 9, were the hardest to protect.

• The guard caught every fake in the 0 and 1 pair and only 36.5% in the 4 and 9 pair.
• After all 100 attacks, the biggest gap between guarded and unguarded digit models was 96.2 percentage points (the 6 and 9 pair).
• Well-trained digit models (1,000 examples) barely needed help: 97.6% unguarded, 97.8% guarded.
• Small training sets gained the most: the guard added 15.12 points with 50 tumor records and 1.03 points with 500.

• Every pair improved. Across 720 MNIST experiments the guarded model’s average accuracy beat the unguarded model’s, by 3.2 points (1 and 8) to 17.6 points (0 and 8).
• Final accuracy tells the bigger story. After 100 attacks the guarded model’s final accuracy led by 26 points (1 and 2) to 96.2 points (6 and 9).
• Under-trained models benefit most. At 250 training digits, 53.2% unguarded against 91.1% guarded; at 1,000 digits, 97.6% against 97.8%.
• Tumor data. Clean accuracy 89.93% to 91.58%; averages under attack 84.5% unguarded, 89.4% with an elliptic envelope, and 91.3% with an isolation forest.
• Longer attacks hurt more. Unguarded tumor models held 90.03% to 90.72% for 50 to 250 attacks and fell to 75.51% at 500; guarded models stayed above 91% and then 88.18%.

Models that keep learning after they are deployed are useful exactly because they adapt, and that same openness lets an attacker teach them wrong answers. A guard that checks each new example against what its class normally looks like adds little cost and protects the models most at risk: the ones that start with little data. The same check also catches honest mistakes, such as a mislabeled sensor reading, before they quietly degrade a city service.

Jangseung needs no change to the classifier and no knowledge of the attack; it is a filter in front of the training set, so it can sit in front of models already in service. The cost is false rejections of genuine data that happens to look unusual, a trade-off any deployment would tune with the forest’s cut.

Per-class density screening is a cheap first line against label-flip and near-copy poisoning, the cheapest attacks an adversary can mount on a continuously retrained model. The paper argues a further benefit: guarding weak models removes the easy surrogate an adversary would attack first and transfer to a stronger model. Stronger poisoning attacks that optimize points to look typical of the claimed class are the natural next test.

Notes from building this page

These notes interpret or question the paper and are offered to the authors for review.

• Which tumor data set. The title and Methods name the diagnostic set, but Methods quote 699 vectors (458 benign, 241 malignant), which are the counts of the original Wisconsin set. The default SVC (RBF, gamma “scale”, unscaled features) gives 89.1% to 91.9% on the 569-record diagnostic set across the paper’s train and test sizes, bracketing the reported 89.93% to 91.58%; on the 699-record original set it gives 95.7% to 97.0%. The page uses the diagnostic set.
• The attack is label flipping. With step t = 0.001 along the unit gradient and tolerance ε = 10⁻⁵, the loss rises by at most 0.001 times the gradient norm per step, and for every poisoned point the browser engine generated in its tumor and MNIST checks, the loop stopped after one step. A poisoned point is therefore a copy of a training point, moved by 0.001, with its label flipped. The page caps the loop at 10 steps; the cap never binds.
• MNIST drops do not reproduce. Under that attack the page’s unguarded MNIST models average 98.5% to 99.3% at every training size (eight pairs, one run each), against the paper’s 53.2% at 250 digits and its final-accuracy gaps of up to 96.2 points. Copies with flipped labels, about two in seven training points at the smallest size, do not move an RBF machine that far. The paper’s drops suggest a larger effective step, rescaled pixels with a larger t, or a different kernel. Question for the authors: which settings produced Figures 1 to 4?
• Why the tumor model inverts. Once flipped copies outnumber the training records, most neighborhoods hold more wrong labels than right ones, and unguarded final accuracy falls to 10% to 16% in the page’s runs (300 training records, 350 or more attacks).
• Algorithm 2, line 16 appends every adversarial point to the guarded set; the text says it joins the unguarded set, and the page follows the text.
• Small inconsistencies. The isolation-forest tumor average is 91.3% in Results and 91.4% in the Conclusions; Methods describe sixteen tumor train/test combinations and Results ten training sizes; which detector the 88.18% figure uses is not stated.
• What the paper does not measure. False rejections of genuine new data, and the guard’s behavior against points optimized to look typical of the claimed class. The first demo shows genuine digits below the cut at graduate and researcher levels.

Try this in class

1. In the first demo, pick five digit pairs you think look alike and five you think look different. Record how many of the 100 fakes the guard turns away for each, and compare the two groups.
2. In the tumor demo, run 500 poisoned records with 50, 200, and 450 training records. Make a table of the final accuracy with and without the guard, and explain why the smallest training set fails first.
3. Discuss as a class: the guard also turns away some real examples. When would that be a price worth paying, and when would it not?