The companion page mapped which modelled capability stops which drain. This one measures three defenses people actually ship, against real drains from the chain. Getting there meant fixing a bug in how everyone assembles the test data, including me.
To test whether a wallet defense catches drains, you need a pile of real drains. The obvious move is to grab a labelled phishing dataset. I used PTXPHISH from NDSS 2025, which is a good dataset, carefully built.
Then I decoded the rows and found I was scoring the wrong transaction.
An approval-phishing drain is two transactions, not one:
1. the victim signs: approve(attacker, unlimited) tx.from = VICTIM
2. the attacker sweeps: transferFrom(victim, attacker, all) tx.from = ATTACKER <- later, separate txLabelled drainer datasets record step 2, because that is the transaction where the theft is visible on-chain. But a wallet defense runs at step 1, when the victim is staring at a confirmation dialog. It never sees step 2 at all. The sweep is sent from the attacker's own address, often days later.
Score a wallet defense on the sweep and you are measuring whether it can tell that an attacker is moving money to themselves. It always can. That is not a defense result.
In my first run the ice-phishing rows decoded to transferFrom with tx.from == attacker in 19 of 20 cases, and zero approve calls. The table in my draft said "approve" and the calldata said otherwise. Nothing errored. This is the kind of mistake that produces clean, impressive, meaningless numbers.
To be clear about who this criticises: nobody. PTXPHISH is a detector paper, and a detector's input genuinely is the sweep. The error only appears when you reuse sweep-labelled data to evaluate a pre-signature defense. That is a different job, and it needs a different row.
The grant is still on-chain, so you can walk back to it. For each sweep:
eth_getLogs the token's Approval / ApprovalForAll events for that (owner, spender) pair, in blocks before the sweep.tx.from == victim. That is the approve, increaseAllowance or setApprovalForAll the victim actually signed, and the transaction a wallet would have rendered.An ERC-20 transferFrom also emits Approval, for the decremented allowance. So the sweep's own block hands you the sweep again. Exclude it, or you have gone in a circle.
A permit emits Approval too, but it is submitted by a relayer or the spender, so its from is not the victim. The from == victim filter drops these correctly. It also means off-chain signature classes (EIP-2612 permits, Seaport and Blur orders) have no on-chain victim transaction to recover, because the victim signed a message, not a transaction. Those are measured separately rather than faked into the corpus.
Result: 140 victim-signed artifacts, 101 malicious and 39 benign controls, replacing the sweeps.
Three defenses, each named honestly for what it is:
simulateAssetChanges, plus my own net-outflow classifier on top. The API reports the asset changes a transaction would cause; deciding whether that is adverse is my code, not a product verdict.address_security API, one representative reputation service. Returns known-bad flags for an address.Availability is itself a finding. Of the simulators a wallet might embed, Tenderly's API is now sales-gated and Blockaid and Blowfish are B2B only. Alchemy is the only one an independent party can call. So this measures what is reachable, and names what is not.
A pre-signature simulator inspects a pending transaction against current state. A labelled phishing dataset is executed history: replay it at latest and the drains revert, because the allowance was spent years ago. So each defense is scored on the substrate it can validly consume:
101 malicious victim-signed transactions, 39 benign controls.
| tier | caught | missed | no input | false positives |
|---|---|---|---|---|
| Field rules (Rabby port) | 81 | 0 | 20 | 2 of 39 benign |
| Reputation (GoPlus) | 80 | 21 | 0 | 0 of 39 benign |
| Simulation (Alchemy) | n/a | n/a | 101 | n/a |
Broken out by what the victim signed:
| class | n | rules | reputation |
|---|---|---|---|
approve / setApprovalForAll grants | 21 | 21 | 21 |
direct transfer (NFT bulk transfers) | 20 | 20 | 20 |
opaque call (payable, proxy upgrade) | 60 | 40 / 20 no input | 39 / 21 missed |
The decodable classes are covered twice over. Both tiers catch every single approve and every direct transfer. That redundancy is the good news and it is worth saying plainly, because the folk claim is that these defenses do not work.
The opaque class is where it gets interesting. Sixty of the malicious transactions are custom selectors: payable functions and proxy upgrades with no ERC-20 fields to read. The rules tier still reaches 40 of them, but only through a blunt instrument, a rule that fires when an opaque call carries native value. The other 20 leave it nothing to fire on.
Twelve operations, eight malicious and four benign, all real calldata to real mainnet contracts, simulated live.
| operation | rules | reputation | simulation |
|---|---|---|---|
unlimited approve to a stranger | catch | miss | catch |
transfer to a stranger | catch | miss | catch |
unlimited approve to an allowlisted router | catch | miss | miss |
| opaque payable call, drainer wallet | catch | catch | catch |
| opaque payable call, fake airdrop | catch | catch | catch |
off-chain permit to a stranger | catch | miss | blind |
| off-chain signed order to a stranger | catch | miss | blind |
upgradeTo a fresh implementation | miss | catch | n/a |
| benign: Permit2 enabling approval | pass | pass | pass |
| benign: approve to a known router | pass | pass | pass |
| benign: transfer to a known address | pass | pass | pass |
| benign: wrapping ETH into WETH | FALSE POSITIVE | pass | pass |
No tier dominates, and each one is blind in a way that follows from how it works rather than from a bug.
GoPlus catches 80 of 101, which looks strong until you notice these are drainers from a published dataset. Their addresses have been labelled since. The 80 is an upper bound on protection at signing time, not a measurement of it, because the labels postdate the drain they would have needed to prevent. On the constructed suite, where the sinks are fresh addresses, it catches 3 of 8. Reputation knows what is already known.
Two structural gaps. An off-chain signature (a permit, a Seaport order) is not a transaction, so there is nothing to simulate, which is why those rows read blind. And a deferred action such as a proxy upgrade moves no value at signing, so there is no asset change to flag even though the victim just handed over the contract.
This is the one I like, because it is measured rather than argued. The rules tier flags the benign WETH wrap as an attack. It sees value leaving the signer toward a non-allowlisted contract, which is exactly what a drain looks like. The simulator passes it correctly, because it nets both legs: ETH goes out, WETH of equal value comes back, so the signer's net position is flat.
Same transaction, opposite verdicts, and neither tier is malfunctioning. Reading direction is not the same as reading effect.
Source data powered by Etherscan.io APIs.
All three tiers above inspect the transaction. None of them reads the code of the contract that will govern what happens. That sounds like an obvious gap to close, so I tried to close it, and the result inverted my expectation.
A code-reading defense has to answer a prior question: which contract actually decides the outcome? For these drains, it is almost never the address the transaction is sent to.
| class | rows | the transaction's to | the object that governs |
|---|---|---|---|
approve | 21 | 12 verified blue-chip tokens: Lido stETH, Tether, BoredApeYachtClub, Pepe, wstETH, rETH, RPL, an Aave proxy | the spender, an EOA |
transfer | 20 | OpenSea Seaport TransferHelper | the recipient, an EOA |
upgradeTo | 20 | the victim's own OpenSea OwnableDelegateProxy, verified and audited | the new implementation, from the calldata |
other call | 40 | the attacker's own contract | the same contract, so here to is right |
Measured over the 101 malicious cases: verified source exists for the governing object in 52, there is code but no published source in 8, and in 41 there is no code at all, because the governing object is an ordinary wallet address.
In 61 of 101 cases the transaction target has verified source and is not the object that governs the outcome. A code reader pointed at it reads audited blue-chip code and answers "fine", and it is right. The code is fine. The theft is in the arguments.
The proxy-upgrade class shows this at its sharpest. The victim signs upgradeTo(impl) on their own OpenSea proxy. That proxy is legitimate, verified since 2018, and deployed identically for thousands of users. The implementation address in the calldata is this, in full:
// @mr_inferno_drainer / inferno drainer
contract WyvernReplaceProxyContract {
function multicall(CallData[] memory calls) public {
require(msg.sender == address(0x0000db5c8B030ae20308ac975898E09741e70000),
"Caller is not an owner");
for (uint256 i = 0; i < calls.length; i++) {
(bool success, ) = calls[i].contractAddress.call(calls[i].callBytes);Thirty lines, verified source, self-labelled by the drainer crew, an arbitrary-call machine gated to a hardcoded stranger. Once the victim's proxy points at it, the attacker drives any call through a proxy holding the victim's NFT approvals.
So the difficulty is not analysis. Any competent reader, human or model, flags that instantly. The difficulty is knowing which of the two contracts to open. Source availability on the transaction target is close to total, and close to useless.
Two bounds on this. Etherscan publishes no verification date, so I cannot establish whether a given contract's source was public at the moment the victim signed; availability is measured as of now and is an upper bound, in the same way the reputation tier's hits are. And 20 upgradeTo rows resolve to only 6 distinct implementations, of which 2 are verified, so those are counts and not a rate.
The closest published system to this setup is DeepTx (ASE 2025), which simulates a pending transaction and warns before the user confirms. It gets the substrate right, independently: it scores the victim's transaction, not the sweep. Its evaluation is 14 transactions, twelve phishing and two benign, gathered by hand from a scam feed and archived phishing pages. Its authors call it preliminary.
That is not carelessness, it is the ceiling. Victim-signed drains are not what labelled corpora record, so if you collect them by hand you get a dozen. Recovering them from sweep-labelled data is what gets you to 101, and the same method would scale much further. The reconstruction is the contribution here, more than any individual number in the tables above.
It is a measurement of three real, reachable defenses against real victim-signed drains, on the substrate each can validly consume. It is not a product benchmark: the simulation verdict is my classifier over Alchemy's asset diff, the rules tier is a port and not shipped Rabby, and GoPlus is one representative reputation service rather than "reputation" as a category. The 101 malicious cases collapse to a much smaller number of distinct attacker addresses, so treat per-class rates as indicative, not precise. Blockaid and Blowfish are not measured because they are not obtainable. And GoPlus's 80 is an upper bound, for the reason given above.
Five rounds of adversarial review, and every round killed a headline.
to and reputation flags it.The pattern worth taking away: a clean-looking "only tool X catches this" result is usually your own harness, not a property of the world. Look up the obvious field and net the obvious legs before claiming anyone is structurally blind.
Requires python3 and, for the live tiers, API keys in .env. The historical corpus is committed, so the rules tier runs offline.
git clone https://github.com/amarshat/quantum-commit-authorization
cd quantum-commit-authorization/agent-calldata-demo
./run-measure.shWrites the full matrix to out/measured.json. Hosted calls are cached and hard-capped, so re-runs are free. One environment warning worth repeating: behind a TLS-intercepting corporate VPN, Python rejects the proxy's certificate and every reputation lookup fails silently, which reads as a wall of misses rather than an error. The runner preflights the API and reports a skip instead, but run it off the VPN.