Notably, none of the the algorithms tested in the cited study are Apple's NeuralHash, or comparable algorithms. They look at aHash, pHash (plus a variant thereof), dHash, and and PDQ (used at Facebook for similar applications, apparently). The first 4 date to between 2004 and 2010; the last is more recent, but conceptually similar - the citation [0] for PDQ puts it in the same bucket of 'shallower, stricter, cheaper, faster' algorithms as the first four.
No one has proposed any of those as 'illegal image detectors'. Apple's NeuralHash may or may not be robust to the same or different perturbations but the cited study provides basically no new information to inform the conversation its press release wants to be a part of.
Neuralhash is so easy to trick that not only can I make small changes that change the hash, I can make small (visible but non-annoying) changes to make an image entirely match an essentially arbitrary hash.
Simply making the hash not match in any single bit is so trivial that it's hardly worth trying. If you give me a 360x360 image of your choice though I'll make one for you if you still have any doubts.
I don't think Apple has ever claimed that NeuralHash was hardened against intentional false positives. And when properly considered as just one component of their CSAM scanning architecture, NeuralHash being hardened against intentional false positives isn't necessary for it to serve its purpose.
> being hardened against intentional false positives isn't necessary for it to serve its purpose.
Except that every time they get a positive, the device will upload the image to Apple, where at some point an actual human will have to check it.
Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This is just a slight twist on the common prank of making people Google terms related to CSAM.
Not only will that cause Apple to violate the privacy of innocent people who were just saving cat pictures from Facebook, it will also require a lot more manpower than Apple is probably anticipating to keep up with the amount of false positives that will come in. And that isn't even considering unintentional false positives.
I don't think Apple is planning to start sending out "oopsie we violated your privacy and looked at your memes/family photos" mails to affected people. But it might come to that too once this planned practice is tested against current or future European privacy laws (and such a notification would be the bare minimum in my opinion).
That won't go over well once the first people share such an experience.
> Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already.
This might actually be brilliant and we might end up thanking them for it. It might pretty much destroy this filter.
> This is just a slight twist on the common prank of making people Google terms related to CSAM.
And here is the problem: there is one failure mode of this and that is if a bunch of ordinary people get jailed for innocent memes before the public gets whiff of whats happening.
So if they do they should probably be targeted as a waterhole "attack" at Apple execs and pro censorship politicians.
Apple employees reading this: think very closely before downloading wallpapers the next few years.
1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers.
2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review.
3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced.
4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.
Your argument works just as well if for any x, NeuralHash(x)=1.
The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.
> Your argument works just as well if for any x, NeuralHash(x)=1.
That's good, because they weren't arguing that NeuralHash is infallible, but that its fallibility does not lead to the consequences outlined by the GP.
> The fact is that a 'first line of defense' that is trivial to fool is useless.
Useless to you, but not to apple. It reduces the set of pictures to process from "everything" to "those identified by the algorithm as designed, plus artificial false positives". It's impossible for the second set to be larger than the first, so it's by definition not useless.
This is why it's strange how people are focusing so hard on the weaknesses of NeuralHash when what we should care about is how well the system as a whole works.
The pictures aren't secret. The hashes are. We don't know what hash algorithm is employed on Apple's server side, so we can't feed that algorithm with these non-secret images and get the hashes.
Apple claims without evidence that there is a second secret hash which is run serverside. Perhaps it's just Microsoft PhotoDNA (which isn't secret anymore), perhaps it's just another copy of neuralhash (which might share the same collisions)-- it isn't currently known to the public.
It is, however, known to anyone inside the project at apple and anyone involved in the abuse image databases at a minimum.
I'm of the opinion that any security properties of this supposed second hash should be disregarded in public analysis of the system-- for all we know it's just a "return true". Other people make a reasonable argument that this additional function makes some of the attacks people describe harder. In practice it probably does, but security analysis should concern itself with the worst cases admitted by the known system.
> This isn't a reasonable stance, it's just a tactic of poisoning the well such that you can never lose the argument.
I refrained from outright accusing you of bad faith when you claimed that the example image was obviously manipulated, though I confess that was my instant impression. I'm sad that you haven't extended me the same courtesy.
It's considered a bad practice and insecure for systems to keep a cryptographic _algorithm_ secret. In a cryptosystem all that should be secret is the key itself which is small and uniformly randomly generated and by construction cannot contain any weaknesses that could be uncovered by inspection (as it's just uniformly selected out of the space of possible keys).
This doesn't at all apply to a perceptual hashing system: the system will certainly contain weaknesses which could be found by chance or by limited black box interrogation, but it could only be cheaply reviewed for flaws with direct access to it.
The fact that secure cryptosystems by construction protect the keys from leakage through usage is part of the reason that it's okay to stuff all the security dependency into the key.
If you used HMAC with an unknown key and allowed me oracle access to see if inputs I sent in matched, I couldn't learn anything about your hmac key (unless the underlying hash was broken). But if we didn't know the weights of neuralhash and I got match oracle access to it and access to a few matching images (maybe 1 per bit of hash), I could likely the entire set of weights just through interrogating an oracle that gives a pass fail. (Also the papers that successfully design adversarial inputs for unknown networks, simply by designing adversarial inputs for a set of entirely different networks)
Moreover, this second perceptual hashing system requires that Apple share the system with third parties-- they must share it with e.g. NCMEC and the agencies of other governments because it is a felony for apple to knowingly handle the unlawful images themselves so they cannot produce the database themselves, it has to be produced by NCMEC that has the special statutory carveout. By contrast, in a standard public key encryption or authentication scheme your private key is shared with no one ever. That makes it much easier to keep secure (though keys often do get leaked and stolen).
Finally, if I steal your pgp key or whatever, it effects a single use or a single user. Most cryptosystems use as many keys as reasonably possible so the loss of one has the limited impact-- and they can be replaced with another uniformly generated random key if there is a concern for a key's security. But there is only one second image hash, they likely cost millions to develop, so the leakage of it would be much more damaging to the system compared to leaking a key.
I'm perfectly willing say that we have a reasonable disagreement on value of the 'secret' additional hashing scheme. The role of obscurity in security systems has been debated for decades in forums far beyond here-- and we're not likely to add more to it in this discussion.
But please don't accuse me of poisoning the well because we have opposing perspectives on this broad subject. I think I've adequately argued why your comparison point isn't a great comparison. We might not agree but we should agree that neither of us is taking a disingenuous position.
> Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already
That's a very optimistic view.
In reality, they need to automate something while they never get any feedback whatsoever if their efforts are successful, let alone worthwhile.
Crooks that attack Paypal at least know when they succeed - they see the money. In this case, not only they won't see the money, they won't know if what they're doing is "overwhelming" the system or not.
What also seems to be missed is that it’s trivial to foil such attacks in an automated way. The work required on Apple’s end would be smaller than that of the attackers.
Complaints about the proposal seem to overly focus on the reliability of singular aspects of the automated detection system when this has already been planned for - additionally those comments seem entirely unaware of the threshold technology in play.
If they implement should we all start intentionally spamming 'hot hash hits' to overwhelm the system? I guess someone would either need the CP hash already or break the law and morals to get one.
Not everyone's morals are the same. There are a lot of people who don't think it's immoral to simply download an image. Especially if you didn't pay for it with money / ads.
And some people don't think it's immoral to stare at CSAM all day long if it's what you're paid to do in order to confirm that it is indeed CSAM especially since this knowledge will be somehow used to prevent further creation of CSAM... somehow? Probably?
I am being facetious, I know, but my point is that humans' morals and ethics are incredibly widely diverse ― especially if we're talking about the "effective" instead of "proclaimed" ethics (i.e., the values people actually follow instead of values they believe/claim to follow).
Uh, the gp points both intentional false positives and intentional false negatives (IE, anyone intentionally keeping CSAM can easily hide it while anyone can create an image that triggers the hash and gets human to scan your image and etc).
Intentional false negatives are mostly irrelevant. If you want to store galleries of illegal material on your phone, there are much better ways to do it which don't involve them being visually mangled, even if only slightly.
Intentional false positives require knowledge of a hash. And even that isn't enough to get a human to scan your image. You would need to acquire a library of A1 CSAM images/hashes. You would need to generate images which match both NeuralHash and the unspecified secondary hash algorithm which runs on Apple's cloud servers. And then you'd need to repeat this trick a sufficient number of times to trigger human review, at which point a few people (probably some minions in a developing country) get to look at the images for 1 second before pressing the "not CSAM" button and having the images sent to a technical team in Cupertino for analysis.
Huh? False negatives and false positives are irrelevant with respect to the utility of the overall system. It means they don't render the system irrelevant.
Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery.
Intentional false positives are irrelevant because there are, depending how you count them, between two and six other layers which would need to be thwarted or exploited in order for intentional false positives in NeuralHash to result in any adverse outcome.
>Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery.
So if the purpose of the system is not to catch persons dealing with CP, what is the purpose? Breaking user's privacy?
We have a history of evil intentions masked by fighting against CP, drugs, money laundry and crime in general.
Facebook is catching millions of CSAM images being directly uploaded by people to Facebook/Messenger of all places. There are still plenty of complete idiots trading CSAM that one could see value in apprehending.
> That's what boggles my mind. I really don't understand how someone could be trading in CSAM without knowing how dangerous it is to trade in CSAM.
One possibility is that these databases are stuff full of material which isn't actually CSAM. There is no accountability for the content of the databases.
Considering the infinitesimal number of prosecutions compared to the claimed number of hits, we should hope that this explanation is actually the truth...
I believe there is an incredibly wide spectrum of most human qualities and traits.
I see no reason to think that general intelligence, technical awareness, utility maximization, impulse control, or risk-reward calculation wouldn’t have similar distributions such that plenty of people would end up in a set prone to share risky material on such channels.
There is pretty much no enforcement mechanism against any crime which we can expect to catch everyone. False positives get bounced out by the safeguards, false negatives wouldn't have been caught in the first place... that still presumably leaves a number of true positives who wouldn't have otherwise been caught.
A system that detects Covid-19 in infected people and doesn't detect it in healthy people, where the errors of both types (false positives and false negatives) are irrelevant, is not only relevant, it's a very well designed system.
> Intentional false positives require knowledge of a hash.
The code/network detecting this will be run on the client. So yes, a 'hash' or an equivalent will be known.
You would have to hope that the secondary system is robust enough to not fall victim to the same manipulated image. And it really is on Apple to first prove it isn't.
> The code/network detecting this will be run on the client. So yes, a 'hash' or an equivalent will be known.
They use cryptography to conceal the hash database from the users to guard it against accountability.
Because of this, if they stuff the database with political or ethnically targeted images no one will be able to tell, you just have to trust that they haven't done that.
Knowledge of a matching hash isn't a big barrier for another reason: Matching images are widely available. NCMEC reports that they had over 20 million matching images reported to them last year. It's a very reasonable assumption that some 4channer will helpfully troll the internet for likely illegal images and put up a large database of likely matches.
False positives can also go the other way: Modify illegal images to match existing lawful images which are already in circulation. One could likely take legal hardcore pornography, crop it to make it impossible to identify the ages of the talent, add captions to falsely claim that they are under-age, tweak it to match targeted images, and submit it to the databases.
No, it is not. The hash is computed on the client, but hash comparison happens on the server, using some cryptographic magic that only reveals a “visual derivative” if enough images match. It’s fairly complex and I don’t recall the details, but the library of “bad” hashes is not accessible.
No, the entire premise of NeuralHash is that no information about your pictures leaves your system unless it has been flagged as 'potentially CSAM'. NeuralHash itself is basically a DB of CSAM hashes encoded as an artificial neural network.
In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash.
Edit: I am almost entirely wrong, see the comment reply from krisoft. Apologies!
> In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash.
There is a fairly complicated cryptographic algorithm which, according to apple, lets them open the safety voucher only if there is a match with the database.
The user devices only have a cryptographicaly blinded version of the hash database which they can’t turn back to real hashes without apple’s private key. This means that the plain neuralhashes are not available to the end users.
The other reason why false negatives are mostly irrelevant is because the hash DB can be updated to include them, so the risk of storing them in iCloud remains the same.
The encrypted list is stored on the devices for security and performance reasons: otherwise it would be too easy to ship people customized lists with arbitrary images. You can't just rapidly revise the list without undermining the privacy protection here.
Also the space of modifications is gigantic, for any given image there should be millions of differently hash variations which can be reached without major visual damage. (at least for neuralhash the figure is billions)... so you can't just stick all of them in the database.
You could stick widely circulating variations in the database, however... sure.
The hashdb is part of the operating system and only gets updated with OS updates.
It can be updated-- at some cost. But if it is very frequently updated then the protection against receiving a targeted list is lost.
To state it another way: Users ask "Why won't the government order apple just gives me a list of the hash of every political image on the internet in order to expose my files?" and the answer is that the database is part of the OS, common to everyone, updated infrequently, so it can't be used in targeted attacks. That protection is lost if the database is frequently updated.
It's also true that already uploaded photos won't be rescanned, but that wasn't the point I was trying to make.
Intentional false negatives are mostly irrelevant. If you want to store galleries of illegal material on your phone, there are much better ways to do it which don't involve them being visually mangled, even if only slightly.
So, you're accepting that Apple's approach is useless for anyone who's "determined" but you still seem to be arguing there's some merit in this complex "architecture" of surveillance.
I disagree that it's useless. In fact I wouldn't be surprised if it's already been effective despite not being implemented yet. The announcement of an impending threat of scanning probably managed to get most CSAM afficinados to swear off iCloud already. And if so, it would be no small irony that the noisy criticism would have ensured that Apple was heard loud and clear by the CSAM community.
I disagree that it's useless. In fact I wouldn't be surprised if it's already been effective despite not being implemented yet. The announcement of an impending threat of scanning probably managed to get most CSAM afficinados to swear off iCloud already.
So the idea is that mass surveillance is great even if it's effect is scaring criminals while it actually only catches people at random?
We've arrived at the opposite of the common reprise of law. No longer "It is Better that Ten Guilty Persons Go Free Than that One Innocent Person be Convicted" but "It is better that Ten Innocent Persons Be Dragooned Than that One Guilty Person Go Free"
All I'm saying is that I understand why Apple doesn't want CSAM on their servers, while also reserving the ability to implement end to end encryption on photo galleries in future.
It's a tough line for them to walk.
Personally I don't see any substantial moral difference to what Apple was planning versus what Google and Facebook already do, aside for the somewhat icky aspect that some initial CPU cycles are occurring on hardware I own.
I'm equally concerned that a purported need to scan for CSAM could be used by legislators as a way to convince the population that the Government needs more encryption backdoors. Personally I'd rather have my photos inconsequentially hashed than have encryption legislatively broken.
Actually, you've made quite a few claims, including some that even ineffective surveillance is good 'cause it makes criminals worried. But anyway...
that I understand why Apple doesn't want CSAM on their servers, while also reserving the ability to implement end to end encryption on photo galleries in future.
Given you've admitted Apple's filtering is ineffective, you have admitted that criminals could put CSAM on Apple servers.
But weird thing is "end-to-end encryption" is entirely useless if everyone is being surveilled before the start and after the end. Sure, Apple wants to be buzzword compliant with privacy and entirely surveil it's users in reality. I suppose I too can see how that would be appealing.
Having illegal activity on a major cloud service makes it harder for those services to offer robust encryption without law enforcement pressuring legislators to mandate backdoors.
This system is a backdoor. We should not be surprised if adding a backdoor reduces pressure to add a backdoor. :)
But in the US it isn't just a backdoor, it's an indirect violation of the user's 4th amendment rights. If the government ordered apple to perform this scanning, the scanning would be unlawful and the result would be inadmissible in court. It is only lawful because Apple performs it voluntarily (or at least that is the pretext maintained in court-- repeatedly Apple staff have maintained to me that the government is requiring them to do this). The government should not get the ability to execute unlawful searches simply because it's able to use soft power to coerce user trusted third parties to perform the searching on their behalf.
The article and kristjansson's post are about intentional false negatives.
My comment was pointing out that the well known fact that the neuralhash is highly vulnerable to preimage attacks (intentional false positives) also means that is is highly vulnerable to intentional false negatives. Changing the 96-bit hash to a specific value while minimizing the visible image change is a LOT harder than just changing any single bit, but the harder task has been amply demonstrated.
Neuralhash is one component, yes, but it is a limiting component. If a slightly tweaked image has a different NeuralHash it will never be detected by their scheme.
I don't think Apple is concerned about the <0.1% of CSAM aficionados who read about Apple's plans to scan for CSAM and thinks "hey, let me blindly trust some tool I found on the internet to slightly mangle my highly illegal images so that I can continue co-mingling them with personal photos in iCloud, accessible to anyone who picks up my phone while unlocked. Or any other device synchronised to the same iCloud account."
The App Store is presumably full of easy to use apps which can store photo galleries in a separately encrypted vault. This is where people will go after being shooed away from iCloud, not image manglers.
Say I'm a cop who wants to get into your phone but can't get a warrant. I happen to have an image on my (Android) phone which looks benign to a human, but triggers Apple's CSAM detectors when scanned. I text it to you, the CSAM detector goes off, and suddenly I have probable cause.
Cops plant illegal drugs on suspects all the time. Why not this?
2. You'd have to somehow convince your target to import all 30 images into their photo gallery;
3. The chances of the images accidentally passing the human review step is infinitesimal;
4. Unlike a bag of drugs which actually do contain drugs, these fake images would be like police finding a bag of sugar with "FAKE COCAINE FOR MANUFACTURING PROBABLE CAUSE" written on it.
Imagine a politician receiving dozens of photos which trigger this CSAM algorithm because he has iCloud photos as defaul, and then somehow someone gets wind of the fact…
BANG: one less. Try to explain to the public that it is only the first line of defense against child sex and that it does not imply anything at all.
This seems very obvious. Neural networks are the "best" image detectors we have. It's documented that they can be easily fooled.
The insidious thing is that this can used as a pretext to make the filters flag more images since this would seem to make it "harder to hide illegal images", until Apple just personally scans everything.
There's a Pandora's box here, too. Bad actors can simply intentionally disseminate memes that match arbitrary bad hashes. Trigger the 20 image (or whatever count) rule for millions of people if you're successful enough.
The system can be attacked at the opposite end as well. Governments can claim hashes are CSAM when the hashes are actually images typically held by political enemies.
I think apple claimed that it requires a database agreed to by at least two governments. Count the countries that discriminate against gay people or authoritarian regimes that regularly cooperate with each other and you quickly realize that Apple just advertised its services to the most morally corrupt and evil people alive.
Matching arbitrary bad hashes for NeuralHash is not sufficient. You also need to somehow match unknown-to-you bad hashes for an unknown-to-you second hash algorithm before you even get to the "30 images" and "human review" parts of the pipeline.
This is a pretty hot take! I think by this logic, your comment is also theft - it uses my brain cycles and calories from my hard earned budget to process.
(Your second paragraph allows for no context or nuance and is just completely unnecessary.)
I've had working versions of software stolen from me by Google chrome auto updating extensions and Android autoupdating apps. There needs to be something like package.json or cargo.toml to protect users from rogue application developers, rogue operating system developers, and rogue chrome extension developers.
Auto update at least in theory provides a value to the user. The algorithm implemented by Apple goes out of its way to ensure that the user wont be able to interpret the result, only Apple and potentially law enforcement will ever know it.
Lots of people would be surprised. Because they aren't educated about how these things work.
I'd wager that project managers working on these teams also don't have the understanding (let alone intuition) to judge whether these risks are present, and so will continue fighting for it. The wager is based on the presumption that someone who did understand these risks would pivot or otherwise not allow this initiative to continue.
PMs fight so hard because from their position their job is at risk if the project takes a nosedive. But they don't know what they're advocating for a lot of the time.
I know quite a few folks like this in FAANG and adjacent spaces.
Hell even the author of the linked article doesn’t seem to understand what hashing is. Underneath a collection of original and modified images they’ve put
“These images have been hashed, so that they look different to detection algorithms but nearly identical to us”
But machine learning! Neural nets! Artificial intelligence!
All words that have meaning that becomes diluted in public perception, and lead to things sort of becoming a "black box", where we don't really know how the models actually do what they do.
Most machine learning models get it right only in 80% to 90% of cases. So if you have millions of users each with thousands of pictures, you can see how often you run into problems. Even with 99% accuracy, the amount of errors is enormous.
Go back on this very site to August posts and read the comments about how infallible apple's special sauce of anti-privacy is. "It's not a hash it's a perceptive hash so changing a few things won't fool it" even though there was rumors that you could false positive it all day with flesh colored cake frosting with raspberries on it.
You are correct, fooling one of the two hashes Apple places between an image and human review is technically easy, with some prerequisites. (The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges).
Talking about how trivial it is on a technical level is, in my opinion, an oversimplification. For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM images. These are images of sexually explicit material where pre-pubescent children are actively involved.
Following introspection of my own personal moral compass, it's clear to me that being knowingly in proximity to such material would not be "easy" for me.
You are correct that the output of a false positive generator would not be CSAM. But one of the two inputs would need to be CSAM or a derivation of it.
I doubt you'd find a list of NeuralHash CSAM hashes. And it's even less likely that you'll find a list which includes both NeuralHash as well as Apple's secondary perceptual hash which has not been made public.
But maybe you could. Let's say hypothetically, such a torrent does exist. Apple could just rev their hash algorithm in the next version of iOS, immediately rendering those lists useless.
If the target hash is sufficient, then yes, that would invalidate my argument. Although that still requires one to come into possession of such a hash string, which might not be "easy" as a purely practical exercise.
All my preimage attack images were created using only the hash of the target, no access to the target image was needed, nor is it obvious to me how it would be helpful at all (at least assuming the goal was for the false positive to look like some unrelated image).
> The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges
At a minimum it is known to the state actors generating these databases. Apple itself has directly acknowledged abuse by state actors as a threat vector, so it is entirely inappropriate to rely on obscurity here. (and would be a bad security practice regardless, since presumably other attackers could steal the other algorithims).
> For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM image
This is false. It requires knowing the hash of the target image, which is completely lawful to possess. You don't have to be in possession of the illegal images.
You cannot assume these hashes are impossible to obtain-- because if they were it would mean the whole scheme was pointless (since it would also mean the unlawful images were impossible to obtain). But even if they were impossible to obtain, an attacker could simply create new "apparent" child abuse material submit it to the databases while at the same time creating whatever false matches against it that they wanted.
You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.
> You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.
What I'm saying is that a whole bunch of stars need to align before any practical use for this exploit becomes even remotely plausible. This is slightly more likely if your adversary is a state actor with strong motivations and no issues accessing and handling CSAM. But consider the necessary steps and ask how likely it is:
1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default.
2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database.
3. Acquire the secondary hash algorithm and build a tool that generates hash collisions for that.
4. Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them.
5. Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious.
6. Hope that your target does actually import all thirty images into their iCloud photo gallery.
7. Hope that your hash collsions all work correctly in practice.
8. Hope that Apple's human reviewers screw up.
9. Hope that none of the 30 images cause any of Apple's reviewers to become suspicious, flagging the matter for technical review, causing Apple to blacklist these hashes and rev their hash algorithms in the next version of iOS.
10. Hope that the child safety organisation which Apple reports this to also screws up.
11. Hope that the law enforcement body which this child safety organisation reports this to also screws up.
12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target.
That's a dozen steps, many of which are exceedingly non-trivial. There's probably a dozen more steps I'm missing or unaware of.
Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.
> 1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default.
True for millions of people, also assumes the functionality won't be expanded once initially deployed.
> 2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database.
Authorities claim these databases have millions of hits per year (NCMEC says 21.7 million hits in 2020), so clearly these are not that hard to obtain.
> Acquire the secondary hash algorithm and build a tool that generates hash collisions for that.
Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security.
> Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them.
So. Ordinary, perfectly lawful hardcore pornography-- just minimally selected so that you can't tell how old the talent is. This is indistinguishable from child abuse material especially in review context which has been described (viewing low resolution sobel transformed 'extracts').
But also: At this point when someone is reviewing any of their images the users privacy is already violated. Moreover, a hacker or state actor which has compromised Apple's systems could intercept the content at this point and monitor it regardless of what the review found.
> Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious.
I can't speak to the claimed other hash, but I've demonstrated that for neuralhash you can make images match hashes without looking modified, and certainly without making them look unappealing.
> 6. Hope that your target does actually import all thirty images into their iCloud photo gallery.
well there you have me, because no one who cares about personal privacy or civil rights should ever use an apple device. :)
> 12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target.
Just being accused or questioned is a devastating negative consequence. It's also not too difficult to find examples of people being criminally charged over totally lawful pornography because the prosecutors couldn't bother to use the internet to look up the actresses DOB.
> Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.
That may be true, but it has a different risk reward tradeoff. By your argument swatting would never happen-- it requires a lot of complicated steps, more so than just pulling the trigger on a gun or swinging a baseball bat yourself.
And for what benefit? As the article shows-- these system's are easily evaded. Should our privately owned computing devices be tattling on us based on opaque naughty lists which are cryptographically protected against accountability? -- just to catch some perverts trading widely circulated gross pictures which they're too stupid to fuzz against detection? and not even that-- as you noted in other posts: The pervs will just use other tools. So we degrade everyone's privacy, further establish the precedent that it's morally acceptable for your own private property to conspire against you, and probably don't actually protect any children.
I think we should admit: Especially in light of the news on the same week this was announced about the doctor that was keeping lots of child porn in their iphotos and was caught by another service. The purpose of this scheme isn't to protect children, it isn't to preserve user privacy, it exists to increase and preserve Apple's profit margin by distancing them from creeps and it does so at the expense of the privacy of their users. Apple is converting their customer privacy into a private profit.
ISTM the whole scheme may actually encourage abusing more children: The one reliable way to get sexual pictures of young people that this system cannot catch is to make new images. If this sort of thing were effective it could be expected to enlarge the commercial market for new abuse images by shifting demand from the back catalog of existing material to new material.
I think you're missing the broader point. This isn't about any one step. I know you can make counter-arguments for each individual step in isolation, but in order for this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit.
> True for millions of people
But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here?
> Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security.
With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to.
For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed.
Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets.
> But also: At this point when someone is reviewing any of their images the users privacy is already violated.
Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.)
> I've demonstrated that for neuralhash you can make images match hashes without looking modified
I haven't seen any examples that didn't look modified. That image of Lenna you posted earlier is quite clearly modified.
> Apple is converting their customer privacy into a private profit.
You couldn't possibly accuse them of doing so surreptitiously.
If there are any consequences of such schemes being implemented, the consequences will have already happened because of similar implementations by Facebook, Google, etc. The only difference with Apple's approach is where certain CPU cycles are occurring. Apple could do all of the hash scanning on cloud servers (like Google) and the functional outcome with respect to your personal liberties would be exactly the same.
> this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit.
>> True for millions of people
> But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here?
Please try to imagine for a moment applying your same style of argument to a typical buffer overflow attack.
Attackers aren't random processes. No CGI is going to get exploited and escalated to root access and website defacement through a random process, sure. Yet the attacks happen.
The fact that the attack only works on 20-40% of people to begin with isn't particularly relevant in modeling the risk.
> With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to. For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed.
The same could have been said about the microsoft PhotoDNA ... they kept it secret to keep "bad guys" from access it. But I have a copy of it now. (also people successfully reproduced it from vague descriptions in academic papers)
> Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets.
I've certainly known people that had their phones full of porn.
> Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.)
Yes they do. They see ("a derivative of") every database match once the user has reached the threshold number of matches.
Edit: On reread I understand what you were arguing there. I misunderstood you at first. I still disagree with you: Images users got from elsewhere are still their private files and are no less so because they didn't make them themselves.
> That image of Lenna you posted earlier is quite clearly modified.
I have a hard time believing that you aren't only saying that because you've seen the original. That image wouldn't strike anyone as unusual.
> You couldn't possibly accuse them of doing so surreptitiously.
I can-- but even if I couldn't, the fact that it's not very surreptitious doesn't make it good.
I think Apple has mislead people about the level of privacy protection provided via cryptowashing, e.g. going on about the private set intersection when it doesn't provide any privacy for your users but just hides the database from the public to prevent accountability. Apple also made misleading claims e.g. about the false positive rate of neuralhash-- claiming that it had "trillion to one" FP rates, when its real FP rate in image collections is higher (e.g. there are colliding imagenet images) and constructed attack images will always FP. The fact that advocates have expended time and effort to dismantle Apple's claims doesn't change that they were misleading to begin with. :)
> The only difference with Apple's approach is where certain CPU cycles are occurring.
To the extent that other parties scanning also applies on person to person messaging that has a reasonable expectation of privacy, I would also hold that the scanning is unethical. But since the service providers own equipment is involved I would also freely admit that the ethically of it is debatable, at least if its disclosed, and that reasonable people could disagree.
In the US this kind of scanning only ends up being lawful at all due to legal inconsistencies. If the communication were not stored it would be an unlawful wiretap (under both federal and many states laws) to scan users private communications. Saving it to a mailspool magically changes the law that applies, even if the behavior visible to the user is identical.
But the distinction that it is the user's on the user's own device eliminates the "it's on the provider's server" excuse. Your own device being programmed against your will to surveil you and report you is just unethical, in my view.
Like many ethical questions there is a boundary. Scanning public communications? Fine. Scanning third party private communications on your own devices? Uhh. Maybe, there be dragons. Your own device scanning your private files and reporting you against your will? Not okay.
Even if the operating behavior is close to the same the principle matters.
You are still trying to skirt round the point that all stars must align for this attack to work. And there are enough stars to make it an absurd proposition. Multiply these likelihoods together and it quickly surpasses the likelihood that the user getting struck by lightning at the exact moment that they win the lottery.
As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I know it's an inappropriate thing to say on HN but claiming that the "image wouldn't strike anyone as unusual" is so far beyond reasonable that I feel myself seriously questioning your sincerity.
I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced.
> As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage.
I think it looks like a scan of a crappy smudged 30 year old photograph. Which is also exactly what it is.
FWIW, I thought your comment saying the lena was obviously manipulated was transparently in bad faith, but I held my tongue-- so don't feel too guilty. I would take a sizable wager on that people wouldn't identify it as anything special, and I'm quite surprised to hear that you think differently.
(In one of the threads I posted in it one of the people through it was the barbara image that was changed rather than the lena one...)
> I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced.
I'm not convinced that you're being serious. I rejected that response because it's self-evidently absurd. There were twelve mandatory prerequisites I identified for successful exploitation and the vast majority of them are not under the influence of any attacker, no matter how non-random the hypothetical attacker is.
Do you understand what happens when you're forced to multiply probabilities?
How does this attacker acquire a functioning copy of this secondary hash algorithm, which for all you know is held at least as securely than Apple holds private keys for stuff like iOS cryptographic signing.
How does that attacker change the mind of a person who would never import hard core porn into their phone's photo gallery?
Do you know if Apple's human review requires fooling one reviewer or multiple? I personally have no idea but as someone who has implemented a human review system for critical data, placing such consequential decisions in the hands of a single human is a non-starter. For all you know, the 30 images will be looked at by 30 different human reviewers. How does the attacker ensure that ALL of them will overlook the absence of a PRE-PUBESCENT CHILD in the photograph?
No sir, I do not think you are not being serious. And that's intended as a compliment, because you're clearly a smart person.
But if you are such a person, who these systems are trying to fight against, it is trivial to defeat the detectors. These systems aren’t designed to hurt the average person.
I agree, building tools which could lightly mangle CSAM enough to change its hash and defeat detection is trivial. Generating false negatives from a hash algorithm is trivial.
If we're going to surrender our privacy in such an absolute way, can we also install programs on cars to stop them from going over 80mph? I mean, if everything is on the table, let's do something that will save lives.
No, instead we've made cars remotely controllable and programmable so that someone can kill you or pedestrians from their computer, then download some CSAM onto the car's infotainment system for good measure.
Is it a slippery slope fallacy to assume they don't have children's best interest in mind and are using this for other reasons? I see China forcing them to run searches on hardware for them to stay in their market, and with the way Apple kowtowed with iCloud encryption, it would not be a suprise if China already did this and they just need to run more models to train and use for their own marketing reasons. They can get metadata from the devices this way so I don't see anything more private about it: they will not only scan your iCloud pictures, they make your phone run their calcuations at no benefit to you, and its not possible to disable what is going to be used for non optional telemetry if you use iCloud.
There's not a doubt in my mind that China already scans domestically distributed content, otherwise they wouldn't have strong-armed Apple into storing Chinese iCloud data in domestic datacenters. If you really want to put on your tinfoil hat, there's an overwhelming lack of evidence to suggest this isn't happening in America, considering Apple's history of PRISM compliance and inevitable pressure from five-eyes.
>. If you really want to put on your tinfoil hat, there's an overwhelming lack of evidence to suggest this isn't happening in America, considering Apple's history of PRISM compliance and inevitable pressure from five-eyes.
I suspect that individuals working with sensitive information in other countries do not use iOS or official Android when they are at work.
In English that can be parsed either way. If you are sincere in your misunderstanding of this grammar I'm going to assume you are a non-English speaker or might have a slight development disorder relate to communication.
I can't image Apple engineers don't know this. But it might have been a negotiating give-away to the (tech-ignorant) government akin to gun control laws banning rifle-mounted bayonets and calling it "gun control progress".
It’s not an image classifier. It’s an image hashing algorithm that does a server-side comparison to a secret corpus of hashes provided by the National Center for Missing and Exploited Children (or a similar body—I forget the details) and equivalent international organizations. Only hashes that appear in multiple countries’ databases are used.
> in its current form, so-called perceptual hashing based client-side scanning (PH-CSS) algorithms will not be a ‘magic bullet’ for detecting illegal content like CSAM on personal devices
Whoever said it would be a magic bullet?
> The researchers say this highlights just how easily people with illegal material could fool the surveillance.
Ha yes, all you need to do is go to a university computing department and ask them to research algorithms to fool the scanner, and then turn it into an easy-to-use app.
Then you chance it with some real CP images. It might work! Or not.
Interesting but I don't think anyone at Apple will be shocked by this.
Plus, my opinion has been for some time that the point of Apple's image-scanning (for Apple) isn't to detect harmful material in iCloud, it's to project an image of not wanting the harmful material stored in iCloud.
Apple has a vested interest in preventing iCloud from becoming "That image storage place for pedophiles."
Also, I'm sure that the FBI is probably pretty okay with "we only catch the people who ever, in a single instance, forget to filter their images through this program that fools the filters" since that's probably like 99% of all people.
That whole notion of "image storage for pedophiles" is deeply stupid and fundamentally illiterate. It is like calling a bank's safe deposit box used to store sealed envelopes "a child porn sanctuary".
When dealing with demagogues it is time to go on the offense - there is a reason why McCarthy was damned by the line "Have you no decency?!"
> I'm sure that the FBI is probably pretty okay with "we only catch the people who ever, in a single instance, forget to filter their images through this program that fools the filters" since that's probably like 99% of all people.
Then why bother with the false-positive-risk inducing neuralhash to being with? Compute the sha256 sum of the decompressed image pixels and use that... at that point a false positive is a successful attack on sha256.
The only downside is that trivial image modification will make the hash not match, but that's also true for the perceptual hashes... just with a slightly more complicated definition of trivial. It will be equally effective against the idiot-pervert -- arguably more effective because there would be a lot less reason for people oppose the deployment of such a scheme.
Using a cryptographic hash would make the software vastly simpler, vastly faster, and effectively eliminate a whole host of privacy attacks.
They could then go further and stop using the cryptographic obfuscation that protects apple against accountability for the content of the database-- just make the database the cleartext sha256 hashes. Doing this would substantially address a whole host of other privacy concerns and would make the system radically simpler and less cpu hungry.
They can't do that with the neuralhash because of how trivial it is to modify images to match target hashes... but after switching to a cryptographic hash they certainly could. (Well, could except the database providers know their database contains a lot of lawful content and are extremely aggressive at preventing the lists of hashes from getting out).
> The only downside is that trivial image modification will make the hash not match, but that's also true for the perceptual hashes... just with a slightly more complicated definition of trivial
Well, the difference between the two version of "trivial" is the difference between "incidental" and "purposefully".
Re-compressing, resizing or cropping the image changes the SHA256, but (with high probability) not the NeuralHash. Those operations happen all the time in the normal course of image distribution. Unlike "running through a tool designed to alter the NeuralHash value".
In other words, your suggestion trades false positive rate for false negative rate, under the argument that any nonzero false negative rate is effectively the same. This argument is not convincing, particularly when your suggestion would reduce the true positive rate from (a hypothetical) 99% to a much lower value.
One could still adopt a scheme which would be robust against recompression much of the time. E.g. downsample the image to a specific bounding size (e.g. 360x360), drop the least significant bit or two of each pixel value. SHA256 the result.
This would be robust against a significant percentage of recompression pipelines, and yet would still maintain a clear guarantee against false positives: Unless SHA256 is successfully attacked, no one can find two images with the same hash unless their 360px/6-bit-per-color downsample is identical. (And small crops-- are they really that common?-- could be handled by adding them to the database. For significant crops the neuralhash won't be preserved in any case.)
I'm aware that attackers can make contrived images that look rather different when downsampling. But that is still a much much stronger promise than neuralhash could make since the downsample hack requires obviously contrived false positive images (unlike neuralhash false positive images).
Essentially neuralhash (and similar approaches) is the solution you get when you weigh maximally in favor of false positives to avoid false negatives. SHA256 the pixels (or the whole image files) is the solution you get when you favor false negatives to avoid false positives. You can hit many points in the middle.
Given that no scheme can be robust against intentional false negatives and that apple is happy to allow 30 hits before treating it as a match, I'd argue that the apple scheme (or similar) is just an objectively bad choice.
Let's assume image compression randomly flips the last bit of every pixel (this is a crude approximation but it should do).
Let's further assume your original image was 1k x 1k and was shrunk by a factor of 3, so each downscaled pixel is the average of 9 original ones.
Let's then assume that all nine pixels have to "flip" in a specific way to change the downscaled one, so a 1/2^9 chance.
The odds of this bit flip propagating past the last two bits you dropped are 1/4 (dropping bits is not very effective at combating random behavior!).
That means there is a 1/2^11 chance that adding compression to the pipeline changes any given downsampled channel-pixel in the 360px/6-bit-per-color hash input.
The SHA256 stays the same if this does not happen in any of the 360 x 360 x 3 channel-pixels. The odds of that are (1 - 1/2^11)^(360*360*3) = 3.4e-83. You could drop two more bits and the odds would still be 2.4e-21 (each dropped bit amounts to taking the square root).
In effect, your suggested scheme would simply never assign the same hash with any amount of compression happening.
Fundamentally, you still have way too many pixels that might spuriously change from tiny perturbations of the input image. Reducing resolution further is actually much more effective than dropping bits, because it both increases the averaging effect and reduces how many pixels have a chance to flip.
While I agree that this is most probably a PR stunt, it's also an interesting conflict of interest: Apple has probably lost some trust in tech communities since, but might conversely build trust with law enforcement, legislators and the broader public so that they may continue without external intrusion.
The irony is, Apple has probably achieved most their goal with the announcement, making the implementation largely redundant.
It's worth stopping to consider that while Apple's plans were a hot topic for the the civil liberties crowd, that's probably piss in the wind compared to how hard the announcement landed among underground CSAM communities across the globe. I dare say that even the most technically inept CSAM aficionados will be acutely aware of this looming threat and will know to keep that shit far away from any of their Apple devices.
(It would be especially ironic if there's now more awareness in those communities of Apple's semi-aborted CSAM scanning than of Google or Facebook's CSAM scanning, which has been operational for many years.)
If legal porn can be modified to match CP, do you think the human reviewer is going to analyze the picture more closely or just press the report button and move on?
No, I think you don't know how the proposed CSAM scanning works:
1. It's only done on cloud photos. That person would have to manually save each photo to their photos library (with iCloud photos turned on) for the scanning to occur.
2. After reaching a threshold # of photos automatically flagged, a human reviewer has to confirm that the photos are illegal before any action is taken.
So just texting someone even real CP won't do anything.
> a human reviewer has to confirm that the photos are illegal before any action is taken
The reviewer actually only confirms that the-photos-as-modified-by-an-undisclosed-transformation are illegal, and this strategic ambiguity is presumably designed to maintain plausible deniability that what Apple subjects its reviewers to is a) not psychologically damaging, and b) not illegal to possess.
Unfortunately this is "security by obscurity", so we have to assume it is vulnerable to image scaling attacks[0].
Unless Apple document their human review process, and allow an independent audit of the technology (to provide a strong mathematical guarantee that it is not susceptible to such image-smuggling attacks) and independent verification that the published process is the one they actually use, then there is an unacceptably high chance that a human reviewer will trigger a legal process against users for files they couldn't tell contained illegal data.
The proposal to perform scanning of photos in iMessage was an entirely different thing. It would look for photos that were likely to be sexually explicit and show a warning message prior to displaying them, but only where:
1. The account is for a child under the age of 13
2. The account was configured with family sharing
3. The feature was enabled by a parent
Results of these scans would remain entirely local to the device and—with the consent of the child—also sent to the parent. But I think this last feature has been dropped.
One feature does blind CSAM detection on iCloud photos. Any photos that are being added to iCloud in the USA have a safety voucher calculated on-device and attached to the upload. The device doesn’t know if there’s a match or not. When the uploads reach Apple’s servers, if there are enough matches, Apple has enough information to decrypt the safety vouchers. The photos are then additionally compared with a second, secret hashing algorithm. If those also match, a human reviewer checks a low-res thumbnail to confirm. If they also confirm, it gets reported. So the initial calculation of the safety voucher is done on-device, but the matching can only be completed on the server, and nothing gets reported unless multiple other checks also match.
The other feature uses machine learning to determine whether or not a photo attached to a message contains nudity. If such an image is received by a child’s iPhone, it’s blurred out and they are told that if they choose to view it, their parents will be told about it. The entirety of this scanning happens on-device.
No one has proposed any of those as 'illegal image detectors'. Apple's NeuralHash may or may not be robust to the same or different perturbations but the cited study provides basically no new information to inform the conversation its press release wants to be a part of.
[0]: https://github.com/facebook/ThreatExchange/blob/main/hashing...