An Antidote to AI Intellectual Property Theft

Artificial Intelligence software technology has generated tremendous press over the past three years as multiple companies released competing systems with lofty claims about the capabilities of those systems. Those same releases have scared the bejeezus out of currently well-paid "think workers" (programmers, musicians, graphical artists, knowledge workers) by suggesting their work can be / will be replaced by AI based systems within MONTHS. These AI stories have also generated enormous attention on a related topic – how the firms creating these AI systems "trained" the expertise into those systems by pointing them at PETABYTES of text / image / audio / video content created by other people. This approach to training is legally problematic because none of these firms obtained prior consent from even a fraction of the creators whose work was used for training. And these systems are not just for-profit systems that won't be sharing those profits with the creators, they are for-profit systems being promoted as REPLACEMENTS for the work those creators perform, posing a direct threat to their future work and income.

This highly illegal, highly automated, wholesale violation of intellectual property rights of millions of individuals poses a variety of technical challenges to the larger economy. However, software developers familiar with the operation of AI systems in both training mode and "production" mode are fighting back. New tools have been developed by creators who know AI operators are feverishly attempting to negate potential copyright infringement rulings in the courts both retroactively for their prior theft and going forward to allow it to continue These new techniques might be enough to thwart the exponential growth of AI on jobs in the near term. They might also reflect a huge financial risk for those who invested heavily in AI software firms and the hardware firms making billions selling the computers, storage and networking to support gigantic AI deployments.

Before explaining how these "AI antidotes" work, a short explanation of how AI systems "learn" and operate is required to explain a core flaw in the resulting system that poses problems for operators and intellectual property owners alike.

How AI Systems "Learn" (Simplified)

Most of the AI models referenced in the public today use either a Large Language Model architecture that accepts text as input and generates text as output or Convolutional Neural Network models optimized for analyzing image data for object recognition or new image generation. Generically speaking, these architectures take a vast collection of inputs provided to "train" the system and chop that data into binary data representing small pieces of the total source content. These sub-tokens are read into memory and millions of mathematical operations are performed on collections of those tokens analyzing the statistical probability of token Z appearing after tokens X and Y. If the training is aimed at creating a model for English prose, content in the training data might contain a string of text like this:

And she's buying the stairway to heaven

Of course, many English speaking humans might immediately recognize that as a lyric within a song. But the AI "knows" nothing. It doesn't even see that as a sentence or phrase. That text might be tokenized down to 1-2 character chunks so it is actually processing THIS:

An d sh e's buy in g th e st ai rway to he av en

(And remember spaces are just another token to this logic like any other character.) But the training process has scanned TERABYTES of similar English prose, not just one line. As those TERABYTES were tokenized then analyzed, the resulting statistics would provide tables that reflect the fact that in the training data,

• the combination SH is usually followed by a vowel or space
• the combination SHE is often followed by either a space or apostrophe
• the combination SHE' is nearly always followed by an S
• the combination TH is often followed by a vowel with E being most common

The training process refines those statistics by periodically scanning a separate cache of data NOT included in the training data then comparing how the predicted "next letter" from the first generation of statistics matched the ACTUAL "next letter" in some of the test data. Where the first generation statistics are found to be significantly off, the training algorithm re-weights those statistics, another round of training is done and the process is repeated. This process generates ADDITIONAL statistics that show how much closer predictions match test data in each round of training to track diminishing returns of spending more compute time on training. When the prediction rate stops improving, training is deemed complete.

So with tens of thousands of probabilities like this derived from analyzing PETABYTES of examples of English prose, accessing those probability tables with an "agent" user interface that accepts a "prompt" of text from a user makes it possible to generate text via statistics that seems like a response to the human's prompt.

What about audio content? With audio data, the low-level "tokens" processed by the training algorithm won't be ASCII codes for characters, they will be some small sample of the binary data of the audio waveform, maybe a half millisecond. If you imagine Robert Plant's voice – or ANYONE's voice – saying or singing that lyric, the vowel and consonant sounds will have characteristic waveforms which result in characteristic digital data patterns that will eventually coalesce, allowing the pure audio input data to be mapped to the lyric text.

What about images and video? The same process applies. Imagine a green square in the middle of a white background as reflected in a bitmap image file. When used to train an AI, the AI can identify the width and height of the overall image file in pixels, then find patterns that the RGB color coding of pixels flips between green and white and the (x,y) pixels where the color changes have some predictable pattern to them. Without anyone ever writing an actual program to TELL the AI what (x,y) scheme would reflect a circle versus square versus rectangle versus triangle, the AI can derive that mapping by scanning millions of images, generating statistics, then running an audit

This two-stage training and processing architecture exhibits two key flaws, one that affects both creators and AI operators and another that affects AI operators and their end users. The first problem will be termed the "provenance problem" and the second problem will be termed the "hacking problem."

AI's Provenance Problem

The provenance problem stems from the fact that an AI system's intermediate statistics optimized during training and its final "operational" statistics used by users do not retain metadata about the sources that drive any particular "inference" the system generates as an output. Furthermore, the tokenized nature of the statistics prevents any human-meaningful "search" from being executed to ask the system "where did you learn this?"

In the simplest terms possible, this loss of provenance information is exactly analogous to the following sequence of events:

• someone providing you ten billion individual temperature readings for the globe
• calculating the average of all ten billion measurements
• taking the average value and transferring that to another system then THROWING AWAY all of the source data
• then being told all of the values originating from source X (ten percent of the entire input data) were wrong and need to be adjusted or removed from the average
• then being told the original raw data points and their statistics (number of samples, average value) cannot be re-supplied so they can be deducted from the original average

If you still had the original data points mapped to their source, these flawed inputs could be removed to allow the average to be recomputed with corrected data or without the flawed data. Without the original data, there is no way to "back out" that bad input from the resulting statistics. The only options are to accept the final model with those known flaws or retrain the model after attempting to block that source from being ingested again for training.

For creators of content, this provenance problem makes it virtually impossible to PROVE an AI operator ingested their content. There's no administrative tool a court can force the operator to run that will scan the training statistics or the operational statistics and find a specific probability driven by ingesting a specific source document. This gives operators of the AI system a huge margin of error in terms of plausible deniability that they DID knowingly scan creator X's content and incorporate that content into their model without permission. This is especially true because other middleman firms may have cached the content and the fetch of that content may have never hit the original creator's system to provide a hint the content was pulled.

For the operators of AI systems, this provenance problem is equally problematic because accessing the PETABYTES of online content needed is very difficult to automate efficiently if thousands of exclusionary rules must be analyzed prior to downloading content and feeding it into training cycles. And if unauthorized content IS injected into a training cycle, even if the operator sees proof after the fact their training DID ingest unauthorized material, there is NO METHOD for surgically removing that offending "knowledge" from the final system. This means AI system developers and operators potentially face huge legal problems if courts rule AI training IS violating copyright protections of individuals en masse.

AI's Hacking Problem

AI's provanence problem leads directly to AI's hacking problem. The need for AI operators to feed TERABYTES of data into training processes and the operator's unwillingness to devote appropriate labor resources to carefully curate content sources both for owner consent and appropriateness of the content leads to AI's hacking problem. It's the old "garbage in, garbage out" problem on steroids. It bears repeating. AI systems don't "know" ANYTHING. All they are doing is generating statistics based on consecutive sequences of very small units of data presented in their training data. Training algorithms are not analyzing the content being ingested at human contextual levels. This means ANY data present in the content fed to the training processes will be used for training.

This poses profound risks to operators and users of AI systems. If an operator trains a system and ingests a gigabyte of suspect content amid a terabyte, that suspect content may have data within it having nothing to do with the intended use of the AI system. If the goal is to create an AI model for optimizing derivative trades on corporate bonds but the AI training data included thousands of blogs pitching get-rich-quick schemes generated by robots for a bunch of crypto-coin fanatics living in their parents' basement, the "noise" from those bad sources will be present in that AI and alter outputs in unpredictable ways.

As one example, AIs for speech recognition might be designed to recognize "Hey Google, do X" and pass a command into the user's phone to do X. However, that AI process might detect a "Hey Google, do X" command in data inputs not perceived by the user. The user might be watching a video that embeds the same command amid other sounds that mask it from being detected by the human but can be spotted by the AI which then dutifully executes the command. That command might be "Hey Google, unlock the front door" which may leave your house wide open to burglars.

An Antidote for AI Intellectual Property Theft

The launch of these for-profit AI systems by giant corporations with stolen content has pissed off a lot of content creators. Unfortunately for the giant AI firms, those firms do not have a lock on all AI talent and software development expertise. And many people with AI expertise and software development skills are content creators who understand the threat posed by AI systems to their own livelihoods. And they have devised new tools to combat the wholesale theft of intellectual property by leveraging these two flaws AGAINST AI systems.

Remember, those two key flaws are:

1. The inability of training processes to discriminate "good" and "bad" inputs (whether due to accuracy or security or legality)
2. The forward-only nature of AI statistics during training and operations – the statistical impacts of "bad" data cannot be surgically identified, quarantined or removed from a model without starting over and explicitly excluding the "bad" input which may prove impossible to do.

Simply put, the antidote to this wholesale content theft involves altering content being posted on public sites with embedded data that doesn't interfere with HUMAN consumption of the content but CORRUPTS the larger data stream seen by a webcrawl engine and fed into AI training. It's the equivalent of having content appearing like this to a human:

And she's buying the stairway to heaven

appear like this electronically to the AI training process:

And encabulator she's turbo buying prefabulated the
aluminite stairway hydrocoptic to marzlevanes heaven

This "poisoning" of the content poses multiple problems to the AI operator.

1. While masked from affecting a human user's experience, the AI will see and process ALL of the data since current ingest automation just slurps in the entire data stream.
2. Altering training bot automation to detect and mask this bogus data requires parsing the raw file just like a browser, incurring additional processing times that would increase compute costs for training by 10x or 100x.
3. Because of the provenance problem, once this polluted data enters the training realm, the AI operator has no viable way to back the data out.
4. Because of the provenance problem, once this polluted data enters the training realm, the AI operator has no way to identify the source that actually supplied the data to make explicit efforts to NOT craw that site.
5. Because the AI operator has no direct clue about how this poisoned data affects the model being generated, the AI operator potentially faces liability issues for harmful or inappropriate outputs it cannot predict.

How effective is this poisoning strategy? The 10:00 point in this highly recommended video by Benn Jordan

depicts what happens to the iterative learning curve when training ingests poisoned data. The rate of improvement bottoms out almost immediately in the training cycles. Bottoming out doesn't mean it reaches the desired quality level, it just means learning stops improving, making additiional training pointless at a much lower quality level. The video includes some samples of output from music generating AIs without poisoned data and with.

In other words, this technique has the potential to completely BREAK the business model for operating an AI off stolen data. Of course, there's another obvious implication of this "white hat" hacking technique. The very mechanism being leveraged by creators simply trying to protect their content can be used by bad actors to subliminally link insecure code or content into AI results by injecting similar hidden noise inside other content being scanned by AI operators.

And that's the real takeaway from this latest news in the world of AI development. There are unique flaws to the underlying mathematics and the financial business model of training and running AI systems that have already put the technology in the same technological circle of hell as the perpetual struggle between hackers and computer users and anti-virus software developers. The nature of the technology guarantees neither party can ever secure a permanent upper hand, putting an upper bound on the true value of the process. It will never get better, only different.

WTH