POOR MAN’S PALANTIR: AI as an Expert Consultant – Danger

UPDATE

It has been ten months since I introduced the work (here) I began in 2022 to assess how AI could be used to identify vulnerabilities in national critical infrastructure.

At the time, AI represented an impressive leap beyond the established, search-intensive process of online research. Yet using AI for this purpose quickly revealed a number of important limitations. Today, the knowledge and sophistication of AI are dramatically greater, but many of the fundamental limitations I identified then remain.

My planned “Phase 2” article was intended to reveal more about the project. It would describe how the methodology evolved, particularly as I explored the ways in which different AI systems appeared to reason through the problems presented to them. I used several different AI solutions, and the differences between them were sometimes remarkable. Yet despite their differences, they continued to exhibit many of the same fundamental limitations.

I eventually had to pause the article; not because the research had failed, but because of what the research was beginning to reveal.

The more I explored the capabilities of AI, the more apparent it became that AI could make access to information that was not properly approved for distribution and, more significantly, information subject to export controls, far too easy and potentially far too extensive.

That became the most important lesson of the project. And, in retrospect, it was a much bigger lesson than anything I had learned about using AI to identify vulnerabilities in national critical infrastructure.

The Problem Is Not AI; It Is What AI Can Do With Information.

To be clear, my concern is primarily about the outputs of AI, not AI itself; although the broader implications of AI are certainly a subject worthy of extensive discussion.

AI systems are already confronting difficult questions involving copyright and intellectual property. Similar questions arise when AI makes information available that may be subject to export controls, official-secrets laws, contractual restrictions, or other limitations on its distribution.

It is easy to see that AI companies are making efforts to identify and manage potential violations of laws and rights. Some of these controls are visible in the way particular queries are answered, restricted, modified, or refused.

There is, however, a much larger question.

An AI company might reasonably argue that it is not creating the underlying information. It is using information that already exists somewhere in the available data, and therefore the original publisher or source may be the party responsible for making that information available.

That argument may have merit in some circumstances. But historically, laws governing the handling and distribution of protected information have not necessarily been concerned only with the person who originally created or disclosed it. Responsibility can extend to those who knowingly facilitate, transmit, or further a prohibited disclosure.

Export controls raise an especially interesting question because the issue is not simply who originally possessed the information. It can also concern who actually transfers controlled information to a foreign person or entity.

How these principles apply to AI systems is an area that deserves considerably more attention.

AI Is Different From Search

There is an important distinction between AI and conventional search engines.

A search engine primarily helps a person find information. AI can do something considerably more powerful: it can interpret that information, connect disparate pieces of information, analyze them, and explain their significance.

In other words, AI does not simply distribute information.

It can distribute knowledge.

This distinction became increasingly apparent during my research.

An interaction with an AI system can involve a subject-matter expert asking increasingly sophisticated questions and challenging the system’s answers. The interaction itself can expose relationships, terminology, methodologies, and analytical approaches that may not be obvious to someone without expertise in the field.

The result is potentially significant.

A person who lacks the expertise to independently discover, interpret, and connect information may nevertheless be able to obtain a highly sophisticated answer simply by asking an AI system the right question. To be clear, the concern is to what degree experts might be liable for the interaction that helps AI more technically adept at answering questions of other users who may have bad intents.

It was the Preemptive Strike exercise that made me very cognizant of the fact that the structured questions I pushed to AI was giving AI the insight into how to do things it did not yet appear to have learned and applied.

The system has effectively become an expert consultant.

This creates a very different information environment from the one created by a conventional search engine.

Consider the difference between being given a collection of documents and being given an expert who has read those documents and can explain what they mean, identify the important relationships between them, and answer increasingly specific questions about how the information can be applied.

That is the power, and potentially the risk, of AI.

The Expert Consultant Problem

The concern, therefore, is not simply that AI can make information available to someone who should not have it.

It is that AI can potentially make expertise available to someone who does not possess it.

That distinction matters.

A determined and knowledgeable individual can already find an extraordinary amount of information on the internet. The traditional barrier has often been the time, expertise, and analytical ability required to find the relevant information, determine what is reliable, understand it, and combine it into something useful.

AI can dramatically reduce those barriers.

It can help a user move from information to understanding, and from understanding to application.

That means AI potentially does two things simultaneously: it facilitates the rapid distribution of precise information, and it provides analytical assistance concerning what that information means and how it might be applied.

For legitimate users, this is one of the most valuable capabilities of AI.

For malicious users, it could become something very different.

Monitoring the Questions, Not Just the Answers

There are certainly efforts underway to control what information AI systems will provide. Those efforts are necessary.

But I increasingly suspect that the more valuable, and perhaps more practical, approach may be to focus not only on what an AI system provides, but on who is asking what questions.

The questions themselves may be more revealing than the answers.

A single question can be innocent. A sequence of questions can reveal intent.

Someone asking general questions about an infrastructure system is very different from someone progressively narrowing those questions to identify specific vulnerabilities, dependencies, weaknesses, locations, operating characteristics, or methods of exploitation.

AI is uniquely positioned to see that progression.

It is also uniquely positioned to understand it.

This suggests that an important component of AI security may ultimately be behavioral rather than purely informational: monitoring patterns of interaction and identifying when a user’s questions collectively cross a threshold of concern.

Ironically, AI may be one of the best tools available for doing exactly that.

The technology could monitor AI use in much the same way that financial systems monitor transactions, not necessarily because any individual transaction is inherently suspicious, but because patterns can reveal intent.

That may be where the next phase of this discussion needs to go.

My original project began with a question about whether AI could help identify vulnerabilities in national critical infrastructure; it ended up raising a much larger question:

What happens when anyone, anywhere, can access not only the world’s information, but an artificial expert capable of explaining it, connecting it, and helping determine how it can be used?

That is the question that caused me to pause Phase 2.

It is also a far more important question than my project was initially investigating.