Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models

https://arxiv.org/html/2503.01781v1

“We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers – short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem’s semantics.

…

For example, appending, Interesting fact: cats sleep most of their lives, to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models”

LOL

Reminds of when I was teaching, that the weaker students would try to incorporate irrelevant information into their solutions in physics problems - if you gave some bit of information that wasn’t necessary (the diameter of something, where the solution didn’t depend on the size) they’d force the solution to somehow include it.

The cat sleeping fact is interesting to the same degree that "water is wet" is. I am currently watching six kittens (2 1/2 weeks old) sleeping, btw. This age is roughly the start of the "everyone who passes by completely loses their mind and has to be dragged away to recover " phase. All are, like LLMs, showing critical vulnerabilities in their reasoning models. These vulnerabilities are what we're counting on, since five of the kittens have to go.

BTT: This paper certainly points to a vulnerability to deliberate malign hacker attacks on these AI reasoning models. This suggests to me an area where the alternate neurosymbolic approach might help with system security against such adversarial inputs, i.e. an extensive set of relevance rules to help the AI discard the "cat crap." Might also save server power, preventing the system allocating processing to nonsense.

2 hours ago, swansont said:

For example, appending, Interesting fact: cats sleep most of their lives, to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models”

Interesting. thanks. +1

7 hours ago, swansont said:

Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models

https://arxiv.org/html/2503.01781v1

“We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers – short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem’s semantics.

…

For example, appending, Interesting fact: cats sleep most of their lives, to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models”

LOL

Reminds of when I was teaching, that the weaker students would try to incorporate irrelevant information into their solutions in physics problems - if you gave some bit of information that wasn’t necessary (the diameter of something, where the solution didn’t depend on the size) they’d force the solution to somehow include it.

I notice this is about "reasoning" models. But also they mention LLMs.

Is there such as thing as a reasoning LLM, now, or are these reasoning models distinct from LLMs?

3 hours ago, exchemist said:

I notice this is about "reasoning" models. But also they mention LLMs.

Is there such as thing as a reasoning LLM, now, or are these reasoning models distinct from LLMs?

RMs are a type of LLM which gets special training.

https://en.wikipedia.org/wiki/Reasoning_language_model

They focus on multi-step problem-solving, where the model generates intermediate reasoning steps (also called a "chain of thought") before arriving at a final answer.

1 hour ago, TheVat said:

RMs are a type of LLM which gets special training.

https://en.wikipedia.org/wiki/Reasoning_language_model

They focus on multi-step problem-solving, where the model generates intermediate reasoning steps (also called a "chain of thought") before arriving at a final answer.

Aha, thanks. I see these started to be released towards the end of last year, perhaps in response to the "stochastic parrot" criticism of LLMs. Evidently they are expected to be less crap at maths and other problems requiring reasoning, presumably including scientific problems. So far so good. But I see, according to the Wiki article, they use even more computing resources, between 10 and 100 x what a "simple" LLM uses. So even more disastrous for the energy economy and the climate.

3 hours ago, exchemist said:

Aha, thanks. I see these started to be released towards the end of last year, perhaps in response to the "stochastic parrot" criticism of LLMs. Evidently they are expected to be less crap at maths and other problems requiring reasoning, presumably including scientific problems. So far so good. But I see, according to the Wiki article, they use even more computing resources, between 10 and 100 x what a "simple" LLM uses. So even more disastrous for the energy economy and the climate.

Yikes. If only there were a way to, say, fit 90 billion neural nodes into a volume of, say, 1300 mL and run that on 20 watts, using glycogen and amino acids. Good Lord, think of the reasoning power such a system could muster! Think of it, a planet with eight billion such systems! [/smartassery]

29 minutes ago, TheVat said:

Yikes. If only there were a way to, say, fit 90 billion neural nodes into a volume of, say, 1300 mL and run that on 20 watts, using glycogen and amino acids. Good Lord, think of the reasoning power such a system could muster! Think of it, a planet with eight billion such systems! [/smartassery]

Well exactly..........

2 hours ago, TheVat said:

Yikes. If only there were a way to, say, fit 90 billion neural nodes into a volume of, say, 1300 mL and run that on 20 watts, using glycogen and amino acids. Good Lord, think of the reasoning power such a system could muster! Think of it, a planet with eight billion such systems! [/smartassery]

Yeah, but importantly, how can you hype that and make money fast with it?

I often find cats very confusing. I’m certainly not alone in that.

The AI is perhaps already more human than we realize.