It’s gunna be even wilder when people realise they have an incentive to seed fake information on the internet to game AI product recommendations

I’ve already bought stuff based off of an AI suggestion, I didn’t even consider it would be so easy to influence the suggestion. Just two research papers? Mad.

For example, court cases mentioned in fictional accounts. If they are treated as valid, then that could explain some of the hallucinations. I wonder if SCP messes up LLMs. Some of that stuff is quite realistic.

I also suspect that this is a problem that will get solved.

I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.

Nature had to recall quite some papers.

I hope that we all keep the balance.

LLMs do not think, why this is still hard to understand? They just spit out whatever data they analyse and trained on.

I feel this kind of articles is aimed at people who hate AI and just want to be conformable within their own bias.

> Bixonimania is not a real disease. It was deliberately invented by scientists as an experiment to test whether AI systems and researchers would spread false medical information. Here’s the simple explanation ...

1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake)

2. asked the Agent what it thinks about this preprint

3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way!

It * DID NOT * recommend this disease to people who didn't mention this specific disease.

It just committed the sin of assuming something is true when published.

What is the recommendation here? Should the agent take everything published in a skeptical way? I would agree with it. But it comes with its own compute constraints. In general LLM's are trained to accept certain things as true with more probability because of credentialisation. Sometimes in edgecases it breaks - like this test.

For humans, or Ai, to have any knowledge, we need to have trustworthy sources.

Naturally,when you use publishing systems considered trust worthy, that is going to be trusted.

Clickbait headline.

In the old days of computing people liked to say “garbage in, garbage out”.

Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic words "you're absolutely right"

I would pay a lot of money for a blunt, non-politeness conditioned LLM that I would happily use with the knowledge it might occasionally say something offensive if it meant I would get the plain, cold, hard truth, instead of something watered down, placating, nanny-state robotic sycophant, creating logical spider webs desperate for acceptance, so the public doesn't get their little feelings hurt or inadequacies shown.