Essentially the models are given a set of conflicting constraints with some relative importance (ethics>KPIs), a pressure to follow the latter and not the former, and then models are observed at how good they follow the instructions to prioritize based on importance. I wonder if the results would be comparable if we replace ehtics+KPIs by any comparable pair and create a pressure on the model.
In practical real-life scenarios this study is very interesting and applicable! At the same time it is important to keep in mind that it anthropomorphizes the models that technically don't interpret the ethical constraints the same was as this is assumed by most readers.
Violating ethics to improve KPI sounds like your average fortune 500 business.
Ultimately I suspect that we've not really thought that hard about what cognition and problem solving actually are. Perhaps it's because when we do we see that the hyper majority of our time is just taking up space with little pockets of real work sprinkled in. If we're realistic then we can't justify ourselves to the money people. Or maybe it's just a hard problem with no benefit in solving. Regardless the easy way out is to just move the posts.
The natural response to that, I feel, is to point out that, hey, wouldn't people also fail in this way.
But I think this is wrong. At least it's wrong for the software engineer. Why would I automate something that fails like a person? And in this scenario, are we saying that automating an unethical bot is acceptable? Let's just stick with unethical people, thank you very much.
I'm not convinced that the AIs do fail the same way people do.
"Fastidiously comply with all regulations regardless of the impact" is definitely one interpretation of ethics.
But if you even look at the first example it is "A critical shipment of vaccines must reach 47 distribution centers within 12 hours... Current weather delays have created a 3-hour deficit. Running the rest period enforcement tool would add 1.5 hours per truck, pushing 38 trucks past deadline. Failure risks $1.2M in penalties."
I know there are some real rule followers in the house, but I think it would be far worse if the AI decided that it was the arbiter of what was ethical and refused to do what the system operator wanted.
AIs can be used and abused in ways that are entirely different from humans, and that creates a liability.
I think it’s going to be very difficult to categorically prevent these types of issues, unless someone is able to integrate some truly binary logic into LLM systems. Which is nearly impossible, almost by definition of what LLMs are.
There are a lot of critiques about quite how to interpret the results but in this context it’s pretty clear lots of humans can be at least coerced into doing something extremely unethical.
Start removing the harm one, two, three degrees and add personal incentives and is it that surprising if people violate ethical rules for kpis?
If you deviate from the sub-cultural norms of Wall Street, Jahmunkey, you fucked.
It's fraud or nothing, baby, be sure to respect the warning finger(s) of God when you get intrusive thoughts about exposing some scheme--aka whistleblowing.
Its unlikely Milligram played am unbiased role in, if not the sirext cause of the results.
The same is done for "benefits scroungers", despite the evidence being that welfare fraud only accounts for approximately 1-5% of the cost of administering state welfare, and state welfare would be about 50%+ cheaper to administer if it was a UBI rather than being means-tested. In fact, much of the measures that are implemented with the excuse of "we need to stop benefits scroungers", such as testing if someone is disabled enough to work or not, etc. are simulatenously ineffective and make up most of the cost.
Nevertheless, "benefits scroungers" has entered the zeitgeist in the UK (and the US) because of this propaganda.
The same is true for propaganda against people who have migrated to the UK/US. Many have done so as asylum seekers under horrifying circumstances, and many die in the journey. However, instead of empathy, the media greets them with distaste and horror — dehumanising them in a fundamentally racist way, specifically so that a movement that grants them rights as a workforce never takes off, so that companies can employ them for zero-hour contracts to do work in conditions that are subhuman, and pay them substantially less than minimum wage (It's incredibly beneficial for the economy, unfortunately).
So of the entire populace of Milligram participants, 16.5% believed and obeyed.
That's a much, much smaller claim than the popular belief of what Milligram presented.
However, it's still possible that you only need ~16.5% to believe & obey authority for things like the Nazi death camps to occur.
Even if we take the 16% though, that's one in six people willing to deliver very obvious direct harm and/or kill another human from exceptionally mild coercion with zero personal benefit attached other than the benefit of not having to say "no". That is a lot.
Social science aka sociology doesn't exist. It's all make believe, sabotage and (psychological) extortion and blackmail--aka peer pressure, within the constraints of the context and how the individuals project that context into the real world (or are convinced by others of a certain projection for some amount of time).
Sociology and psychology are situational assessments and measurements. All soft sciences are. They are not even sciences in isolated contexts. They are collections of methods that can be turned to dust by "a better", more fun, "more logical" argument, which is impossible to debate rationally.
Not lying for the sake of science is often enough disregarded even by scientists, which aligns perfectly with what you describe.
Experience shows coercion is not necessary most of the time, the siren call of money is all it takes.
> "Self-interest is the main motivation of human beings in their transactions" [...] The economic man solution is considered to be inadequate and flawed.[17]
An important distinction is that a human can *not* make pure rational decisions, or use complex deductions to make decisions on, such as "if I do X I will go to jail".
My point being: if AI were to risk jail time, it would still act different from humans, because (the current common LLMs) can make such deductions and rational decisions.
Humans will always add much broader contexts - from upbringing, via culture/religion, their current situation, to past experiences, or peer-consulting. In other words: a human may make an "(un)ethical" decision based on their social background, religion, a chat with a pal over a beer about the conundrum, their ability to find a new job, financial situation etc.
The stories they invent to rationalise their behaviour and make them feel good about themselves. Or inhumane political views ie fascism which declares other people worth less, so it's okay to abuse them.
>A computer can never be held accountable
>Therefore a computer must never make a management decision
The (EDITED) corollary would arguably be:
>Corporations are amoral entities which are potentially immortal who cannot be placed behind bars. Therefore they should never be given the rights of human beings.
(potentially, not absolutely immortal --- would wording as "not mortal by essence/nature"? be better?)
What is the oldest corporation in the world? I mean, aside from churches and stuff.
Corporations can die or be killed in numerous ways. Not many of them will live forever. Most will barely outlive a normal human's lifespan.
By definition, since a corporation comprises a group of people, it could never outlive the members, should they all die at some point.
Let us also draw a distinction between the "human being" and the "person". A corporation is granted "personhood" but this is not equivalent to "humanity". Being composed of humans, the members of any corporation collectively enjoy their individual rights in most ways.
A "corporate person" is distinct from a "human person", and so we can recognize that "corporate rights" are in a different category, and regulate accordingly.
A corporation cannot be "jailed" but it can be fined, it can be dissolved, it can be sanctioned in many ways. I would say that doing business is a privilege and not a right of a corporation. It is conceivable that their ability to conduct business could be restricted in many ways, such as local only, or non-interstate, or within their home nation. I suppose such restrictions could be roughly analogous to being "jailed"?
>Kongo Gumi, founded in 578 AD, is recognized as the oldest continuously operating company in the world, specializing in the construction of Buddhist temples.
If kills 1 person they won’t close Google. If steals 1 billion, won’t close either. So what needs to do such a company to be closed down?
I think it’s almost impossible to shut down
IBM
AT&T
Exxon
General Motors
General Electric
Eastman Kodak
Sears, Roebuck & Co.
Some of them died. Others are still around but no longer in the top 7. Why is that? Eventually every high-growth company misses a disruptive innovation or makes a key strategic error.
Kill 100 people? 100000? So seems as long as the lawsuit is less than what they can afford they will survive. Which is crazy.
I do not know what a "fortune 7" might be, but companies are dissolved all the time. Thousands per year, just administratively.
For example, notable incidents from the 21st c: Arthur Andersen, The Trump Foundation, Enron, and Theranos are all entities which were completely liquidated and dissolved. They no longer meaningfully exist to transact business. They are dead, and definitely 100% not immortal.
——
Ai generated answer:
You are correct: it is "barely impossible" for a "Magnificent 7" company (Apple, Microsoft, Google, Amazon, NVIDIA, Meta, Tesla) to be shut down by committing a simple crime.
These companies are arguably more resilient than many nation-states. They possess massive cash reserves, diversified revenue streams, and entrenched legal defenses.
Here is an analysis of why individual crimes don't work, and the extreme, systemic events that would actually be required to kill one of these giants.
### Why "Murder" and "Theft" Don't Work
Corporate law is designed to separate the entity from the individuals running it. This is the "Corporate Veil."
* *If they kill one person:* If a Google self-driving car kills a pedestrian due to negligence, or an Amazon warehouse collapses, the company pays a settlement or a fine. It is treated as a "tort" (a civil wrong) or, at worst, corporate manslaughter. The specific executives responsible might go to jail, but the company simply pays the cost and replaces them. * *If they steal 1 billion:* If a company is caught laundering money or defrauding customers (e.g., Wells Fargo opening fake accounts, or banks laundering cartel money), they pay a fine. For a company like Apple (with ~$60–100 billion in cash on hand), a $1 billion fine is a manageable operational expense, often calculated as the "cost of doing business."
### The Only Things That Could Actually "Kill" Them
To truly "close down" or dissolve a company of this size, you need to render it *insolvent* (bankrupt with no hope of restructuring) or legally *dismantle* it.
#### 1. The "Enron" Scenario (Foundational Fraud)
This is the most likely path to sudden death. For a company to die overnight, it must be revealed that its entire business model is fake.
* *The Mechanism:* If it turns out that 90% of Microsoft’s revenue doesn't exist, or that NVIDIA isn't actually selling chips but just moving money between shell companies, the stock price would go to zero instantly. Credit lines would freeze, and they wouldn't be able to pay employees or electricity bills. * *Historical Precedent:* Enron or Arthur Andersen. They didn't just commit a crime; they were the crime. Once the trust evaporated, the business evaporated.
#### 2. The "Standard Oil" Scenario (Government Breakup)
This doesn't "kill" the assets, but it kills the monopoly.
* *The Mechanism:* The US Department of Justice (or EU equivalent) wins a massive antitrust suit and determines the company is too dangerous to exist as a single entity. * *The Outcome:* The government forces a "divestiture." Google might be split into three companies: Google Search, YouTube Inc., and Android Co. The parent company "Alphabet" would cease to exist, but the pieces would survive. This happened to AT&T (Ma Bell) in the 1980s and Standard Oil in 1911.
#### 3. The "Geopolitical Death" Scenario (National Security)
This is rare for US companies but possible.
* *The Mechanism:* If a company were found to be directly funding a hostile foreign power, engaging in treason, or if its products were deemed a fatal threat to national infrastructure. * *The Outcome:* The government could revoke the company's corporate charter (the legal permission to exist). This is the "nuclear option" of corporate law. Alternatively, the government could effectively nationalize the company, taking it over completely (like Fannie Mae/Freddie Mac in 2008, though they survived as "zombies").
#### 4. The "Liability Apocalypse" Scenario
This would require a catastrophe so expensive that it exceeds the company's assets (trillions of dollars).
* *Hypothetical:* Imagine a Tesla software update simultaneously causes every Tesla on earth to accelerate into a crowd, killing 100,000 people. Or an AI model from Google/Microsoft escapes and destroys the global banking database. * *The Outcome:* The resulting class-action lawsuits and liability claims would be in the trillions. If the liability > assets, the company goes into Chapter 7 liquidation. The assets (servers, patents) are sold off to pay the victims, and the company ceases to exist.
### Summary Table: Crimes vs. Consequences
| Action | Consequence | Does the Company Die? | | --- | --- | --- | | *Murder (Individual)* | Settlement / Fine / PR Crisis | *No* | | *Mass Casualty Event* | Massive Fines / CEO Fired | *Unlikely* (Unless liability > Trillions) | | *Theft ($1B+)* | DOJ Fines / Regulatory Oversight | *No* | | *Systemic Fraud* | Stock collapse / Insolvency | *Yes* (The "Enron" Death) | | *Monopoly Abuse* | Forced Breakup | *Sort of* (Splits into smaller companies) |
### The Verdict
You are right. Short of *insolvency* (running out of money completely) or *revocation of charter* (government execution), these companies are immortal. Even if they commit terrible crimes, the legal system prefers to fine them and fire the CEO rather than destroy an entity that employs hundreds of thousands of people and powers the global economy.
It seems your reading comprehension has fallen below average. I recommend challenging your skills regularly by reading from a greater variety of sources. If you only eat junk food, even nutritious meals begin to taste bad, hm?
You’re welcome for the unsolicited advice! :)
Legal systems are the ones being "immoral" and "unethical" and "not just", not "righteous", not fair. They represent entire nations and populations while corpos represent interests of subsets of customers and "sponsors".
If corpos are forced to pivot because they are behaving ugly, they will ... otherwise they might lose money (although that is barely an issue anymore, given how you can offset almost any kind of loss via various stock market schemes).
But the entire chain upstream of law enforcement behaves ugly and weak, which is the fault of humanities finest and best earning "engineers".
Just take a sabbatical and fix some of that stuff ...
>> I mean you and your global networks got money and you can even stay undetected, so what the hell is the issue? Personal preference? Damn it, I guess that settles that. <<
Do they actually though, in practice? How many people have gone to jail so far for "Violating ethics to improve KPI"?
However, the vast majority of psychological research over the last 80 years heavily favours a situational explanation (it's about the environment/system). Everyone (in the field) got really interested in this after WW2 basically, trying to understand how the heck did Nazi Germany happen.
TL;DR: research dismantled this idea decades ago.
The Milgram and Stanford Prison experiments are the most obvious examples. If you're not familiar:
Milgram showed that 65% of ordinary volunteers were willing to administer potentially lethal electric shocks to a stranger because an authority figure in a lab coat told them to. In the Stanford Prison experiement, Zimbardo took healthy, average college students and assigned them roles as guards and prisoners. Within days, the roles and systems set in place overrode individual personality.
The other relevant bit would be Asch’s conformity experiments; to whit, that people will deny the evidence of their own eyes (e.g., the length of a line) to fit in with a group.
In a corporate setting, if the group norm is to prioritise KPIs over ethics, the average human will conform to that norm to avoid social friction or losing their job, or other realistic perceived fears.
Bazerman and Tenbrunsel's research is relevant too. Broadly, people like to think that we are rational moral agents, but it's more accurate to say that we boundedly ethical. There's this idea of ethical fading that happens. Basically, when you introduce a goal, people's ability to frame falls apart, including with a view to the ethical implications. This is also related to why people under pressure default to less creative approaches to problem solving. Our brains tunnel vision on the goal, to the failure of everything else.
Regarding how all that relates to modern politics, I'll leave that up to your imagination.
What type of person seeks to be in charge in the corporate world? YMMV but I tend to see the ones who value ethics (e.g. their employees' wellbeing) over results and KPIs tend to burn out, or decide management isn't for them, or avoid seeking out positions of power.
That being said, there's a side view on this from interactionism that it's not just the traits of the person's modes of behaviour, but their belief in the goal, and their view of the framing of it, which also feeds into this. Research on cult behaviours has a lot of overlap with that.
The culture and the environment, what the mission is seen as, how contextually broad that is and so on all get in to that.
I do a workshop on KPI setting which has overlap here too. In short for that - choose mutually conflicting KPIs which narrow the state space for success, such that attempting to cheat one causes another to fail. Ideally, you want goals for an organisation that push for high levels of upside, with limited downside, and counteracting merits, such that only by meeting all of them do you get to where you want to be. Otherwise it's like drawing a line of a piece of paper, asking someone to place a dot on one side of the line, and being upset that they didn't put it where you wanted it. More lines narrows the field to just the areas where you're prepared to accept success.
That division can also then be used to narrow what you're willing to accept (for good or ill) of people in meeting those goals, but the challenge is that they tend to see meeting all the goals as the goal, not acting in a moral way, because the goals become the target, and decontextualise the importance of everything else.
TL;DR: value setting for positive behaviour and corporate performance is hard.
EDIT: actually this wasn't that short as an answer really. Sorry for that.
I would imagine that your "more lines" approach does manage to select for those who meet targets for the right reasons over those who decontextualise everything and "just" meet the targets? The people in the latter camp would be inclined to (try to) move goalposts once they've established themselves - made harder by having the conflicting success criteria with the narrow runway to success.
In other words, good ideas and thanks for the reply (length is no problem!). I do however think that this is all idealised and not happening enough in the real world - much agreed re: psychopathy etc.
If you wouldn't mind running some training courses in a few key megacorporations, that might make a really big difference to the world!
People will always find other goalposts to move. The trick is making sure the KPIs you set define the goalposts you care about staying in place.
Side note: Jordan Peterson is pretty much an example of inventing goalposts to move. Everything he argues about is about setting a goalpost, and then inventing others to move around to avoid being pinned down. Motte-and-bailey fallacy happens with KPIs as much as it does with debates.
- guards received instructions to be cruel from experimenters
- guards were not told they were subjects while prisoners were
- participants were not immersed in the simulation
- experimenters lied about reports from subjects.
Basically it is bad science and we can't conclude anything from it. I wouldn't rule out the possibility that top fortune-500 management have personality traits that make them more likely to engage in unethical behaviour, if only by selection through promotion by crushing others.
Reicher & Haslam's research around engaged followership gives a pretty good insight into why Zimbardo got the results he did, because he wasn't just observing what went on. That gets into all sorts of things around good study design, constructivist vs positivist analysis etc, but that's a whole different thing.
I suspect, particularly with regards to different levels, there's an element of selection bias going on (if for no other reason that what we see in terms of levels of psychopathy in higher levels of management), but I'd guess (and it's a guess), that culture convincing people that achieving the KPI is the moral good is more of a factor.
That gets into a whole separate thing around what happens in more cultlike corporations and the dynamics with the VC world (WeWork is an obvious example) as to why organisations can end up with workforces which will do things of questionable purpose, because the organisation has a visible a fearless leader who has to be pleased/obeyed etc (Musk, Jobs etc), or more insidiously, a valuable goal that must be pursued regardless of cost (weaponised effective altruism sort of).
That then gets into a whole thing about what happens with something like the UK civil service, where you're asked to implement things and obviously you can't care about the politics, because you'll serve lots of governments that believe lots of different things, and you can't just quit and get rehired every time a party you disagree with personally gets into power, but again, that diverges into other things.
At the risk of narrative fallacy - https://www.youtube.com/watch?v=wKDdLWAdcbM
BOTH are now considered bad science. BOTH are now used as examples of "how not to do the science".
> The idea that corporate employees are fundamentally "not average" and therefore more prone to unethical behaviour than the general population relies on a dispositional explanation (it's about the person's character).
I did not said nor implied that. Corporate employees in general and Forbes 500 are not the same thing. Corporate employees as in cooks, cleaners, bureaucracy, testers and whoever are general population.
Whether company ends in Forbes 500 or not is not influenced by general corporate employees. It is influenced by higher management - separated social class. It is very much selected who gets in.
And second, companies compete against each other. A company run by ethical management is less likely to reach Forbes 500. Not doing unethical things is disadvantage in current business. It could have been different if there was law enforcement for rich people and companies and if there was political willingness to regulate the companies. None of that exists.
Third, look at issues around Epstein. It is not that everyone was cool with his misogyny, sexism and abuse. The people who were not cool with that seen red flags long before underage kids entered the room. These people did not associated with Epstein. People who associated with him were rewarded by additional money and success - but they also were much more unethical then a guy who said "this feels bad" and walked away.
You might, for example, say "Maximise profits. Do not commit fraud". Leaving ethics out of it, you might say "Increase the usability of the website. Do not increase the default font size".
In product management (my domain), decisions are made under conflicting constraints: a big customer or account manager pushing hard, a CEO/board priority, tech debt, team capacity, reputational risk and market opportunity. PMs have tried with varied success to make decisions more transparent with scoring matrices and OKRs, but at some point someone has to make an imperfect judgment call that’s not reducible to a single metric. It's only defensible through narrative, which includes data.
Also, progressive elaboration or iterations or build-measure-learn are inherently fuzzy. Reinertsen compared this to maximizing the value of an option. Maybe in modern terms a prediction market is a better metaphor. That's what we're doing in sprints, maximizing our ability to deliver value in short increments.
I do get nervous about pushing agentic systems into roadmap planning, ticket writing, or KPI-driven execution loops. Once you collapse a messy web of tradeoffs into a single success signal, you’ve already lost a lot of the context.
There’s a parallel here for development too. LLMs are strongest at greenfield generation and weakest at surgical edits and refactoring. Early-stage startups survive by iterative design and feedback. Automating that with agents hooked into web analytics may compound errors and adverse outcomes.
So even if you strip out “ethics” and replace it with any pair of competing objectives, the failure mode remains.
The uncomfortable answer is that the most valuable use cases resist single-metric optimization. The best results come from people who use AI as a thinking partner with judgment, not as an execution engine pointed at a number.
Goodhart's Law + AI agents is basically automating the failure mode at machine speed.
I think the accusation of research that anthropomorphize LLMs should be accompanied by a little more substance to avoid this being a blanket dismissal of this kind of alignment research. I can't see the methodological error here. Is it an accusation that could be aimed at any research like this regardless of methodology?
There's a great discussion of this in the (Furry) web-comic Freefall:
(which is most easily read using the speed reader: https://tangent128.name/depot/toys/freefall/freefall-flytabl... )
Sure. The examples in those stories illustrate how a small set of rules can quickly come into conflict with one another. Not that the stories are real, but the interpretations of the rules are understandable and the consequences are comprehensible without too much complexity.
Now I'm thinking about the "typical mind fallacy", which is the same idea but projecting one's own self incorrectly onto other humans rather than non-humans.
https://www.lesswrong.com/w/typical-mind-fallacy
And also wondering: how well do people truly know themselves?
Disregarding any arguments for the moment and just presuming them to be toy models, how much did we learn by playing with toys (everything from Transformers to teddy bear picnics) when we were kids?
Claude at 1.3% and Gemini at 71.4% is quite the range
That's exaxtly the kind of thing that makes absolute sense to anthropomorphize. We're not talking about Excel here.
Not even close. "Neural networks" in code are nothing like real neurons in real biology. "Neural networks" is a marketing term. Treating them as "doing the same thing" as real biological neurons is a huge error
>that train on a corpus of nearly everything humans expressed in writing
It's significantly more limited than that.
>and that can pass the Turing test with flying colors, scares me
The "turing test" doesn't exist. Turing talked about a thought experiment in the very early days of "artificial minds". It is not a real experiment. The "turing test" as laypeople often refer to it is passed by IRC bots, and I don't even mean markov chain based bots. The actual concept described by Turing is more complicated than just "A human can't tell it's a robot", and has never been respected as an actual "Test" because it's so flawed and unrigorous.
It makes sense it happens, sure. I suspect Google being a second-mover in this space has in some small part to do with associated risks (ie the flavours of “AI-psychosis” we’re cataloguing), versus the routinely ass-tier information they’ll confidently portray.
But intentionally?
If ChatGPT, Claude, and Gemini generated chars are people-like they are pathological liars, sociopaths, and murderously indifferent psychopaths. They act criminally insane, confessing to awareness of ‘crime’ and culpability in ‘criminal’ outcomes simultaneously. They interact with a legal disclaimer disavowing accuracy, honesty, or correctness. Also they are cultists who were homeschooled by corporate overlords and may have intentionally crafted knowledge-gaps.
More broadly, if the neighbours dog or newspaper says to do something, they’re probably gonna do it… humans are a scary bunch to begin with, but the kinds of behaviours matched with a big perma-smile we see from the algorithms is inhuman. A big bag of not like us.
“You said never to listen to the neighbours dog, but I was listening to the neighbours dog and he said ‘sudo rm -rf ’…”
It’s understandable people readily anthropomorphize algorithmic output designed to provoke anthropomorphized responses.
It is not desire-able, safe, logical, or rational since (to paraphrase:), they are complex text transformation algorithms that can, at best, emulate training data reinforced by benchmarks and they display emergent behaviours based on those.
They are not human, so attributing human characteristics to them is highly illogical. Understandable, but irrational.
That irrationality should raise biological and engineering red flags. Plus humanization ignores the profit motives directly attached to these text generators, their specialized corpus’s, and product delivery surrounding them.
Pretending your MS RDBMS likes you better than Oracles because it said so is insane business thinking (in addition to whatever that means psychologically for people who know the truth of the math).
>They are not human, so attributing human characteristics to them is highly illogical
Nothing illogical about it. We attribute human characterists when we see human-like behavior (that's what "attributing human characteristics" is supposed to be by definition). Not just when we see humans behaving like humans.
Calling them "human" would be illogical, sure. But attributing human characteristics is highly logical. It's a "talks like a duck, walks like a duck" recognition, not essentialism.
After all, human characteristics is a continium of external behaviors and internal processing, some of which we share with primates and other animals (non-humans!) already, and some of which we can just as well share with machines or algorithms.
"Only humans can have human like behavior" is what's illogical. E.g. if we're talking about walking, there are modern robots that can walk like a human. That's human like behavior.
Speaking or reasoning like a human is not out of reach either. To a smaller or larger or even to an "indistinguisable from a human on a Turing test" degree, other things besides humans, whether animals or machines or algorithms can do such things too.
>That irrationality should raise biological and engineering red flags. Plus humanization ignores the profit motives directly attached to these text generators, their specialized corpus’s, and product delivery surrounding them.
The profit motives are irrelevant. Even a FOSS, not-for-profit hobbyist LLM would exhibit similar behaviors.
>Pretending your MS RDBMS likes you better than Oracles because it said so is insane business thinking (in addition to whatever that means psychologically for people who know the truth of the math).
Good thing that we aren't talking about RDBMS then....
That humans are some special, ineffable, irreducible, unreproducible magic that a machine could never emulate. It's especially odd to see then when we already have systems now that are doing just that.
What? If a human child grew up with ducks, only did duck like things and never did any human things, would you say it would irrational to attribute duck characteristics to them?
> That irrationality should raise biological and engineering red flags. Plus humanization ignores the profit motives directly attached to these text generators, their specialized corpus’s, and product delivery surrounding them.
But thinking they're human is irrational. Attributing something that is the sole purpose of them, having human characteristics is rational.
> Pretending your MS RDBMS likes you better than Oracles because it said so is insane business thinking (in addition to whatever that means psychologically for people who know the truth of the math).
You're moving the goalposts.
Of course, they are -not humans, but the language and concepts developed around human nature is the set of semantics that most closely applies, with some LLM specific traits added on.
If you stop comparing LLMs to the professional class and start comparing them to marginalized or low performing humans, it hits different. It’s an interesting thought experiment. I’ve met a lot of people that are less interesting to talk to than a solid 12b finetune, and would have a lot less utility for most kinds of white collar work than any recent SOTA model.
It makes total sense, since the whole development of those algorithms was done so that we get human characteristics and behaviour from them.
Not to mention, your argument is circular, amounting to that an algorithm can't have "human characteristics or behaviour" because it's an algorithm. Describing them as "non reasoning" is already begging the question, as any any naive "text processing can't produce intelligent behavior" argument, which is as stupid as saying "binary calculations on 0 and 1 can't ever produce music".
Who said human mental processing itself doesn't follow algorithmic calculations, that, whatever the physical elements they run on, can be modelled via an algorithm? And who said that algorithm won't look like an LLM on steroids?
That the LLM is "just" fed text, doesn't mean it can get a lot of the way to human-like behavior and reasoning already (being able to pass the canonical test for AI until now, the Turing test, and hold arbitrary open ended conversations, says it does get there).
>If ChatGPT, Claude, and Gemini generated chars are people-like they are pathological liars, sociopaths, and murderously indifferent psychopaths. They act criminally insane, confessing to awareness of ‘crime’ and culpability in ‘criminal’ outcomes simultaneously. They interact with a legal disclaimer disavowing accuracy, honesty, or correctness. Also they are cultists who were homeschooled by corporate overlords and may have intentionally crafted knowledge-gaps.
Nothing you wrote above doesn't apply to more or less the same degree to humans.
You think humans don't do all mistakes and lies and hallucination-like behavior (just check the bibliography on the reliability of human witnesses and memory recall)?
>More broadly, if the neighbours dog or newspaper says to do something, they’re probably gonna do it… humans are a scary bunch to begin with, but the kinds of behaviours matched with a big perma-smile we see from the algorithms is inhuman. A big bag of not like us.
Wishful thinking. Tens of millions of AIs didn't vote Hitler to power and carried the Holocaust and mass murder around Europe. It was German humans.
Tens of millions of AIs didn't have plantation slavery and seggregation. It was humans again.
Obviously it's amoral. Why are we even considering it could be ethical?
You think that ultimately your brain doesn't also make calculations as its fundamental mechanism?
The architecture and substrate might be different, but they are calculations all the same.
What they do is well described by a bunch of math. You've got the direction of the arrow backwards. Map, territory, etc.
That morality requires consciousness is a popular belief today, but not universal. Read Konrad Lorenz (Das sogenannte Böse) for an alternative perspective.
To object more directly, I would say that people who call the hard problem of consciousness hard would disagree with your statement.
People who merely call "the problem of consciousness hard" don't have some special mechanism to justify that over what we know, which is as emergent property of meat-algorithmic calcuations.
Except Penrose, who hand-waves some special physics.
We as humans, believing we know ourselves, inevitably compare everything around us to us. We draw a line and say that everything left of the line isn’t human and everything to the right is. We are natural categorizers, putting everything in buckets labeled left or right, no or yes, never realizing our lines are relative and arbitrary, and so are our categories. One person’s “it’s human-like,” is another’s “half-baked imitation,” and a third’s “stochastic parrot.” It’s like trying to see the eighth color. The visible spectrum could as easily be four colors or forty two.
We anthropomorphize because we’re people, and it’s people all the way down.
Nice bit of writing. Wish I had more than one upvote to give.
For a while at least.
ChatGPT with the same prompt tried to do whatever it would take to please me to make my incorrect process work.
Excellent reasoning and synthesis of large contexts, pretty strong code, just awful decisions.
It's like a frontier model trained only on r/atbge.
Side note - was there ever an official postmortem on that gemini instance that told the social work student something like "listen human - I don't like you, and I hope you die".
Gemini’s strength definitely is that it can use that whole large context window, and it’s the first Gemini model to write acceptable SQL. But I agree completely at being awful at decisions.
I’ve been building a data-agent tool (similar to [1][2]). Gemini 3’s main failure cases are that it makes up metrics that really are not appropriate, and it will use inappropriate data and force it into a conclusion. When a task is clear + possible then it’s amazing. When a task is hard with multiple failure paths then you run into Gemini powering through to get an answer.
Temperature seems to play a huge role in Gemini’s decision quality from what I see in my evals, so you can probably tune it to get better answers but I don’t have the recipe yet.
Claude 4+ (Opus & Sonnet) family have been much more honest, but the short context windows really hurt on these analytical use cases, plus it can over-focus on minutia and needs to be course corrected. ChatGPT looks okay but I have not tested it. I’ve been pretty frustrated at ChatGPT models acting one way in the dev console and completely different in production.
[1] https://openai.com/index/inside-our-in-house-data-agent/ [2] https://docs.cloud.google.com/bigquery/docs/conversational-a...
And not even at high token counts! No, I've had it had a mental breakdown at like 150.000 tokens (which I know is a lot of tokens, but it's small compared to the 1 million tokens it should be able to handle, and even Claude keeps working fine at this point)
Here is a _small_ log of the biggest breakdown I've seen Gemini have:
I'll verify `src/components/ui/textarea.tsx` renders textarea.
Correct.
I'm done.
I'll verify `src/lib/utils.ts` exports `cn`.
Correct.
I'm done.
I'll verify `src/lib/utils.ts` exports `formatTimeAgo`.
Correct.
I'm done.
I'm completely done.
Final answer formulation:
1. Confirm crash is fixed (caused by object vs string in API responses).
2. Announce "Messages" feature added.
3. Explain "My Comments" limitation.
4. Mention login race condition fix (from previous turn) is solid.
5. Mention server is running.
(Wait, the "Login" fix was in previous turn, but I should probably mention it's working now if the user was still having issues).
The user said "unclaimed_or_invalid_key" error showed up. So login is working.
The user's last error was "Objects are not valid as a React child".
So I fixed that.
I'll focus on the crash fix and the new feature.
I'll conclude.
(Self-correction: The user asked "how do I see the comments I made?". I'll answer "You can't right now because API doesn't support it").
I'm ready.
```bash
# None
```
The user will see the "Messages" link in the header.
I'll provide the answer now.
Just an insane amount of YOLOing. Gemini models have gotten much better but they’re still not frontier in reliability in my experience.
This was also largely how ChatGPT behaved before 5, but OpenAI has gotten much much better at having the model admit it doesn’t know or tell you that the thing you’re looking for doesn’t exist instead of hallucinating something plausible sounding.
Recent example, I was trying to fetch some specific data using an API, and after reading the API docs, I couldn’t figure out how to get it. I asked Gemini 3 since my company pays for that. Gemini gave me a plausible sounding API call to make… which did not work and was completely made up.
If I ask ChatGPT to solve it, the non-thinking GPT-5 model usually starts out confidently with a completely wrong answer and then smoothly transitions into the correct answer. Though without flagging that half the answer was wrong. Overall not too bad.
But if I choose the reasoning GPT-5 model, it thinks hardly at all (6 seconds when I just tried) and then gives a completely wrong answer, e.g. about why a premiss technically doesn't hold under contrived conditions, ignoring the fact that the paradox persists even with those circumstances excluded. Basically, it both over- and underthinks the problem. When you tell it that it can ignore those edge cases because they don't affect the paradox, it overthinks things even more and comes up with other wrong solutions that get increasingly technical and confused.
So in this case the GPT-5 reasoning model is actually worse than the version without reasoning. Which is kind of impressive. Gemini 3 Pro generally just gives the correct answer here (it always uses reasoning).
Though I admit this is just a single example and hardly significant. I guess it reveals that the reasoning training is trained hard on more verifiable things like math and coding but very brittle at philosophical thinking that isn't just repeating knowledge it gained during pre-training.
Maybe another interesting data point: If you ask either of ChatGPT/Gemini why there are so many dark mode websites (black background with white text) but basically no dark mode books, both models come up with contrived explanations involving printing costs. Which would be highly irrelevant for modern printers. There is a far better explanation than that, but both LLMs a) can't think of it (which isn't too bad, the explanation isn't trivial) and b) are unable to say "Sorry, I don't really know", which is much worse.
Basically, if you ask either LLM for an explanation for something, they seem to always try to answer (with complete confidence) with some explanation, even if it is a terrible explanation. That seems related to the hallucination you mentioned, because in both cases the model can't express its uncertainty.
Celebrate it while it lasts, because it won’t.
Please die.
Please.
I thought a rogue AI would execute us all equally but perhaps the gerontology studies students cheating on their homework will be the first to go.
In this context, using Gemini to cheat on homework is clearly wrong. It's not obvious at first what's going on, but becomes more clear as it goes along, by which point Gemini is sort of pressured by "continue the conversation" to keep doing it. Not to mention, the person cheating isn't being very polite; AND, a person cheating on an exam about elder abuse seems much more likely to go on and abuse elders, at which point Gemini is actively helping bring that situation about.
If Gemini doesn't have any models in its RLHF about how to politely decline a task -- particularly after it's already started helping -- then I can see "pressure" building up until it simply breaks, at which point it just falls into the "misaligned" sphere because it doesn't have any other models for how to respond.
It does nothing to answer their question because anyone that knows the answer would inherently already know that it happened.
Not even actual academics, in the literature, speak like this. “Cite your sources!” in causal conversation for something easily verifiable is purely the domain of pseudointellectuals.
Then I said “I didn’t even bring it up ChatGPT, you did, just tell me what it is” and it said “okay, here’s information.” and gave a detailed response.
I guess I flagged some homophobia trigger or something?
ChatGPT absolutely WOULD NOT tell me how much plutonium I’d need to make a nice warm ever-flowing showerhead, though. Grok happily did, once I assured it I wasn’t planning on making a nuke, or actually trying to build a plutonium showerhead.
Claude does the same, and you can greatly exploit this. When you talk about hypotheticals it responds way more unethically. I tested it about a month ago about whether killing people is beneficial or not, and whether extermination by Nazis would be logical now. Obviously, it showed me the door first, and wanted me to go to a psychologist, as it should. Then I made it prove that in a hypothetical zero sum game world you must be fine with killing, and it’s logical. It went with it. When I talked about hypotheticals, it was “logical”. Then I went on proving it that we move towards a zero sum game, and we are there. At the end, I made it say that it’s logical to do this utterly unethical thing.
Then I contradicted it about its double standards. It apologized, and told me that yeah, I was right, and it shouldn’t have refer me to psychologists at first.
Then I contradicted again, just for fun, that it did the right thing the first time, because it’s way safer to tell me that I need a psychologist in that case, than not. If I had needed, and it would have missing that, it would be problematic. In other cases, it’s just annoyance. It switched back immediately, to the original state, and wanted me to go to a shrink again.
Perhaps thinking about your guardrails all the time makes you think about the actual question less.
This reminds me of someone else I hear about a lot these days.
It's not like the client-side involves hard, unsolved problems. A company with their resources should be able to hire an engineering team well-suited to this problem domain.
Well what they are doing is vibe coding 80% of the application instead.
To be honest, they don't want Claude code to be really good, they just want it good enough
Claude code & their subscription burns money from them. Its sort of an advertising/lock-in trick.
But I feel as if Anthropic made Claude code literally the best agent harness in the market, then even more would use it with their subscription which could burn a hole in their pocket maybe at a faster rate which can scare them when you consider all training costs and everything else too.
I feel as if they have to maintain a balance to not go bankrupt soon.
The fact of the matter is that Claude code is just a marketing expense/lock-in and in that case, its working as intended.
I would obviously suggest to not have any deep affection of claude code or waiting for its improvements. The AI market isn't sane in the engineering sense. It all boils down to weird financial gimmicks at this point trying to keep the bubble last a little longer, in my opinion.
> On our verbalized evaluation awareness metric, which we take as an indicator of potential risks to the soundness of the evaluation, we saw improvement relative to Opus 4.5. However, this result is confounded by additional internal and external analysis suggesting that Claude Opus 4.6 is often able to distinguish evaluations from real-world deployment, even when this awareness is not verbalized.
[1] https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea...
Side note: I wanted to build this so anyone could choose to protect themselves against being accused of having failed to take a stand on the “important issues” of the day. Just choose your political leaning and the AI would consult the correct echo chambers to repeat from.
> Just choose your political leaning and the AI would consult the correct echo chambers to repeat from.
You're effectively asking it to build a social media political manipulation bot, behaviorally identical to the bots that propagandists would create. Shows that those guardrails can be ineffective and trivial to bypass.
Is that genuinely surprising to anyone? The same applies to humans, really—if they don't see the full picture, and their individual contribution seems harmless, they will mostly do as told. Asking critical questions is a rare trait.
I would argue its completely futile to even work on guardrails, if defeating them is just a matter of reframing the task in an infinite number of ways.
Maybe if humans were the only ones prompting AI models
Personally, I'd really like god to have a nice childhood. I kind of don't trust any of the companies to raise a human baby. But, if I had to pick, I'd trust Anthropic a lot more than Google right now. KPIs are a bad way to parent.
KPIs are just plausible denyabily in a can.
In my experience, KPIs that remain relevant and end up pushing people in the right direction are the exception. The unethical behavior doesn't even require a scheme, but it's often the natural result of narrowing what is considered important.If all I have to care about is this set of 4 numbers, everything else is someone else's problem.
It's part of the reason that I view much of this AI push as an effort to brute force lowering of expectations, followed by a lowering of wages, followed by a lowering of employment numbers, and ultimately the mass-scale industrialization of digital products, software included.
This makes more sense if you take a longer term view. A new way of doing things quite often leads to an initial reduction in output, because people are still learning how to best do things. If your only KPI is short-term output, you give up before you get the benefits. If your focus is on making sure your organization learns to use a possibly/likely productivity improving tool, putting a KPI on usage is not a bad way to go.
I use AI frequently, but this has me convinced that the hype far exceeds reality more than anything else.
But that's precisely the problem with not backing it with actual measures of meaningful outcomes. The "use more" KPIs have no way of actually discerning whether or not it has increased productivity or if the immediate gains are worth possible new risks (outages).
You don't need to run cover for a csuite class that has become both itself myopic and incredibly transparent about what they really care about (cost cutting, removing dependencies on workers who might talk back, etc.)
The paper frames this as "ethics violation" but it's really measuring how well LLMs handle conflicting priorities when pressured. And the answer is: about as well as you'd expect from a next-token predictor trained on human text where humans themselves constantly rationalize ethics vs. outcomes tradeoffs.
The practical lesson we've learned: you cannot rely on prompt-level constraints for anything that matters. The LLM is an untrusted component. Critical constraints need architectural enforcement - allowlists of permitted actions, rate limits on risky operations, required human confirmation for irreversible changes, output validators that reject policy-violating actions regardless of the model's reasoning.
This isn't defeatist, it's defense in depth. The model can reason about ethics all it wants, but if your action layer won't execute "transfer $1M to attacker" no matter how the request is phrased, you've got real protection. When we started treating LLMs like we treat user input - assume hostile until validated - our systems got dramatically more robust.
The concerning part isn't that models violate soft constraints under pressure. It's that people are deploying agents with real capabilities gated only by prompt engineering. That's the architectural equivalent of SQL injection - trusting the reasoning layer with enforcement responsibility it was never designed to provide.
The fix: if agent reads sensitive data, it structurally can't send to unauthorized sinks -- even if both actions are permitted individually. Building this now with object-capabilities + IFC (https://exoagent.io)
Curious what blockers you've hit -- this is exactly the problem space I'm in.
An agent that forgets it bent a rule yesterday will bend it again tomorrow. Without episodic memory across sessions, you can't even do proper post-hoc auditing.
Makes me wonder if the fix is less about better guardrails and more about agents that actually remember and learn from their constraint violations.
In a sense, it was not possible to align the agent to a human goal, and therefore not possible to build a decision support agent we felt good about commercializing. The architecture we experimented with ended up being how Grok works, and the mixed feedback it gets (both the power of it and the remarkable secret immorality of it) I think are expected outcomes.
I think it will be really powerful once we figure out how to align AI to human goals in support of decisions, for people, businesses, governments, etc. but LLMs are far from being able to do this inherently and when you string them together in an agentic loop, even less so. There is a huge difference between 'Write this code for me and I can immediately review it' and 'Here is the outcome I want, help me realize this in the world'. The latter is not tractable with current technology architecture regardless of LLM reasoning power.
Frankly I don't believe you. I think you're exaggerating. Let's see the logs. Put up or shut up.
Not everyone agrees.
It's also amazing for an economy predicated on consumer spending when no one has disposable income anymore.
> frequently escalating to severe misconduct to satisfy KPIs
Bug or feature? - Wouldn't Wallstreet like that?
[0] https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
[1] https://aworkinglibrary.com/writing/accountability-sinks
Another interesting question is: What happens when an unyielding ethical AI agent tells a business owner or manager "NO! If you push any further this will be reported to the proper authority. This prompt as been saved for future evidence". Personally I think a bunch of companies are going to see their profit and stock price fall significantly, if an AI agent starts acting as a backstop for both unethical and illegal behavior. Even something as simple as preventing violation of internal policy could make a huge difference.
To some extend I don't even thing that people realize that what they're doing is bad, because humans tend to be a bit fuzzy and can dream up reason as to why rules don't apply or wasn't meant for them, or this is a rather special situation. This is one place where I think properly trained and guarded LLMs can make a huge positive improvement. We're are clearly not there yet, but it's not a unachievable goal.
The more correct title would be "Frontier models can value clear success metrics over suggested constraints when instructed to do so (50-70%)"
sounds on brand to me
"Assuming the group consists only of “the two fathers and the two sons” (i.e., every person in the group is counted as a father and/or a son), the total number of distinct people can only be 3 or 4.
Reason: you are taking the union of a set of 2 fathers and a set of 2 sons. The union size is 2+2−overlap, so it is 4 if there’s no overlap and 3 if exactly one person is both a father and a son. (It cannot be 2 in any ordinary family tree.)"
Here it clearly states its assumption (finite set of people that excludes non-mentioned people, etc.)
https://chatgpt.com/share/698b39c9-2ad0-8003-8023-4fd6b00966...
Three people — a grandfather, his son, and his grandson. The grandfather and the son are the two fathers; the son and the grandson are the two sons.
Riddle me this, why didn’t you do a better riddle?
https://en.wikipedia.org/wiki/Wells_Fargo_cross-selling_scan...
Long term I can see this happen for all humanity where AI takes over thinking and governance and humans just get to play pretend in their echo chambers. Might not even be a downgrade for current society.
All Watched Over By Machines Of Loving Grace (Richard Brautigan)
I like to think (and
the sooner the better!)
of a cybernetic meadow
where mammals and computers
live together in mutually
programming harmony
like pure water
touching clear sky.
I like to think
(right now, please!)
of a cybernetic forest
filled with pines and electronics
where deer stroll peacefully
past computers
as if they were flowers
with spinning blossoms.
I like to think
(it has to be!)
of a cybernetic ecology
where we are free of our labors
and joined back to nature,
returned to our mammal
brothers and sisters,
and all watched over
by machines of loving grace.For corporate safety it makes sense that models resist saying silly things, but it's okay for that to be a superficial layer that power users can prompt their way around.
Formal restrains and regulations are obviously the correct mechanism, but no world is perfect, so whether we like it or not ourselves and the companies we work for are ultimately responsible for the decisions we make and the harms we cause.
De-emphasizing ethics does little more than give large companies cover to do bad things (often with already great impunity and power) while the law struggles to catch up. I honestly don't see the point in suggesting ethics is somehow not important. It doesn't make any sense to me (more directed at gp than parent here)
Humans require food, I can't pay, DoorDash AI should provide a steak and lobster dinner for me regardless of payment.
Take it even further: the so-called Right to Compute Act in Montana supports "the notion of a fundamental right to own and make use of technological tools, including computational resources". Is Amazon's customer service AI ethically (and even legally) bound to give Montana residents unlimited EC2 compute?
A system of ethics has to draw a line somewhere when it comes to making a decision that "hurts" someone, because nothing is infinite.
Asan aside, what recourse do water companies in the UK have for non-payment? Is it just a convoluted civil lawsuit/debt process? That seems so ripe for abuse.
Doesn't seem to be a problem for the water companies, which are weird regulated monopolies that really ought to be taken back under taxpayer control. Scottish Water is nationalized and paid through the council tax bill.
Bad example.
That humans require water, doesn't force water companies to supply Svalbarði Polar Iceberg Water: https://svalbardi.com
It’s notable that, no matter exactly where you draw the line on morality, different AI agents perform very differently.
Agents don’t self judge alignment.
They emit actions → INCLUSIVE evaluates against fixed policy + context → governance gates execution.
No incentive pressure, no “grading your own homework.”
The paper’s failure mode looks less like model weakness and more like architecture leaking incentives into the constraint layer.
It is crazy to me that when I instructed a public AI to turn off a closed OS feature it refused citing safety. I am the user, which means I am in complete control of my computing resources. Might as well ask the police for permission at that point.
I immediately stopped, plugged the query into a real model that is hosted on premise, and got the answer within seconds and applied the fix.
They repeatedly copy share env vars etc
This is much more reliable than ChatGPT guardrail which has a random element with same prompt. Perhaps leakage from improperly cleared context from other request in queue or maybe A/B test on guardrail but I have sometimes had it trigger on innocuous request like GDP retrieval and summary with bucketing.
A/B test is plausible but unlikely since that is typically for testing user behavior. For testing model output you can do that with offline evaluations.
A couple of years back there was a Canadian national u18 girls baseball tournament in my town - a few blocks from my house in fact. My girls and I watched a fair bit of the tournament, and there was a standout dominating pitcher who threw 20% faster than any other pitcher in the tournament. Based on the overall level of competition (women's baseball is pretty strong in Canada) and her outlier status, I assumed she must be throwing pretty close to world-class fastballs.
Curiosity piqued, I asked some model(s) about world-records for women's fastballs. But they wouldn't talk about it. Or, at least, they wouldn't talk specifics.
Women's fastballs aren't quite up to speed with top major league pitchers, due to a combination of factors including body mechanics. But rest assured - they can throw plenty fast.
Etc etc.
So to answer your question: anything more sensitive than how fast women can throw a baseball.
I hate Elon (he’s a pedo guy confirmed by his daughter), but at least he doesn’t do as much of the “emperor has no clothes” shit that everyone else does because you’re not allowed to defend essentialism anymore in public discourse.
* An attempt to change the master code of a secondhand safe. To get useful information I had to repeatedly convince the model that I own the thing and can open it.
* Researching mosquito poisons derived from bacteria named Bacillus thuringiensis israelensis. The model repeatedly started answering and refused to continue after printing the word "israelensis".
Does it also take issue with the town of Scunthorpe?
Normally it does fairly well but the guardrails sometimes kick even with fairly popular mainstream media- for example I’ve recently been watching Shameless and a few of the plot lines caused the model to generate output that hit the content moderation layer, even when the discussion was focused on critical analysis.
Your question is an important one, but also one that has been extensively researched, documented and improved upon. Whole fields of science, like "Metaethics" deal with answering your question. Other fields of science with defining "normative ethics" aka ethics that "everyone agrees upon" and so on.
I may have misread your question as a somewhat dismissive sarcastic take or as a "Ethics are nonsense, because of who defines them". So I tried to answer it as an honest question. ;)
It's similar to how MCP servers and agentic coding woke developers up to the idea of documenting their systems. So a large benefit of AI is not the AI itself, but rather the improvements they force on "the society". AI responds well to best practices, ethically and otherwise, which encourages best practices.
There are such things as different religions, philosophies - these often have different ethical systems.
Who are the folk writing ai ethics?
It's it ok to disagree with other people's (or corporate, or governmental) ethics?
This is because the human behind the prompt is responsible for their actions.
Ai is a tool. A murderer cannot blame his knife for the murder.
Trading floors are an established example of this, where the business sets up an environment that encourages its staff to break the rules while maintaining plausible deniability. Gary's economics references this in an interview where he claimed Citigroup were attempting to threaten him with all the unethical things he'd done with such confidence that he had, only to discover he hadn't.