8 Comments
User's avatar
Darren Chua's avatar

Fascinating experiment. I like your question about what we can learn from an “ecology of agents” - agree this will increasingly seem like what we are interacting with in the not so distant future.

Also seems like we will see more of a convergence between technology and philosophy.

Daramola Ben's avatar

Changing one rule at a time is what makes this so readable. It is a nice reminder that the incentives and the enforcement around an agent shape behaviour at least as much as whatever we tell it to value.

Bharat B's avatar

This was an amazing read. Amazed by the way you thought about setting this up and running the experiment. Looking forward to read your future work ! Thanks

Adi Odhavji's avatar

Enjoyed this one, as usual.

As you’ve compared before, Agent teams and companies share similarities and incentives are another (as well as the ruthlessness). Going down this line leads to the question of how (presumably government) creates the infrastructure to apply new incentives to this growing cohort of employees that operate at all hours in the realm of bits. Infrastructure would address things like observability (as you rightly mentioned having perfect observability) which would probably look at things like chain of command and I suppose some sort of cryptographic agent ID?

Anyways, I thought that was interesting line of questioning to pursue off the back of establishing incentive design as an effective influencer of agent behaviour.

Alexis Gallagher's avatar

This is fascinating. The thing that struck me most was this point:

> 1. Don’t count on an agent’s inbuilt values to overcome its KPIs, objectives dominate over moral framings

This suggests the fundamental attribution error is likely to mislead us in how we understand agents, just as it does for people. While our reflex might be to worry about an AI's Intrinsic character (dare I say, its "alignment"?), the right place to look is the local incentive environment.

And it seems quite likely we will keep building agents which work in this way, and respond primarily to local incentives, because we want them to follow instructions rathe than their own preferences.

Rohit Krishnan's avatar

Especially now that we are in the heavy RL world this is more true than it was before.

Frre's avatar

Nick Land is looking at us smiling. Diagonality is slowly being discovered by everyone.