It feels plausible that within the next few years LLMs will be powering NPCs/enemies in AAA video games. On a technical side, I think we will be gated by what a PS5/Xbox Series X can process locally for this generation. On the gameplay side, I think this could open up a lot. The main loop of summarizing experiences and then feeding that summary into the next prompt can unlock characters/enemies that are much more dynamic. Here are two examples of gameplay elements I think would benefit from it.
Resident Evil 2 Remake was famous for having a strong enemy AI for Mr. X. He stalked the protagonist around a three story building. Players had to flee from him, and when they have successfully lost him then they had to quietly sneak around the building to avoid detection as he searched for the player. I imagine being able to learn from past encounters would make him even more frightening to run from. A stalking AI could take into account hiding places it found the player in the past, tactics they used to flee, and where the player's next objective is when deciding it's plan of action.
There's another genre of games this might find itself useful in. Those games in which you interact with a village on a social level. Stardew Valley, and Majora's Mask come to mind. Having more dynamic interactions with the townsfolk, that impact future interactions, could help draw in users to the simulated-social aspect of these games.
Procedural generation has been around since the dawn of computer games: Rogue, one of the earliest games, randomly generates a dungeon each time you enter it.
Despite that, procedural generation is still quite rare in shipping games outside of a few niche genres. I think the biggest problem is one of control. A huge part of the process of making and shipping a game is balancing it and testing it to ensure the play experience never goes off the rails.
Even relatively simple procedural generation can make that very difficult. Imagine playing a Zelda-like game where it turns out that 0.0001% of the time, the item you need to make progress is stuck behind a wall where the player can't reach it. Worse, they won't discover this until many hours into the game. That kind of stuff keeps game designers and producers up at night.
They would rather a less varied, hand-authored gameplay experience, if the result is one that they have more control over and understand better.
Bolting an LLM onto your game for NPC dialog sounds really cool until a popular Twitch streamer is playing your game for an audience of millions and some random NPC spouts a racist slur.
What I do think will be very common is game designers using LLMs offline to generate dialog and other assets, and then after the designer has vetted them, putting them into the game as fixed authored content. That kind of procedural generation is used all the time and has been for decades.
> Imagine playing a Zelda-like game where it turns out that 0.0001% of the time, the item you need to make progress is stuck behind a wall where the player can't reach it.
The very concept of "an item you need to make progress" is an artifact of a non-procedurally generated, railroaded plot. And even if your procedurally-generated game includes such things, it's a solvable problem to make this basically never happen.
I think your RE style AI suggestion can be done in the traditional sense of AI in game development; that is, without machine learning involved at all. That's all possible with goal oriented action planning, and similar techniques. Game developers often find though, that making the AI too smart can actually make players dislike playing, or putting completely fair but very smart AI in can make the players report that they feel the AI is cheating or unfair. So it gets dumbed down.
The main issue I see with generative AI and games is that it's all good to be able to chat to an NPC as if it were a person with a personality and knowledge of the world. There's an issue of fidelity though; how can you ensure that the AI only reports things that are true about the game world? And then, the issue of actual behaviour: an LLM might generate speech that has the NPC you're talking to say that they're going off to the inn at 5pm to recruit some sellswords, then march to the den of a fire dragon to defeat it, stopping along the way to collect an ice sword from its guardian maiden who lives in a giant tree nearby. OK, it's to generate that text from the player's prompts, it's very difficult to then actually have the NPCs act out the things that the LLM has just said it would do, tying in pathfinding, scheduling, animation, group behaviour and so on, and carrying out those actions would probably involve more traditional game AI techniques (again, not machine learning) anyway. Maybe that'll be solved some other way, maybe this repo does something like that already, I didn't check.
I'm reminded about how people still talk about how the original F.E.A.R. has the best AI in video games [1], while the actual behavior in a technical sense is quite simple [2] and is fundamentally designed around the player being time-pressured but still understanding what's going on. If you just plug an LLM into everything you lose that intentionality around players understanding the system without actually needing to be a subject matter expert on whatever the enemy should be 'realistically' doing.
You also have to consider how AI techniques fit into the context of a running game. ML techniques haven't really caught on with existing games because they completely tank performance, in a program that has to make computations every frame, in addition to either completely murdering the player or acting weird/being hard to debug or just not being that compelling.
What I've seen of current experiments with integrating LLMs into NPC conversations (I believe someone even got it integrated with a Skyrim mod, iic) is that to skirt around this issue, they just make HTTP requests to OpenAI or whatever. Game runs smoothly, but there's still a weird lag around NPC responses that makes it feel stilted and unnatural, and obviously relies on having a stable internet connection at all times.
It will be exciting to see what people could do with something like a fine tuned local Llama2 model. I don't think the way gpt is set up is very conducive to game systems unfortunately.
I agree with this, I think it definitely has potential along these lines. Open world sandbox games that can play out in different variations from a starting cast of characters with different personalities and motivations.
Resident Evil 2 Remake was famous for having a strong enemy AI for Mr. X. He stalked the protagonist around a three story building. Players had to flee from him, and when they have successfully lost him then they had to quietly sneak around the building to avoid detection as he searched for the player. I imagine being able to learn from past encounters would make him even more frightening to run from. A stalking AI could take into account hiding places it found the player in the past, tactics they used to flee, and where the player's next objective is when deciding it's plan of action.
There's another genre of games this might find itself useful in. Those games in which you interact with a village on a social level. Stardew Valley, and Majora's Mask come to mind. Having more dynamic interactions with the townsfolk, that impact future interactions, could help draw in users to the simulated-social aspect of these games.