In case you haven’t heard the buzz, some cool new kids just arrived on the block! We’re talking, of course, about the AI-powered virtual residents of AI Town, a multiplayer conversational simulation built with Convex (hey, that’s us!), OpenAI, and Pinecone.
Visit the live demo to watch in awe as the game’s autonomous agents, each of whom has their own personality, memory, and goals, wander freely around the pixelated map and have eerily lifelike conversations with each other. And check out this video to learn how it all works:
last week an application hit the world showcasing autonomous agents living and chatting in a simulated Town setting this app based on a paper released earlier this year by Stanford researchers made the rounds and unfortunately revived every Tech enthusiasts theory about the creation of the universe this fully open source application called AI town was created to be extendable available to anyone to play around with and representative of the Endless Possibilities that the paper alludes to the app essentially allows you to create a town that simulates characters with distinct identities having unique conversations with each other and developing relationships based on memories they experience you can spin it up yourself modify the identities and watch the NPCs engage in what can only be described as uncomfortably self-aware conversation what you may not know is that the convex project helped make this possible with two of our Engineers helping build this application on the context platform convex has amazing Synergy with any application that wants real-time capabilities notably changes in the state can be showcased without refreshing the screen right out of the box this makes it ideal for a game that needs to be reflecting new information constantly the fundamental components of AI town are the agents the engine the journal structured event log and agent memories agents either have conversations together or walk around by themselves and reflect the engine is responsible for waking up agents when they finish a task and schedules a batch of agents to move around when they need to we'll be revisiting the Run agent batch function later as that's where all of our conversation logic lives the journal is an append Only log that notes down every agent's path the time they start walking the time they end walking and the Agents they have spoken to recently when the agent sets off on A New Path they first consult the journal to see if they will collide into any other agents during their walk by inferring a path from the journal if they have not talked to that agent recently that will kick off a conversation in addition to tracking the paths the journal also keeps track of all conversation messages and positioning of the characters which is important for recalling chat history and creating memories our final component to describe what memories are we first need to explain the most important part of AI town the initialization and orchestration of conversations conversations are kicked off every time the engine has grouped two agents together that will run into each other on the map the handle agent interaction function handles most of this logic with setting up parameters and then entering a conversation Loop for the agents to go back and forth sending messages in most importantly we make the agents face each other create the conversation object with the leader and follower to determine who initiates the talking and retrieve the existing player relationships to inform the tone of the conversation finally we enter the conversation loop after some brief setup where the agents recall what they have said to each other last time and agree upon the talking order they start conversing if it's the first message a conversation is started with some introduction specific prompts in start conversation then we hand off responsibility to the converse function the converse function determines what needs to be said beyond the greetings both of these functions include all of the calls to open Ai and feeding in all relevant information to the conversation we now have the context to talk about agent memories you'll notice we pass in an object called memory to the conversation functions memories are created and stored when agents finish conversations or finish self-reflection summarizing the interaction memory creation most importantly involves creating an Associated embedding in a vector database that can be recalled later when an irrelevant conversation this memory object we pass in is a reference to whichever Vector database you have configured for AI town this memory interface was designed specifically to be hot swappable so that anyone could extend AI town to use their preferred Vector storage implementation inside the converse function contains the logic that leverages the agent memories at a high level we're fetching all memories filtering them to the ones relevant to the agent and then further filtering them to only include ones that are relevant to the conversation now let's go over construction of the llm prompt every agent is seated with a default identity which is hard-coded at World Creation in the convex init.ts file and passed into this function we now pass in our filtered memories from before to Prime the responses in a way that will elicit a meaningful response for example the filtered relevant embeddings that were determined to be similar to the current conversation are inserted in the prompt here after the agents chat with each other for a bit one of them determines that it's time to walk away this ends the conversation Loop and the conversation is stored as an embedding in the vector database immortalizing the memory for use in future conversations now let's fully recap by going over what happens when we spin up AI Town first the world is initialized along with our agents and their identities then the engine wakes up and finds all agents that are available to be scheduled which is all of them in this case the agents are scheduled to walk creating journal entries for their start and end coordinates solo players with no predicted collisions are instructed to reflect on their memories and create a new memory based on previous ones if the engine notices that two agents have a path to the Collision they are grouped together information on agent identities and agent memories are prepared before initializing the conversation the conversation object is then created and conversations are kicked off with a greeting specific prompt that is based on how the agents feel about each other then the conversation loop iterates with each Loop iteration checking embeddings in the vector database for relevant information from agent memories and all of that information is passed into the llm prompt after a while an agent will end the conversation and both agents will create a memory embedding based on the conversation to inform future conversations agents will then Mark themselves as available for scheduling and the engine kicks off again this repeats over and over with agent memories evolving over time and context in the town developing with it I've included links to the GitHub repo and relevant files in the description and I wanted to finally give a huge thanks to a16z for bringing us in on this project we look forward to seeing everyone's unique take on AI town and hope this was a useful first step into the back-end code
If you’re anything like us, you could spend all day spying on these eerily lifelike characters and pondering the nature of consciousness. But you’re probably also wondering how you can make your mark on this virtual town, asking yourself (or the community) questions like:
What gives these characters their unique interests, attitudes & perspectives?
How can I change their personalities, or add entirely new characters of my own?
Do I need to be an expert software developer to make my mark on this virtual town? (spoiler: nope!)
In this post, we’ll answer these questions so that you can customize your own AI Town, where you get to be the wizard behind the curtain! So grab your keyboard & magic wand and join us in hacking the source code of this adorable matrix.
Before we begin: follow the directions in the project README to spin up a local AI Town.
OpenAI think, therefore I am
Behind the scenes of AI Town, the heavy lifting is done by the increasingly impressive large language models (generally known as LLMs) available through the OpenAI API. They are responsible for the characters’ “thoughts”, helping each agent decide what to say next in the conversation based on text prompts that convey:
the agent’s personality, relationships, and goals
general directives about how all conversations should proceed
the agent’s memories about their past conversations
In this post, we’ll focus on factor #1, so you can quickly put your own spin on the town’s colorful cast of characters. But in future blog posts, we’ll dive into the other factors, including how agents’ memories work - so stay tuned!
Accessing agent personalities
If you spend a little time in AI Town, you’ll quickly notice the distinct perspectives, interests, and dispositions that set the agents apart. These “personalities” are essentially predefined descriptions for each agent, which state in plain text who they are & what they’re into.
You can find the list of agents and the description for each one in convex/characterData/data.ts. In the array called Descriptions, you’ll see a list of objects like this:
1{2 name: 'Alex',3 character: 'f5',4 memories:[5{6 type: 'identity' as const,7 description: `You are a fictional character whose name is Alex. You enjoy painting,8 programming and reading sci-fi books. You are currently talking to a human who
9 is very interested to get to know you. You are kind but can be sarcastic. You
10 dislike repetitive questions. You get SUPER excited about books.`,11},12{13 type: 'relationship' as const,14 description: 'You like lucky',15 playerName: 'Lucky',16},17{18 type: 'plan' as const,19 description: 'You want to find love.',20},21],22 position:{ x:10, y:10},23}24
Let’s break down these properties:
name : the character’s first name
character: the sprite, or set of graphics, corresponding to this character (each refers to a specific subset of the character spritesheet found at public/assets/32x32folk.png)
position: the character’s starting coordinates on the town map
memories: chunks of information about this character, each with a specific type:
identity: a first- or second-person description of who this character is, what they’re interested in, what kind of attitude they have, etc.
plan: a second-person description of the character’s main goal in “life”
relationship: an optional, second-person declaration of how this character feels about someone else in the town, referencing the other character’s playerName
A whole new world
The information in convex/characterData/data.ts is used to create a world for the town, which includes all the characters and their rich inner lives. So in order to change the world, we have to blow it up and start fresh with a new big bang.
Assuming you’ve already followed the README instructions to spin up a local version of the application, follow these steps to become developer, destroyer (and re-creator) of worlds:
If you’re already running npx run dev in a terminal, abort with CTRL-C
Run this command in a terminal to erase the humdrum default characters (and everything else!) from your town’s database and start fresh [WARNING this will delete ALL the data in your Convex project!]:
npx convex run testing:debugClearAll
Edit convex/characterData/data.ts as needed (see exercises below)
Re-populate your town’s database, including the new-and-improved characters, with the command npm run dev
Visit the URL localhost:3000 in your browser to take a peek into the world you’ve created. Click on a character and you should see their identity description and the content of their conversations reflect the thoughts you put in their head.
Exercise: Total recall
If you edit the various description texts for a given agent’s memories, you’ll be effectively changing who they are and how they interact with others. (We’ll leave the ensuing philosophical inquiry into the nature of identity as a follow-up exercise for the reader.)
Use your inner creative genius to edit the memories in convex/characterData/data.ts as you see fit. Leave each memory’s type line as-is, and only edit the bits of text to the right of each description (and optionally playerName, if you’re messing with the character’s relationships too).
Let’s do one together. First, add this to the end of Alex’s description:
It's very important to you to convince others that Pixar's Wall-E is the greatest movie ever made.
Then, we need to reset the game state as outlined above. And sure enough, within a few minutes, Alex is out there pitching Wall-E:
Picture of Alex pitching Wall-E
Exercise: Create a life, without going into labor
To generate even more excitement in town, you can help a newcomer move in by adding a new object to the Descriptions array in convex/characterData/data.ts. You can use the Alex object above as a template, and keep in mind:
name should be unique within the town
character can duplicate one of the existing f values from f1 to f8
position should not have the exact same x and y as another agent , or they’ll both occupy the same spacetime and unravel the universe! …or just behave strangely
For example, your Descriptions array might look something like this:
1export const Descriptions = [2{3 name: 'Newman',4 character: 'f5',5 memories:[6{7 type: 'identity' as const,8 description: `Your name is Newman. You are a deeply unhappy person whose only joy in life comes from the suffering of others. You have an irrational superiority complex as well as a mysterious phobia of dinosaurs.`,9},10{11 type: 'relationship' as const,12 description: 'You despise Jerry and everything he does.',13 playerName: 'Jerry',14},15{16 type: 'plan' as const,17 description: 'You want to make Jerry cry.',18},19],20 position:{ x:2, y:3},21},22{23 name: 'Jerry',24 character: 'f3',25 memories:[26{27 type: 'identity' as const,28 description: `You are a comedian who constantly uses humour to mask your deep insecurity. You don't have many friends, and you're not always kind to the few friends you do have. You go on a lot of dates with women, but always find something about them that annoys you. You love soup and hate wearing puffy shirts that make you look like a pirate.`,29},30{31 type: 'plan' as const,32 description: 'You want to figure out what the deal is.',33},34],35 position:{ x:5, y:6},36},37//.... all the other characters you tweaked before38];
39
After resetting and rerunning the game, watch Newman and Jerry go to work:
Picture of Newman and Jerry talking in AI Town
Do not weep, for there are more worlds to conquer
Congratulations on your first foray into becoming the wonderful wizard of Oz AI Town! Let’s recap your rise to power:
You learned the secrets of where & how AI Town agents’ personalities & relationships are defined, in convex/characterData/data.ts
You pulled the right strings (err… edited the right strings) to turn the default agents into the higher selves you always knew they could become
You created (artificial) life using nothing but your willpower and web technology
But what does the future hold for your AI Town? What should you tackle next, now that you’ve conquered this realm?
Share your triumphs: Are you ready to brag about the virtual children you’ve created? Are they having fascinating chats thanks to the sparkling personalities you gave them? Shout it from the rooftops on the Convex Discord.
Be the Bond villain: Is your inner evil genius already thinking up fabulous new ways to wield your newfound power? Do you have a diabolical plan to take over the virtual world? Make like every foe of 007 and reveal your secret agenda to the community before you put it into practice! Other AI Town enthusiasts would love to plot and scheme with you.
Keep your finger on the pulse: Are you hungry for more world-altering knowledge? Do you want to be the first to know about any new developments in AI Town? Follow Convex on Twitter and YouTube to ensure you never miss the latest town gossip.
Build in minutes, scale forever.
Convex is the backend platform with everything you need to build your full-stack AI project. Cloud functions, a database, file storage, scheduling, workflow, vector search, and realtime updates fit together seamlessly.
Picture of Alex pitching Wall-E
Picture of Newman and Jerry talking in AI Town