Anjana Vakil

a year ago

Building AI Town's Backend: Character Identities

In case you haven’t heard the buzz, some cool new kids just arrived on the block! We’re talking, of course, about the AI-powered virtual residents of AI Town, a multiplayer conversational simulation built with Convex (hey, that’s us!), OpenAI, and Pinecone.

Visit the live demo to watch in awe as the game’s autonomous agents, each of whom has their own personality, memory, and goals, wander freely around the pixelated map and have eerily lifelike conversations with each other. And check out this video to learn how it all works:

If you’re anything like us, you could spend all day spying on these eerily lifelike characters and pondering the nature of consciousness. But you’re probably also wondering how you can make your mark on this virtual town, asking yourself (or the community) questions like:

• What gives these characters their unique interests, attitudes & perspectives?
• How can I change their personalities, or add entirely new characters of my own?
• Do I need to be an expert software developer to make my mark on this virtual town? (spoiler: nope!)

In this post, we’ll answer these questions so that you can customize your own AI Town, where you get to be the wizard behind the curtain! So grab your keyboard & magic wand and join us in hacking the source code of this adorable matrix.

Before we begin: follow the directions in the project README to spin up a local AI Town.

OpenAI think, therefore I am

Behind the scenes of AI Town, the heavy lifting is done by the increasingly impressive large language models (generally known as LLMs) available through the OpenAI API. They are responsible for the characters’ “thoughts”, helping each agent decide what to say next in the conversation based on text prompts that convey:

1. the agent’s personality, relationships, and goals
2. general directives about how all conversations should proceed
3. the agent’s memories about their past conversations

In this post, we’ll focus on factor #1, so you can quickly put your own spin on the town’s colorful cast of characters. But in future blog posts, we’ll dive into the other factors, including how agents’ memories work - so stay tuned!

Accessing agent personalities

If you spend a little time in AI Town, you’ll quickly notice the distinct perspectives, interests, and dispositions that set the agents apart. These “personalities” are essentially predefined descriptions for each agent, which state in plain text who they are & what they’re into.

You can find the list of agents and the description for each one in convex/characterData/data.ts. In the array called Descriptions, you’ll see a list of objects like this:

{
    name: 'Alex',
    character: 'f5',
    memories: [
      {
        type: 'identity' as const,
        description: `You are a fictional character whose name is Alex.  You enjoy painting,
      programming and reading sci-fi books.  You are currently talking to a human who
      is very interested to get to know you. You are kind but can be sarcastic. You
      dislike repetitive questions. You get SUPER excited about books.`,
      },
      {
        type: 'relationship' as const,
        description: 'You like lucky',
        playerName: 'Lucky',
      },
      {
        type: 'plan' as const,
        description: 'You want to find love.',
      },
    ],
    position: { x: 10, y: 10 },
 }

Let’s break down these properties:

• name : the character’s first name
• character: the sprite, or set of graphics, corresponding to this character (each refers to a specific subset of the character spritesheet found at public/assets/32x32folk.png)
• position: the character’s starting coordinates on the town map
• memories: chunks of information about this character, each with a specific type:
  • identity: a first- or second-person description of who this character is, what they’re interested in, what kind of attitude they have, etc.
  • plan: a second-person description of the character’s main goal in “life”
  • relationship: an optional, second-person declaration of how this character feels about someone else in the town, referencing the other character’s playerName

A whole new world

The information in convex/characterData/data.ts is used to create a world for the town, which includes all the characters and their rich inner lives. So in order to change the world, we have to blow it up and start fresh with a new big bang.

Assuming you’ve already followed the README instructions to spin up a local version of the application, follow these steps to become developer, destroyer (and re-creator) of worlds:

1. If you’re already running npx run dev in a terminal, abort with CTRL-C
2. Run this command in a terminal to erase the humdrum default characters (and everything else!) from your town’s database and start fresh [WARNING this will delete ALL the data in your Convex project!]: npx convex run testing:debugClearAll
3. Edit convex/characterData/data.ts as needed (see exercises below)
4. Re-populate your town’s database, including the new-and-improved characters, with the command npm run dev
5. Visit the URL localhost:3000 in your browser to take a peek into the world you’ve created. Click on a character and you should see their identity description and the content of their conversations reflect the thoughts you put in their head.

Exercise: Total recall

If you edit the various description texts for a given agent’s memories, you’ll be effectively changing who they are and how they interact with others. (We’ll leave the ensuing philosophical inquiry into the nature of identity as a follow-up exercise for the reader.)

Use your inner creative genius to edit the memories in convex/characterData/data.ts as you see fit. Leave each memory’s type line as-is, and only edit the bits of text to the right of each description (and optionally playerName, if you’re messing with the character’s relationships too).

Let’s do one together. First, add this to the end of Alex’s description:

It's very important to you to convince others that Pixar's Wall-E is the greatest movie ever made.

Then, we need to reset the game state as outlined above. And sure enough, within a few minutes, Alex is out there pitching Wall-E:

Picture of Alex pitching Wall-E

Exercise: Create a life, without going into labor

To generate even more excitement in town, you can help a newcomer move in by adding a new object to the Descriptions array in convex/characterData/data.ts. You can use the Alex object above as a template, and keep in mind:

• name should be unique within the town
• character can duplicate one of the existing f values from f1 to f8
• position should not have the exact same x and y as another agent , or they’ll both occupy the same spacetime and unravel the universe! …or just behave strangely

For example, your Descriptions array might look something like this:

export const Descriptions = [
  {
    name: 'Newman',
    character: 'f5',
    memories: [
      {
        type: 'identity' as const,
        description: `Your name is Newman. You are a deeply unhappy person whose only joy in life comes from the suffering of others. You have an irrational superiority complex as well as a mysterious phobia of dinosaurs.`,
      },
      {
        type: 'relationship' as const,
        description: 'You despise Jerry and everything he does.',
        playerName: 'Jerry',
      },
      {
        type: 'plan' as const,
        description: 'You want to make Jerry cry.',
      },
    ],
    position: { x: 2, y: 3 },
  },
  {
    name: 'Jerry',
    character: 'f3',
    memories: [
      {
        type: 'identity' as const,
        description: `You are a comedian who constantly uses humour to mask your deep insecurity. You don't have many friends, and you're not always kind to the few friends you do have. You go on a lot of dates with women, but always find something about them that annoys you. You love soup and hate wearing puffy shirts that make you look like a pirate.`,
      },
      {
        type: 'plan' as const,
        description: 'You want to figure out what the deal is.',
      },
    ],
    position: { x: 5, y: 6 },
  },
//.... all the other characters you tweaked before
];

After resetting and rerunning the game, watch Newman and Jerry go to work:

Picture of Newman and Jerry talking in AI Town

Do not weep, for there are more worlds to conquer

Congratulations on your first foray into becoming the wonderful wizard of Oz AI Town! Let’s recap your rise to power:

• You learned the secrets of where & how AI Town agents’ personalities & relationships are defined, in convex/characterData/data.ts
• You pulled the right strings (err… edited the right strings) to turn the default agents into the higher selves you always knew they could become
• You created (artificial) life using nothing but your willpower and web technology

But what does the future hold for your AI Town? What should you tackle next, now that you’ve conquered this realm?

Share your triumphs: Are you ready to brag about the virtual children you’ve created? Are they having fascinating chats thanks to the sparkling personalities you gave them? Shout it from the rooftops on the Convex Discord.

Be the Bond villain: Is your inner evil genius already thinking up fabulous new ways to wield your newfound power? Do you have a diabolical plan to take over the virtual world? Make like every foe of 007 and reveal your secret agenda to the community before you put it into practice! Other AI Town enthusiasts would love to plot and scheme with you.

Keep your finger on the pulse: Are you hungry for more world-altering knowledge? Do you want to be the first to know about any new developments in AI Town? Follow Convex on Twitter and YouTube to ensure you never miss the latest town gossip.