Recent Posts

Going Multi-Node with Elixir

Posted on 17 May 2018

From the last update of ExVenture, ExVenture can now be configured to run in a multiple node configuration. In this post I'll show you the basics of how I did that.

What is ExVenture?

ExVenture is a multiplayer text-based game, a Multi-User Dungeon (MUD), server.

You can see more information about ExVenture at exventure.org and GitHub. You can see it running on MidMUD. There is also a public Trello board.

Starting point

ExVenture was heavily geared towards running as a single node to start out. I used Registry very heavily, which only works for a single node. I also have a few local cache processes that out of the gate wouldn't work well when spanning multiple nodes.

My first thoughts where about splitting up the app into an umbrella app and figuring out how to boot the web on one node, the telnet connection on another, etc. I started talking about this in the MUD Coders Guild, and I got a question of "why?"

This was a great question and got me thinking about what else I could do.

I eventually settled on trying to get the same application booting on all nodes and a leader node starting the processes that can only exist once in the cluster, e.g. the world.

Clustering

First up was connecting up nodes in an automated fashion. Since my app was heavily geared towards being a single node, this should be fine. Each node wouldn't talk to each other and the entire world would be spun up on each node.

This was extremely simple with libcluster. It was as simple as installing the hex package and adding this to my configuration files:

config :libcluster,
  topologies: [
    local: [
      strategy: Cluster.Strategy.Epmd,
      config: [hosts: [:"world1@host", :"world2@host"]]
    ]
  ]

Then when starting my app I switched to booting with iex to get the sname flag available.

iex --sname world1 -S mix
iex --sname world2 -S mix

Picking a Leader

Once the world was clustered, I started on picking a leader. I had heard about Raft before, but never really looked into it.

If you are interested in clustering at all, I'd highly recommend giving the paper a read. It is very simple to follow along and understand. Which was the main point of creating Raft, a simple to understand consensus algorithm.

For ExVenture, I went with implementing my own Raft module because I only wanted the leader election part of Raft. I don't need (at least yet!) the rest of Raft.

You can see that in the Raft module. There is a lot to this module that doesn't need to be covered here, but it boils down to the group picks a leader and that leader calls the subscriptions that care about who was leader.

A leader is picked

Once a leader is picked, the Game.World.Master process uses pg2 to find all of the other Game.World.Master processes and sees what zones are alive in the cluster.

After finding out what nodes are alive it spins up the zones not online across the cluster, using a simple rebalacing algorithm.

Global process registry

I also switched to using the global process registry as part of this. I started looking at swarm but it seemed to be something different than what I was looking for.

I will most likely end up changing this in the future but switching {:via, Registry, {Game.NPC, id}} to {:global, {Game.NPC, id}} was an extremely simple change that worked. So I went with it and haven't looked back.

Messages spanning the cluster

The game was now officially multi-node and you could play on either iex servers and see other players and NPCs on the other one.

There was only one final step and that was setting up pg2 groups for each of my caches and my communication layer.

This is very simple and can be done as follows in the init function and a slight change to casting to your GenServers.

@key :items

def init(_) do
  :ok = :pg2.create(@key)
  :ok = :pg2.join(@key, self())

  #...
end

def insert(item) do
  members = :pg2.get_members(@key)

  Enum.map(members, fn member ->
    GenServer.call(member, {:insert, item})
  end)
end

Next Steps

With this up and running I was able to get MidMUD in a multi-node (3 world servers) set up for production. It has been working out pretty well so far and only once I found a large bug (of not being able to communicate across nodes in channels.)

I will post more updates as I continue enhancing the distributed nature of ExVenture.

You can see everything described here in these two pull requests, #37 and #39.

ExVenture Updates for April 2018

Posted on 25 Apr 2018

The last month of ExVenture had a lot of updates to clustering and some extra world details.

The documentation website is exventure.org. You can see the latest additions here on MidMUD, my running instance of ExVenture. There is also a public Trello board now.

Also check out The MUD Coder's Guild, it's a slack team devoted to developing MUDs.

Distributed Erlang

The biggest new feature of ExVenture in the last month is the improved support for erlang node clustering. I started with adding libcluster to join nodes together. Then I had the world start spanning the cluster via a really simple leader election.

Next up was using pg2 to have the player registry and send cache updates across the cluster. The last step was using :global as the registration mechanism for world processes. I tried out swarm for this but it had some weird properties of restarting all of the processes when a single one died.

You can see this step in PR #37.

Raft

The most recent step was implementing the leader election of Raft. It's mostly done, but it should handle rejoining nodes and adding nodes to the cluster. Once a leader is picked, I have a set of modules that get called on the winner node. On start up this will spin out the world across the cluster.

You can see this step in PR #39.

Damage Types

Early on in the month I added custom damage types and damage resistances. Each damage type has an opposing stat that reduces the damage. This step also added an "echo back" of the damage that was actually applied. A character calculates the effects to send over, sends them, and hears back what actually happened.

You can see this in PR #27 and PR #28.

Custom Colors

This is easier to show in pictures than text.

Custom colors in the admin

Customize your colors

You can see this in PR #29 and PR #31.

Listening

A new command was added, listen. This lets you tune into noises that are in the same room as you. This feature has lots of room to grow as more things make sound.

You can see this in PR #33.

NPC Status Engine

NPCs can change their status line and listen text via emotes now. As part of this they set a status key to eventually enable gating of events they do. Such as they only start combat if they are in a certain "mood" and can get out of it later on.

PR #34 has this.

Smaller Tweaks

  • Admin panel tweaks for events
  • Refactoring events

Next Month

For next month I want to continue with the distributed part of ExVenture. I definitely need to get the world rebalacing as nodes fallout and re-join the cluster. I would like to continue with the other game additions from this month. Listening and the npc status engine are both very good foundations for a lot of extra features in the future.

Once I have more work in the clusting of ExVenture I want to do a deep dive blog post.

Looking at ExVenture's Supervision Tree

Posted on 11 Apr 2018

I was watching The Hitchhiker's Guide to the Unexpected (YouTube link) by Fred Hebert and in that there is a neat exercise of writing out your supervision tree on a whiteboard and seeing how things would fail. With this you could better determine what happens to your application as things go wrong.

I decided this would be a good exercise to do on ExVenture. This is a fairly long post that goes through the full supervision tree for ExVenture.

You can see ExVenture in action on MidMUD.

Supervision Tree

ExVenture Supervision Tree

This is the supervision tree that ExVenture ships with now. There are roughly 3 levels in the photo.

First Level

This is the top level directly underneath the application. It contains, in start up order:

  • Data.Repo - the Ecto repo
  • Web.Supervisor - the Phoenix supervisor
  • Game.Registries - a collection of Registrys
  • Game.Supervisor - a the top level supervisor of the game
  • A ranch listener is also started at this level, but it spins off into the ranch application

At this level the supervision strategy is rest_for_one. This is fine because if the Repo dies the rest of the app should be rebooted, something went wrong. As we'll find later on the loads process with an ID to fetch from the database to ensure a clean state is fetched on process restarts (if something crashes.)

Second Level - Web.Supervisor

This supervisor is mostly sitting on top of the Phoenix Endpoint along with a few process monitors for the TelnetChannel and a Cachex cache. It is handled by a one_for_one strategy. This is fine as none of them are really connected to the other, this supervision level is mostly to break sections up for my benefit.

Second Level - Game.Supervisor

This supervisor contains the "world" along with supporting processes. In start up order:

  • Game.Config - an agent that caches game configuration
  • Game.Caches - a supervisor of Cachex caches along with GenServer processes that are related to caching
  • Game.Server - a tiny process that used to do more, but now keeps player telemetry up to date
  • Game.Session.Supervisor - the supervisor for player sessions
  • Game.Channel - a gen server that tracks player sessions and which channels they are joined to, inspired by Phoenix Channels
  • Game.World - the supervisor that supervises the game world, see more below
  • Game.Insight - a small GenServer that tracks bad command parsing
  • Game.Help.Agent - an agent that load internal game help

This level has one_for_one as its strategy. At this level most sub-trees are fairly separate and can handle rebooting (to my knowledge) without interfering with other sub-trees.

Third Level - Game.World

This is the heart of the app. It contains everything the user interacts with in the game. Its direct children are Zone.Supervisor supervisors. This level has a strategy of one_for_one. This is fine because each zone is self contained and can reboot on its own.

Zone.Supervisor

This level has in startup order:

  • Game.Zone - the zone's state, which tracks what rooms/npcs/shops are online
  • Game.Room.Supervisor - A supervisor of rooms that belong to the zone
  • Game.NPC.Supervisor - A supervisor of NPCs that belong to the zone
  • Game.Shop.Supervisor - A supervisor of shops that belong to the zone

The reboot here is one_for_all. If any of these processes die something bad happened and the whole zone should restart. To further go into this, the Zone process tracks processes inside the sibling supervisors and if that dies then the rest should go as well. If the supervisors at this level died something really bad beneath them happened and the rest should be restarted.

When the sibling supervisors start they are started with the zone id. With this they figure out which children should be loaded at boot. Tese supervisors start processes as transient because they may be terminated normally and should not be rebooted, e.g. if someone deletes a spawner for an NPC then the process will be terminated cleanly.

The sibling supervisors are also a one_for_one strategy. This is fine as each process under them are fairly self contained and separated mostly for programmer benefit, this could probably be a big bag of processes directly under the Zone.Supervisor.

Take Aways

While doing this I was able to rework some of the tree. I pushed Game.Config further up the tree since that seems important. I also pushed more GenServers into the Cache sub-tree since they were similar.

One of the other reasons I did this was to figure out how to split up the app on separate nodes. This exercise taught me that it's currently not as easy as I was hoping. I figure the Web tree could be pulled off without doing much of anything, yet I found out that the Game tree is connected in a few spots that prevent it from immediately being pulled off. This would have been an annoying lesson to learn as I did that, now I know before hand and can fix the problems I found first.

In going multi-node, each of the first level would be good as a separate OTP app in an umbrella app. I had previously started with that but the application was too new for that to be useful. If I split them up again, I can boot nodes that are just for web, just for telnet connections, or just the world. I think this is a next step for going multinode.

I hope this was useful reading through seeing why I picked what I did and also finding out I had a few things ordered wrong. I hope you go through your own apps and try out a similar exercise on them.

ExVenture Updates for March 2018

Posted on 27 Mar 2018

The last month of ExVenture had a lot of game improvements in addition to a lot of tweaks and bug fixes. I say game improvements because mechanics were updated in regards to leveling and gaining stat improvements.

The documentation website is exventure.org. You can see the latest additions here on MidMUD, my running instance of ExVenture. There is also a public Trello board now.

Also check out The MUD Coder's Guild, it's a slack team devoted to developing MUDs.

Honing your stats

The biggest change (so big I reset characters on MidMUD) is being able to hone your basic stats. I got rid of your class boosting your stat a certain amount each level along with your level being added. It now increases each stat by 1 each level, and the top two stats get an extra point. The top two stats are chosen that level by the skills you used. If you use something that uses a lot of strength, your strength will go up faster.

To further customize your character, you can hone them. This uses your gained experience points and "spends" them on increasing a stat by 1 (or 5 if it's health/skill.) See more information on the help for hone.

Web Client

The web client got a very big update, commands are now clickable! As part of a semantic color update, I start sending {command}help config{/command} instead of {white}help config{/white}. This lets the web interface know that you can click those. The game can also send a different command to send when clicking a command, so you can have display text and the command itself.

A tooltip also displays:

command tooltip

Announcements

I added a small "blog" to the home page that lets admins post announcements about the game. You can sticky posts and write in markdown. There is an atom feed that is linked from the homepage so players can subscribe without knowing the feed URL. You can also simply not publish them to preview them on the home page as an admin.

Here is a sample announcement.

Hint system & Configuration

A hint system was introduced as a simple form of tutorial. For instance, after receiving your first tell each sign in you will see a message such as You can reply with reply my message. With this I introduced a player configuration system to disable these messages.

Also in the config system is disabling regeneration notices, changing your prompt, and setting the pager size. See more information at MidMUD's help.

Say improvements

I played some of the tutorial from Discworld Mud and I was inspired to add in a say to and whisper set of commands. Whispering sends your message to only the character it is directed at and everyone else sees only that you're whispering to each other. Say to directs a message at someone in the room for all to see.

Later, I was looking over the cheatsheet for CurryMUD and saw all the cool emote and say features it has. This inspired me to add in speaking at someone (in a cleaner fashion) and adverb phrases. I am very happy with how this feature turns out. You can be much more expressive in local chatter now. Adverb phrases also carries over to channel chat.

Bug Fixes

Several process crashing bugs were found, luckily no player was disconnected because of them. The worst one was sending in not an integer into quest info. Ecto was not happy about that. Other bugs include:

  • Running was missing up/down
  • Display gold if only gold was in the room, no other items
  • Remove command only worked for chest items
  • Configuring the prompt broke sign up
  • NPC combat target bug, you could trick the NPC into not attacking you when you entered a room
  • Updating an NPC crashed the NPC

Small Tweaks

  • Skills have an effect whitelist now, no damaging yourself while performing a healing command
  • Items have the same whitelist on a use
  • Movement in/out
  • Skills have a cool down
  • AFK status
  • Name map layers in the admin panel
  • Movement should indicate direction for other players, "Guard left north", this is also clickable!
  • Global broadcast when someone signs in or out
  • Start of a real templating engine for messages inside the game

Next Month

I didn't do the extra detail of the world like I had hoped last night, but I am pretty happy with what did get done instead. I think in the next month I want to have more game elements in place. Maybe data defined damage types.

Compiling External Resources with Elixir

Posted on 05 Mar 2018

The other week I started playing around with the @external_resource tag in Elixir. I wanted to do something similar to a translations file for the admin panel in ExVenture for help text. I didn't want to default to a YAML file as I've heard the elixir community isn't thrilled with it, so I took a look around to see what I could do.

What I ended up with was a macro that could compile a text file into Elixir functions. It also live reloads with the Phoenix code reloader in development because of @external_resource, this is really cool to see working.

Help File Format

This is the file format I wanted to end up with. Keys and values essentially separated by new lines.

room.ecology:
  Room ecology changes the color of the map "icon" in the map grid.

room.feature:
  Room features are appended to the end of a room's description.

You can see the full file here.

Help Macro

The end result of the macro will give us an API as follows:

iex> Web.Help.get("room.feature")
"Room features are appended to the end of a room's description."

This works via a macro (full file here) that loads the file and compiles it into quoted functions. The import sections are copied below:

defmacro __using__(file) do
  help_file = Path.join(:code.priv_dir(:ex_venture), file)
  {:ok, help_text} = File.read(help_file)
  quotes = generate_gets(help_text)
  quotes = Enum.reverse([default_get() | quotes])
  [external_resource(help_file) | quotes]
end

defp generate_gets(help_text) do
  help_text
  |> String.split("\n")
  |> Enum.reject(&(&1 == ""))
  |> Enum.map(&String.trim/1)
  |> convert_to_map()
  |> Enum.map(fn {key, val} ->
    quote do
      def get(unquote(key)) do
        unquote(val)
      end
    end
  end)
end

defp external_resource(help_file) do
  quote do
    @external_resource unquote(help_file)
  end
end

The __using__ macro reads the file and generates the get quotes and the external resource quote. The external_resource/1 function is how the code live reloads when editing the text file. The generate_gets/1 function parses the help file to generate a list of quoted functions, something I've never tried before. It was pretty cool to see you can return a list of quoted functions and get the same result.

Conclusion

It is really cool to see the live reloading work. This may or may not be the best way to do what I want, but I had a lot of fun writing this. I think if these help files grow to be hundreds of keys long I will want to change up how this is parsing, but for now it works out well.

Lastly, you can check out more ExVenture at exventure.org and see my running instance at midmud.com.

Eric Oestrich
I am:
All posts
Creative Commons License
This site's content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise specified. Code on this site is licensed under the MIT License unless otherwise specified.