Recent Posts

ExVenture Updates for October 2018

Posted on 26 Oct 2018

The last month of ExVenture was a lot of minor refactors and other chores that needed to happen plus a migration back to working on Gossip.

Links for MidMUD & ExVenture:

Gossip

I started working on Gossip again. There is a new support flag that you can support, which loads basic information about the games that are online, games. You can also receive events for when a game connects and disconnects now. This should help you keep local caches more up to date. See the Gossip Docs for more information about the new events.

Gossip also got a HUGE refactor of it's socket code. It had started getting fairly spaghetti like so I spent a weekend refactoring it to be much nicer, this pull request shows how that went.

Grapevine

I also launched a new site called Grapevine that is for players on the Gossip network. Games have a profile page now, which shows the basic game information plus current players and how to connect and play yourself.

Players also have a profile which shows the in game characters you've registered. Grapevine is a full Gossip application which means you can send a tell to it. This allows grapevine to know who your player is and can link the character to your profile.

There is a lot of work on this site that I want to do in the future. The biggest of which will be achievements. See this issue on Gossip's GitHub if you're interested in helping shape what these look like.

Gossip Elixir Client

The Gossip elixir client also got a major refactor thats about to wrap up. It was getting large enough to finally have some patterns take root and be able to refactor to be nicer (it might have also been drifting towards spaghetti code as well oops.) This pull request shows this refactor in action.

The client also got updated to include all of the current features that Gossip the server supports. You can also not support every feature that Gossip and the client can handle by providing only the callback modules that you wish to support. I need to update the docs for this but expect a 1.0 client soon!

Raisin

Games supporting Gossip is starting to have a minor upswing and I want to be ready for when this takes off. To help with that I started Raisin to handle moderation. Raisin is a network application like Grapevine and logs the goings on of the Gossip network. This is all it does at the moment, but this is the place where network admins will be able to moderate the community.

Backbone Sync

To support Grapevine and Raisin, Gossip needed a new level of connecting game which I've called a network application. This application gets socket super powers when it connects and starts to receive sync events. When Grapevine loads it gets a set of sync/games and sync/channels events, and will continue to get them as anything changes.

To keep code duplication minimal, when I started Raisin I pulled out the common sync code into a separate OTP application. It manages its own repo and keeps sync code in one spot. gosisp-backbone is up on GitHub.

Small Tweaks

  • API representers to handle outputting in various formats
  • Remote erlang QR code library for an elixir version
  • Player saves get updated via a common function
  • Create a character struct for a user, continuing the migration to multiple characters per account
  • Gossip update indexes to use the SQL lower() function
  • Gossip can have user agent repo links
  • Gossip blocks certain game names
  • Gossip can have connections for your game
  • Gossip's your game page got a spruce up

Social Updates

This month was pretty good for ExVenture on the social front. Lots of new stars across all of the projects!

There are several new patrons over at the Patreon as well. Welcome and thanks for supporting the project!

The discord group has several new members in the last month.

I was also promoted to be an admin of The MUD Coders Guild this month. I look forward to helping make Gossip and co a more official part of the guild. It was already unofficially official since there is a #guild-outreach channel where we all talked about new features, but now we can take it to bigger and better heights!

Next Month

Over the next month I will most likely keep going with Gosisp, Grapevine, and Raisin. In particularly Raisin needs to be able to have some power on the network. That will be fun to write the events for.

I also want to keep moving on achievements. I will most likely add them into ExVenture first, before starting on Gossip itself. I would like to be ahead of the network for Gossip instead of letting ExVenture play catch up.

Gossip Sync Websocket Protocol

Posted on 16 Oct 2018

In the last few weeks I started on a new project that is tied to Gossip, the chat network for text-based games (aka MUDs.) This project is called Grapevine. Grapevine is a player site where you can register your characters across any game that is connected to Gossip.

Links for Gossip & Grapevine:

Sync Protocol

For Grapevine, I wanted to have it connect with the standard Elixir client for Gosisp which meant it needed to live on the same socket as a standard game. I went with adding in a Gossip network-level application that pretends to be a game for most parts, but when one connects it gives the socket super powers.

The biggest of those super powers at the moment is recieving sync/* events. When the socket connects it gets a set of sync events for all of the channels and games that Gossip knows about. Each event is bundled up into 10 channels or games, to help out a little bit with network overhead.

A sync event looks similar to this:

{
  "event": "sync/channels",
  "payload": {
    "channels": [
      {"id": 1, "name": "gossip", "description": "...", "hidden": false}
    ]
  }
}

When Grapevine receives this it creates or updates a local version of that remote record. I called this the backbone sync, you can view it here on Gossip's side.

These events also trigger for any creates or updates while the socket is connected. When a new game is made or a game is edited, Gossip broadcasts an internal sync event that the socket is subscribed to. This immediately pushes out and keeps the remote applications up to date always.

Raisin

One of the main reasons for wanting to head this socket route, is that I also wanted other higher privileged applications to live on the network. The next one being Raisin, which will be a moderation tool for the network.

With the start of Raisin, I wanted to reuse the same sync code across both projects. I had previously done this for Rails applications via Rails Engines, but I wasn't sure how to accomplish this. After talking with some fine folks over at the MUD Coders Guild, I think I stumbled across something that works.

I pulled out all of the common code into a new library that Grapevine and Raisin depend on. This library hosts its own ecto repo so it can internally save the data. Configuration wise, it's the same database as Grapevine or Raisin. That lets you use the backbone internal schemas outside of the backbone library.

I'll have more to say about this as I continue to explore it.

Conclusion

It is extremely exciting to watch these sync events propagate. The closest I've seen to this are webhooks, but being connnected to a duplexed socket is another thing entirely.

I hope you all check out the code backing all of this, and maybe even decide to join the network yourself as a player!

ExVenture Updates for September 2018

Posted on 25 Sep 2018

The last month of ExVenture resulted in a lot bug fixes in preparation for ElixirConf with some web client features towards the end of the month.

Links for MidMUD & ExVenture:

Web Client

The biggest feature of the month is definitely the web client additions. Last week I was inspired by a post on /r/mud about playing a MUD while using less of your keyboard and more of your mouse. I've had in the back of my mind how to do this and finally got around to it.

ExVenture web client action bar

The new additions are a target bar, that shows all of the other characters in the room and allows you to view and change your target, and an action bar that allows for up to 10 skills to be displayed. The action bar currently only auto populates with skills as you level and train new skills. The system is set up to allow for custom commands, but I haven't gotten around to configuring that part yet.

I am very happy with where it's at after a week of work. There are many updates in the future for this area, but it's a great foundation. The best part of this is you can now wander around and do combat on mobile.o

Performance

I couldn't help myself and kept looking into performance constraints for a single server. I was able to remove a few sort of unnecessary SQL checks and a few other optimizations to unlock nearly 3x more concurrent bots on my desktop. I managed to push to 7100 bots at the same time.

ExVenture concurrent player guage

I did not commit a lot of the changes for this, since it restructures some core stuff at the moment. But I did start an issue to move towards this.

What I did commit was migrating session data to an ETS table, for reads outside of the session registry process.

Gossip

Gossip was fairly quiet last month, but did see some bug fixes. You will no longer receive heart beat messages before you authenticate. There were also a few small UI tweaks that went in place.

Translations

I started looking into translating ExVenture into other languages. Right now any text that is in a command is being run through gettext. This is very early work but I need to start somewhere, since I'm sure a lot needs to change depending on what language is being translated into.

I also set up a new translation site on Heroku, if anyone wants to help translate ExVenture let me know in the discord. I can get you access to that.

Smaller Tweaks

  • Save an external discord ID to push for Mudlet
  • Hint system sends messages when nothing has been entered on first connect
  • Rename user tuples to player tuples, moving towards many characters per account
  • Help text updates
  • Fix all config options being able to be set to true/false
  • Cast data that goes straight to ecto (quest info #)
  • Disconnect an individual user from the admin
  • Admin panel has script editing for NPCs in the new react component
  • Global features can have tags, to better view adding them
  • Say with brackets was filtered out as an adverb phrase even if it was in the middle of your text
  • Client.Map GMCP push on connect, hopefully letting Mudlet pick up on an MMP map
  • Disable skills for use and view
  • Custom commands for using an item, you can drink potions now
  • Global replacements have a read through cache, nodes dying did not repopulate the cache
  • Overworld was not properly rendering newly made exits in the admin

Social Updates

This was another big month for ExVenture and associated projects. I released Squabble as a separate repo and it's picked up a lot of stars after the introduction post.

There are several new patrons over at the Patreon as well. Welcome and thanks for supporting the project!

ExVenture itself picked up a lot of new stars this month, gaining about another 50 or so after ElixirConf. There are also a lot of new people in the discord group, many are first time learners of Elixir. Maybe I'll see you there too!

Next Month

Given how off I usually am with guessing what takes my interest over the next month, whatever I put here will most likely be wrong. But I would like to continue with the web client, adding in configuration and keyboard short cuts. I also need to swing back around to the overworld at some point and continue fixing that up and making it fit more into the game.

I also might pull off some of the "boring" things that I've been meaning to get around to, such as adding in forums and letting each user have multiple characters. I think both would be good additions and I've been at least moving slowly towards the later.

Introducing Squabble, Simple Leadership Election

Posted on 10 Sep 2018

As part of my ElixirConf talk, I showed off Squabble my new Elixir package for selecting a leader in your cluster. Squabble uses the leadership election part of the Raft Protocol. This was pulled out of ExVenture, my clustering text based multi-player game server.

What is this?

Squabble takes the leadership election part of Raft and plugs it into your Erlang/Elixir nodes. Each node has a single Squabble process that communicates with the other Squabble processes across the cluster.

Together they vote for a leader and track the other nodes coming and going. Whenever a leader is selected, a callback module you provide is called on the new leader node. This lets you do a single set of work across the cluster.

Why do I want this?

ExVenture uses this to rebalance the stateful virtual world. The world is broken up into zones (think of a whole city as a zone) and they can be pushed across the cluster. When a node goes down it takes down part of the world. The leader notices this and then detects which zones are not alive across the cluster and spins them up again.

Why not use the Raft package?

This is separate from the Raft Elixir package, it is trying to do something different. I want a simple system that only picks a leader node in my cluster. I don't need a log or anything else that comes with the Raft protocol.

This is not intended as a replacement for the Raft elixir package.

How do I use this?

Setting up Squabble is pretty simple. First You add it to your supervision tree after your application is clustered. You can cluster your nodes with libcluster.

This is the setup ExVenture uses.

children = [
  {Cluster.Supervisor, [topologies, [name: ExVenture.ClusterSupervisor]]},
  {Squabble, [subscriptions: [Game.World.Master], size: @cluster_size]},
  # ...
]

Supervisor.start_link(children, opts)

When the leader is selected, the World.Master is invoked to rebalance the cluster and start the world. Any a node dies, the World.Master will see what is no longer alive in the cluster and start those processes again.

defmodule Game.World.Master do
  @behaviour Squabble.Leader

  @impl true
  def leader_selected(term) do
    GenServer.cast(__MODULE__, :rebalance_zones)
  end

  def handle_cast(:rebalance_zones, state) do
    rebalance_zones()
  end

  defp rebalance_zones() do
    # finds alive zones and determines which are not started,
    # and starts them
  end
end

Conclusion

I hope you give Squabble a look. It doesn't fit all needs, but for a simple set of callbacks in a small cluster it works very well. Check it out and help make Squabble even better!

ExVenture Performance Tweaks

Posted on 05 Sep 2018

In the week or two leading up to ElixirConf I started trying to see how far I could push ExVenture with concurrent players. These are the three fairly simple tweaks I took to go from maxing out at 230 concurrent players to maxing out at 3500 concurrent players on the same machine.

Tests were performed on a desktop with a Intel Core i7-6700K and 64GB of RAM. I used VentureBot to connect to my local instance going across the local network, including wifi. I used a copy of MidMUD as the world.

Single process overloaded by messages

The first time I ran VentureBot pointed at my local development version of ExVenture, I was able to get to around 230 players before room processes started falling over. The room process is a router of sorts for players in the same place of the world.

Anything that happens in a room by a player is broadcast to every other player in the room. Player processes also call the room to get the current state of it before acting on it through commands. The combination of these two was causing the room process to not be able to keep up with the level of messages.

Enter a side process event bus. Any notifications that the room process wants to do is pushed into this side process. This unblocks the room process from performing the relatively slow notification of characters in the room.

This pushed concurrent players up to about 600. You can see this in Pull Request #72.

Single process overloaded by state size

ExVenture has a player session registry that keeps track of connected players. This is very similar to the Elixir Registry, but is a custom process on each node. The next thing that could not keep up with players was this.

When a player connects, their full user, class, skills, race, etc are all preloaded from the database. This is then pushed into the session registry. The session registry process was being slowed down by the state it held.

Almost all of this data was not required by other characters in the world. The answer then was to massively cut down what was stored in the session registry. I created a new Character.Simple struct that the session actually stored.

This pushed concurrent players up to about 1200. You can see this in Pull Request #73.

Processes overloaded by inbox size

At this point something I really was not expecting to happen happened. I ran out of ram. Mind you this is on a desktop with 64GB of it.

The reason was almost for exactly the same reason as the previous tweak. I had fixed the registry, but the player gets pushed around a lot in messages and other process state (like the room process.)

MidMUD has about 250 rooms, which meant that there was an average of 5 players per room, but every player spawns in the same room on start. This meant that a few rooms were getting up to 30-40 players in the same room. Each action a player took was then pushed to 40 other processes. Because the player struct was preloaded to the brim it was making a lot of copies of that data and blowing my RAM.

This isn't a memory leak per se, but it wasn't good.

The way around this then was to use the same simple character struct for all messages. After adding this ram usage dropped substantially. 1200 players took about 1GB of ram vs the previous 50GB.

This was Pull Request #74.

Conclusion

This is the final guage I was able to get from Grafana on concurrent players.

Final player count

For reference, most MUDs these days are over the moon with anything above 20, the top one getting ~800 players. ExVenture pushed into MMO territory with these changes.

The current reason ExVenture cannot take more is because the session registry can't keep up with new players connecting which fetches the full list to see if they are already in the game. To get around this I would change up the session registry to act more like the Elixir Registry and create a worker pool to manage incoming calls.

But, there's almost no reason to try and fix this. 3500 real players is a very, very far off problem in the real world. I'll settle for 15x performance for now.

I hope these small tweaks are actionable in your own Elixir applications, especially message size between processes.

Eric Oestrich
I am:
All posts
Creative Commons License
This site's content is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise specified. Code on this site is licensed under the MIT License unless otherwise specified.