How CSMAME works

This is the first of a series of posts which should explain how CSMAME/CSMESS & MAMEHub work internally. Please comment if you have any questions: I want to make sure the explanation is clear and coherent.

What to sync

Part 1 covers figuring out what data to sync among clients. These are core elements to sync:

Snap data: This includes everything that is stored on disk when one does a state save. It typically contains all of the memory address space for all of the chips and anything in the chip registers and/or cache. Note that if you have all of the inputs (mentioned below), you can start from the power-up time of the machine and replay everything exactly, so syncing snap data is not necessary, but it prevents new people who join the game from having to replay the entire machine history and is also necessary for correcting desyncs (desyncs will be mentioned in future post). NVRAM: NVRAM stands for "Non-volatile Random Access Memory" and contains data that persists from one power cycle to another (the term "power cycle" means the time from which the machine is powered on until it is powered down). Examples include the battery in the super nintendo cartridge that saves games or the high scores and settings on many arcade machines. When starting a machine for the first time, a copy of the NVRam is created and put in the nvram/ directory. Inputs: These are a ordered list of nested tuples in the form (player, (time, inputs)). I will cover how the inputs are assembled in another part.

There are some things that could/should be synced but aren't. Among these are:

Dip switches: Right now dip switches are not synced so it is the responsibility of the players to set the same switches at the same time and ensure that they match, otherwise desyncs can occurs (desyncs will be explained in a later post). CHD Diffs: CHD stands for "Compressed Hard Disk". Despite the name, CHDs are also used to represent CDs or other large media. Many modern arcade machines contain hard drives instead of storing data on NVRAM because it's cheaper, and this savings is significant given the amount of data in modern games. The CHD format is basically a bit-for-bit representation of a hard disk. Just like in NVRAM, the data on the hard disk can change and the changes must persist between power cycles. Because the CHD files are so big, it doesn't make sense to make a copy of them when a machine starts up like NVRAM. Instead, a DIFF file is created, which only contains the differences between the original CHD and the current CHD. For example, if you have some credits saved on the machine, the # of credits will be in the DIFF file, but other stuff (e.g. the character art) will not be in the file. At the moment, MAMEHub does not load DIFF files and assumes that CHDs are unchanged on startup.

How connecting works

Starting a client/server

When someone starts MAMEHub in either server or client mode, here's what happens:

All of the items to be synced (see part 1) are marked as MemoryBlock objects The MAME initialization code runs. This sets up MAME, initializes the chips in the machine, loads any initial ROM/RAM, and creates/loads the NVRAM. At this point the server creates a copy of all items to be synced, storing their initial values. These are called Initial Blocks and are useful in saving bandwidth when doing the initial sync. MAMEHub starts a UDP server socket and listens for incoming connections. Every 30 seconds (if the game supports save states), MAMEHub makes a new copy of all the items to be synced. These are called the Sync Blocks. If the game does not support save states, the Initial Blocks and the Sync Blocks are the same.

Connecting

First off, MAMEHub is a peer-to-peer system. It's also a fully-connected system, meaning that all peers communicate to each other directly. There is a "server" which is the first player to join the game, and this is for resolving conflicts among the clients, but from a network standpoint the server is just another player. There are many reasons for this:

If there was a centralized server (also called a client-server model), data would have to go from player A, to the server, to player B. This (almost always) takes longer than sending data from player A to player B directly. The server would need enough bandwidth to receive everyone's inputs and broadcast them to everyone else. Most household internet services do not have this upload capacity. You might ask "But wait, doesn't counter strike and (insert favorite net game here) use a client-server model???" Yes they do, and there are several reasons why they do:

If everyone is connecting to a server, only the server needs to have a port exposed to the outside world. With a fully-connected system, everyone needs a port exposed. Getting every player to have an open port is difficult. Most people running game servers are tech savvy enough to open a port, and most players are not. Because the game designers are making the game, they know what things in the game change (like the player's position) and what things do not change (the artwork). This means the game state (the things that can change in a game) are really tiny.

A Client-Server architecture (Most games)

A fully-connected architecture (MAMEHub)

When the game state is tiny, a client-server architecture is nice because the server can just blast the clients with an entire copy of the game state every second, and this quickly resolve desyncs (desyncs will be explained later). In the case of a game on MAMEHub, there is no good way to know what chunks of data are going to change and what aren't, so this benefit cannot be exploited. The game state in something like counter-strike might be 4 KB, but in MAMEHub it could be 128 MB or even larger.

With that out of the way, here's what happens when a new client joins a server (i.e. right after the client has finished the earlier 5 steps): CLIENT: The client tries to connect to the server's IP address. If this fails, an error code is returned and the client exits. SERVER: The server sends the client's IP address to any other clients that are already connected. OTHER CLIENTS: The other clients try to connect to the new client (called the candidate). The client then reports back to the server whether it was able to connect to the candidate or not. SERVER: The server gathers reports from the other clients. If any of the clients can't connect to the candidate, the candidate is rejected and the client fails and exits. SERVER: If the client is accepted, a player is assigned to the client. If the client is rejected, the server tells the other clients to disconnect from the candidate client and not to wait for inputs from them. Alright, now that all the hand shaking is done and we are sure this client will be in the game, the server has to get the client up to speed. SERVER: The server calculates the XOR between the Initial Blocks and the most recent Sync Blocks. The way XOR works, if the data is the same, the XOR will be zero. It will only be non-zero where the data has changed. Because most of the data does not change, there will be a lot of zeroes. SERVER: The server then LZMA-compresses the result and sends it to the client. Because there are so many zeroes, even 128MB of data will typically compress down to less than 1 MB. Of course, the longer the game has been running, the more data has changed and the larger this will be. CLIENT: The client uses the XOR data and the client's Initial Blocks to recreate the state of the game at the time of the last sync. CLIENT: The client then has to "catch up" from the time of the last sync to the present. All of the other clients and the server wait for the client to catch up, and then everyone starts broadcasting inputs (broadcasting inputs will be covered later)

How inputs work

In this section I'm going to discuss how inputs are sent and received among the clients. Although we say "inputs" we really mean "input state". If no buttons are being pressed, that fact is just as important as if all of them are being pressed, so data needs to be flowing regardless of if buttons are actually being pressed.

How inputs work in MAME In MAME, an input can be analog or digital. Digital inputs include the keyboard, digital joysticks, gamepad buttons, etc. Analog inputs include the mouse, light-guns, analog joysticks, steering wheels, etc.

Inputs are arranged in the following hierarchy:

• At the top level is a series of input ports. Each input port contains:
  • The default digital mask. This can be client-specific so it has to be sent. Each digital input is a single bit in the mask. For example, bit 0 in the mask might be for the "start" button on the SNES, and bit 1 is for "select".
  • The current digital mask. This is the value that changed as a client presses and releases buttons
  • A list of analog devices. Each analog device contains:
    • Accumulated delta
    • Previous value (for calculating inertia)
    • Sensitivity (the lower this number, the more sensitive the input)
    • Reverse (in case the client wants the input to go in reverse (e.g. for flight simulators, some people want to go up when they press down)).

MAME keeps a mapping from user inputs (like keyboard commands, mouse clicks, etc.) to in-game commands (button presses, etc.). For example, an in-game input might be "NBA Jam turbo button", but the user inputs are "button 2 on logitech joystick or (control + w)". Because several keys/buttons can be mapped to a single in-game button, the user inputs are called an input sequence. So you can think of MAME as having a map which maps in-game inputs to input sequences.

At every frame of emulation, MAME does this:

• Input frame:
  • Loops through all ports
  • For each port, loop through all devices
    • For each device, loop through all inputs
    • Look at the mapping and poll the hardware to set the value of the input (binary value if digital, integer value if analog)
• Game frame:
  • During the game frame, the seq_pressed(...) function is called by the game to check if an input is pressed (e.g. move the character to the right if the "move right" input is set). The seq_pressed() function checks the input values that were computed in the input frame, so game frame knows nothing about physical keyboards, joysticks, etc.

How inputs work in MAMEHub In MAMEHub the input arrangement is the same, but the runtime is vastly different.

The first problem is that the player a client controls is decided by the server when the client starts (see part 2). This means that the first client to connect to the server will control player 2, the second will control player 3, etc.. One problem with this is that each client would have to set the inputs for the player they are for that particular game, and the next time they played, they might have to set inputs for another player. This would be a nightmare. To avoid this nightmare, MAMEHub maps in-game inputs from one player to another (called the Player Field Map), and assumes that every client uses the controls for player 1. There is a player field map for every player (denoted by X) except for 1, which maps player X's input to player 1's input. Here's an example:

1. A client controlling player 2 presses the 'w' key on his keyboard
2. The "Player 1 Move Up" input has the 'w' key as an input sequence
3. The player field map for player 2 maps "Player 1 Move Up" to "Player 2 Move Up"

At every frame of emulation, MAMEHub first does this:

• Input frame:
  • Create the Future Input Time, which is the current game time plus the latency estimator (latency and latency reduction will be explained in another post)
  • If the future input time is earlier than the previous future input time from the last frame (because the latency estimator has gone down), set the future input time to the previous value plus 1.
  • Do everything MAME does in the input frame (see above)

At this point in the input frame, the input values contain the future inputs. They contain what the current client wants the inputs to be, but because of delays in the internet and other things, they aren't what the inputs actually are. The inputs also do not reflect the inputs set by other clients. Note too that we haven't dealt with the player assignment problem. Every client assumes they are player 1, so every client is trying to move the first player or insert coins in the first coin slot. The next block of the input frame deals with all this:

• Input frame (part 2):

  • Create an input state. The input state is a list of input values for a particular client, in the same order as they are seen when looping through the ports and devices.
    • To create the input packet, first loop through all of the devices
    • Then check if the player value:
      • If the player value isn't set, assume this is a shared device like a keyboard or a shared coin slot and append the input value into the input state.
      • If the player value does not match the client's player value (set by the server on connecting, see part 2), append a "0" to the state
      • Otherwise, check the input field map (explained above), and append the input value from the mapped field to the packet. For example, if we are looking at "Player 2 move up", it will append the value from "Player 1 move up".
  • Now that we have what the client wants to do in an input state and mapped to the correct player, we create an input packet that contains an input ID (which is an incremental counter), the future input time, and the input state. This input packet is then broadcast to all other clients (sending/receiving is explained below). So now the client has sent out how they want the inputs to change in the future, but we still need to know what to do right now. Here's the remaining logic:

• Input frame (part 3):

  • Loop through all of the devices for all ports and clear their input values (set everything to 0).
  • Wait until an input packet from every client with a future input time greater than the current time has been received.
    • If we get a packet from the future, we know that there will be no more packets from the past because of assumption #1 (see below). Otherwise, we would never know when we have all the information necessary to process a frame.
    • If the client hasn't been connected for more than a second, we know that the client will not produce any inputs because of assumption #2 (see below) and we can skip them. This prevents deadlocks when two clients join at the same time.
  • For each client, find the input packet that is most recent but still before or exactly on the current time.
  • For each of these packets, compute the bitwise OR of their input state. This is called the merged input state because it contains the inputs from all clients smashed together into one input state. Take the merged input state and set all of the devices for all ports based on this state.

• Game frame:

  • (Nothing changes from regular MAME)

Sending/Receiving inputs

Now that we know how input packets are created and used, we need to talk about sending/receiving. There are two ways input packets are sent:

• When a new client connects and is accepted (see part 2), the server sends all inputs from all clients to the new client. The server has to send everyone's inputs because a previous client might have come and gone and that client's inputs are still necessary to recreate the history of the emulation.
• At every frame of emulation, an input packet is created and broadcast to all clients (see above).

When a client receives an input packet, the input ID is checked to see if it is the input the client is expecting. The client expects inputs 0,1,2,3,.... in order from each other client. In MAMEHub v1, the order was guaranteed by the network protocol, but in v2 we removed this guarantee to reduce latency (latency and latency reduction will be explained in another post).

Assumptions

1. For a given client, input times are monotonically increasing. This means if you send an input at time t, the next input must be at t+x, where x is a positive number. This also means loading save states that go back in time or resetting the machine are not supported.
2. For the first second after a client connects, the input state must be the default values (all buttons unpressed, all analog inputs set to their defaults). This gives new clients a chance to send inputs in the future without having inputs in the present.