PokeMMO Network Protocol 3: Speaking the Protocol

Intro

Two posts in, we can open a frame. Post 1 gave us the length prefix and the handshake, Post 2 gave us the cipher, the integrity check, and the compression. Strip all of that away and what is left is a packet, an opcode followed by a body.

This post is about that body. What is inside a packet, and how you read it. Going in I expected some elegant serialization framework. There is not one. The real game does the least clever thing possible, and honestly that makes it easier to reverse.

A packet is just a class with methods

In the real client, a packet is a class with an ID and a handful of methods. One method writes its fields onto the outgoing buffer. One method reads its fields back off the incoming buffer. One method handles it, the logic that actually runs when the packet arrives. That is it. There is no schema, no field table, no registry describing the layout. The layout lives entirely inside those read and write methods, as a sequence of plain buffer calls.

This is exactly what we hooked in the snooper posts without fully appreciating it. The send hook sat on the write method, the one that fills the buffer. The receive hook sat on the handle method, the one that runs after the read method has already turned bytes into fields. The read and write methods are two halves of one thing, the packet’s structure, written out by hand.

So there is no shortcut. To learn a packet’s structure you read its write or read method top to bottom and write down what each call does. Let us do exactly that with a real one.

Reading a real packet, the login packet

Here is the login packet, opcode 0x11, client to server. This is decompiled and renamed, the field and method names are recovered guesses and will not match the next client build. It is the packet the client sends with your credentials.

public final class LoginPacket extends LoginOutgoingPacket {
   public final String  username;
   public final String  password;
   public final boolean rememberMe;
   public final boolean rememberPassword;
   public final OtpToken otpToken;      // may be null
   public final byte[]  hardwareId;

   @Override
   public void encode(ByteBuffer buf) {
      writeString(this.username, buf);              // 1
      buf.put((byte) (this.rememberMe ? 1 : 0));    // 2

      buf.put((byte) this.hardwareId.length);       // 3
      buf.put(this.hardwareId);                     // 4

      if (this.otpToken != null && this.otpToken.present) {
         buf.put((byte) 1);                         // 5  auth tag: token
         buf.put((byte) this.otpToken.value.length);
         buf.put(this.otpToken.value);
      } else {
         buf.put((byte) 0);                         // 5  auth tag: password
         writeString(this.password, buf);
         buf.put((byte) (this.rememberPassword ? 1 : 0));
      }

      writeString(clientLocale, buf);               // 6
      buf.putInt(31914);                            // 7  fixed constant
      buf.putInt(BuildInfo.revision);               // 8
      buf.put(platformCode);                        // 9  one enum byte
      buf.put((byte) selectedServer.id.length);     // 10 length-prefixed blob
      buf.put(selectedServer.id);
   }
}

Read straight down, that method is the packet’s structure. Turned into a wire layout it looks like this.

And here is one instance as raw bytes, taking the password path, for the user Ash with password hu. This is the decrypted, decompressed body. The outer opcode plus the length, cipher, and checksum from the first two posts wrap around it.

11                            opcode 0x11 (login)
41 00 73 00 68 00 00 00       username "Ash" as UTF-16LE, 00 00 terminator
01                            rememberMe = true
04                            hardwareId length = 4
DE AD BE EF                   hardwareId
00                            auth tag = 0 (password path)
68 00 75 00 00 00             password "hu"
00                            rememberPassword = false
65 00 6E 00 00 00             client locale "en"
AA 7C 00 00                   client revision 31914 as a little-endian int
39 30 00 00                   installation revision 12345 as a little-endian int
01                            platform code
03                            server id length = 3
01 02 03                      server id

That is a whole login packet, and nothing about it is mysterious once you accept that the encode method is the spec.

The building blocks

Look back at that layout and you have basically seen the whole toolbox. A handful of shapes show up again and again across every packet in the protocol.

• Fixed-width integers. A byte is a byte, a short is two bytes, an int is four. There are no variable-length integers here. If you come from other MMOs you will reach for a varint decoder out of habit, and it will be wrong. Every number is a plain fixed-width value.
• Booleans. One byte, zero or one. Cheap and everywhere.
• UTF-16LE strings. Text is UTF-16 and terminated by a two-byte null. That is why every character in the dump above has a trailing 00.
• Length-prefixed blobs. A one-byte length followed by that many bytes. The hardware id, the token, and the server id all use this exact pattern. When you see a small count byte followed by bytes, this is almost always what it is.
• Tagged branches. The auth tag at step 5 is a real fork in the layout. One byte selects which data comes next, token or password. Whenever a packet can carry one of several things, expect a tag byte in front choosing between them.

OpenMMO’s tidy version

Writing a read method and a write method by hand for every single packet is a lot of repetitive, error-prone typing, and it is easy for the two halves to drift out of sync. OpenMMO takes a different route. It describes each packet once, as a declarative recipe of fields, and derives both reading and writing from that one description.

// the same login fields, described once instead of twice
object LoginRequestCodec : PacketCodec<LoginRequest>() {
  override fun CodecScope<LoginRequest>.body(): LoginRequest {
    val username = field(Utf16LeNullTerminated, LoginRequest::username)
    val rememberMe = field(Bool, LoginRequest::rememberMe)
    val hwid = field(bytesPrefixed(U8), LoginRequest::hardwareId)
    // ... and so on
    return LoginRequest(username, rememberMe, hwid /* ... */)
  }
}

Same bytes on the wire, opposite authoring style. The game hand-writes two methods per packet, the imperative way. OpenMMO writes one recipe and gets both directions from it. When you read the reference code, that recipe is what you will see, and it maps one to one back to the encode method above.

Game vs OpenMMO. This is the biggest structural difference in the whole series. The codec is an OpenMMO convenience, not something that exists in the real client. Do not go looking for it in a decompile, you will not find it.

The login flow

One packet is a word. A protocol is a conversation. The login server runs a small state machine from the moment you connect to the moment it hands you off to a game server. At the message level it goes roughly like this.

connect
  |
  |  client -> LoginRequest (0x11)           credentials, hwid, revision
  |
  |  server -> MfaChallenge (0x08)           only when logging in from a new device
  |  client -> MfaResponse  (0x08)
  |
  |  server -> ToS (0x14)                    only on first login
  |  client -> ToSConfirmation (0x04)
  |
  |  server -> LoginResponse (0x01)          accepted, or a kick (0x05)
  |
  |  client -> RequestGameServerList (0x02)
  |  server -> GameServerList (0x22)         the servers and their addresses
  |
  |  client -> JoinGameServer (0x03)
  |  server -> GameServerNodes (0x03)        handoff details
  |
done, connect to the game server and close this connection

Notice the opcodes collide across directions. 0x03 and 0x08 each mean one thing from the client and another from the server, which is the opcode-plus- direction point from Post 1. Notice too that several steps are conditional. No 2FA, no challenge. Unchanged terms, no ToS step. The happy path is short.

The game protocol, from a distance

Once you reach the game server, the protocol gets big. Where login is about 20 packets, the game protocol is a few hundred, spread across nearly the whole one-byte opcode range, and this is the layer that turns compression on. Everything from the first two posts still applies unchanged. It is the same frames, the same cipher, the same read-and-write-method packets, just a lot more of them.

I am deliberately not enumerating packets here to keep the post short.

Wrapping up

That is the protocol, end to end. A length-prefixed frame, an integrity check and a stream cipher set up by a signed handshake, an optional compression layer, and inside it packets that are nothing more than a class with a read method, a write method, and a handler. No varints, no schema, no magic. Just fields written to a buffer in order.

From here the interesting work is on the game protocol itself, verifying those packets against real traffic one subsystem at a time. Battle is the obvious first target. If I end up doing that, it will be its own series.

Thanks for reading. If you have questions, my contact info is on the About page.