Sunday, July 20, 2008

Socket Programming with Flex and Apache Mina

Recently, I've been doing a significant amount of socket programming between a flex application and our Java servers. Namely, we were building a game in Flex with enable real-time multiplayer. Since HTTP connections are way too slow for real-time multiplayer games, we needed to use socket layer communications organized into channels. Here's a couple things I learned in the process.

Version 1
The first version I whipped together used standard TCP blocking sockets with pools of acceptor threads, worker threads, and SQS monitoring threads (since the multiplayer servers communicate with the web-application servers via Amazon SQS). This was a straight forward process, with the main difficulty being to guaranteeing synchronized access to resources shared by multiple threads. The result was more complex than desired and could handle a max load of only a few hundred concurrent users. Beyond that, the CPU time would become bogged down by switching contexts between worker threads, leaving little CPU resources for actually doing the work. Not good enough... scrap it, on to round 2.

Version 2
The second version involved using Java NIO non-blocking socket connections, which promised potential scalability to thousands of concurrent users with very few threads, and thus very little context swapping and a lot of CPU resources for actual work. The downside is that writing an NIO server is significantly more complicated and, in the words or a friend, really a pain in the ass. Fortunately, I only spent a few hours designing how I wanted to proceed before the same friend saved me days of work pointed me in the direction of Apache Mina.

Version 3
The third version was implemented with Apache Mina in roughly 2 days and has worked wonderfully thus far. The documentation for Mina is quite good, with plenty of sample applications. Mina is basically a barebones Java NIO server that can be configured to do whatever you need from a non-blocking server. It handles all of the gritty Java NIO and low-level communication and provides the programmer with standard web-application conventions like Sessions and a filter chain. Wow, that was awesome.

Implementing a server based on Mina is done with three components:
  1. A protocol encoder
  2. a protocol decoder
  3. a protocol handler
The first two become part of the filter chain between Mina and the net, and handle the job of encoding/decoding your internal representation of a command to/from the bytes that are sent across the net. The beauty of this is that, internally, your command can be a POJO offering whatever convenience methods you desire, as opposed to the ByteBuffer it is received as. The protocol handler is the class that does all of the work. It includes such functions as sessionCreated, sessionIdle, sessionClosed, and messageReceived (which are called when their names would suggest).

Overall, the Mina version was both the simplest in design and implementation, as well as the best performing. I'm extremely happy with the choice.

On the Flex side, I decided to mimic the Mina structure by implementing a protocol encoder/decoder as well as a handler, which has worked out quite well. Since working with Mina had already involved thinking in that manner, porting the structure to Flex was the most natural procedure.

So, basic intro aside, here are a few of the "gotchas" that I encountered:

Policy File Requests
The first thing Flex will do when opening a Socket connection is send a policy file request to the host and port to which you are trying to conect. You have two options in this scenario. The first is to have your server check for <policy-file-request /> and respond with the a policy file. The second is to have Flex read the policy file from an alternate location via Security.loadPolicyFile("wherever") before attempting to open the socket connection. I found the later to be more convenient since it removed the need for my mulitplayer server to know how to handle policy file requests. There are a few restrictions with this route, though.

See here for the full reference, but the basic rules are these:
  1. A policy file served from above port 1024 cannot grant access to a port below 1024
  2. Policy files served via an xmlsocket (rather than an Http request to an actual crossdomain.xml) must be terminated by a null byte
  3. Policy files served via an xmlsocket must include the to-ports attribute of the allow-access-from tag
  4. Access to ports below 1024 can only be granted via a call to loadPolicyFile
An example call would be Security.loadPolicyFile("xmlsocket://localhost:5000"); Upon receiving a valid policy file, Flex will then establish the desired socket connection.

Flushing the Flex Socket
You would think that a call to socket.flush() would be necessary every time the Flex app writes to the socket. In practice this isn't the case, but you should always flush anyway. I only mention this because it is an oddity. Namely, in Mac OS X browsers, Flex commands written to the socket are received fine by the server without ever flushing the socket on in the Flex client. In Windows browsers, however, failing to flush the socket will cause the command not to be sent on all writes to the socket after the first (the first write will go through fine without flushing). Surprised the hell out of me when I discovered this (which I did by writing the app on a Mac and failing to flush the socket, which worked fine, until I tested it on a Windows machine and found out it wasn't working).

2 Messages Sent !(necessarily)= 2 Messages Received
In your protocol decoder (the one that receives incoming requests and converts the bytes into your internal representation of a command), you have to maintain a state to monitor where you are in receiving data. For example, you may receive one full command, only part of a command, or more than a full command at any given time. In short, this is due to Nagle's algorithm to increase network efficiency by decreasing the number of packets sent (full description). Your protocol decoder needs to know how to handle these situations.

The case is simplest when receiving a whole command, since reading all bytes available will result in one full command. However, the other cases are made easier to handle by including the length of the incoming message in it's header. For example, I defined a very lightweight protocol of 1 byte to indicate the type of message, 2 bytes to indicate the length of the message, then the message, which could be of variable size. Thus, when receiving some data, you can check the length to see how much you need, and then handle the incoming data based on how much you have and how much you need. A longer description with good examples is available here on the Mina website. I mention it here because it is definitely worth reading before writing any code, and if you neglect it, you'll end up with code that will behaive unpredictably.

That's about all I can think of right now, I started this post days ago and just came back to it today, so I might add anything else if I remember, or maybe as a new post. Happy socket programming!

7 comments:

splendidness said...

Thanks a million for pointing me in the same direction.
Every similar project I done in the past has taken the same approach to your version 1, and every time it's been hard work catching all of the interesting ways in which a client can disconnect. Mina has covered all of this as well as doing most of my thread handling work for me.
The documentation is quite good but I found the best approach was to decide which example was the closest match to my requirements and then use that as a model.
It's a great project and will help me be much more ambitious in the future.

narup said...

hi, thanks for this good article i am also starting on a similar work,
so just wondering did you use XML socket connections or binary socket connections can you just provide me some clue on the whole architecture.

Ryan said...

Hi narup, I was responding to your comment and it started getting long, so I turned it into a new post. Here's the link http://blog.augmentedfragments.com/2008/09/socket-programming-with-flex-and-apache.html

hope that helps!

Flex programming said...

hey

Really nice tips!

will try to keep it in my mind

thanks!

aaa kitty20101122 said...

hermes belts
prada sunglasses
adidas nmd
adidas nmd runner
stephen curry shoes
retro jordans
adidas online shop
tory burch shoes
cheap jordans
air force 1

aaa kitty20101122 said...

adidas nmd
air jordans
http://www.kobeshoes.uk
adidas tubular
golden goose
air max 2016
true religion outlet
kobe shoes
nike polo
jordan 11 retro

Pervez Joarder said...

Outstanding post. This is absolutely exactly what specifically my partner and I in fact won’t to learn. I must say I as an example your internet site we all really like your time and effort dissertation planet. Your article is very helpful on behalf of American state and plenty of others to work out. Ready toil is able to completely return on your web site for extra stuff. Now click Payday Loans Thanks for good work.