Hibernate Bug when counting Extra Lazy Collections mapped with @Where

For a quick background, the @Where annotation can be put on a Collection to specify additional SQL conditions. The @LazyCollection annotation can be used to specify when and how to load a collection. For example, using LazyCollectionOption.EXTRA tells hibernate to prefer extra queries to overfilling eagerly. This means go beyond Hibernate's normal lazy loading and is useful when dealing with very large collections, since it will cause hibernate to execute COUNT statements when accessing the size of an unloaded collection, rather than load the whole collection normally. If you know that you frequently query the size of a large collection without needing any of the actual elements, this boosts performance.

However, today I had an odd situation. I have a collection that was mapped with both a @Where annotation and @LazyCollection(LazyCollectionOption.EXTRA). When querying the size of my collection, I was getting a result of 2, yet when I attempted to iterate the collection, the size was returning 0. Excuse me? How is that?

Well, after enabling SQL logging and taking a look at what was going on, I discovered that when executing a COUNT query as specified by @LazyCollection(LazyCollectionOption.EXTRA), Hibernate ignored the @Where annotation. When loading the elements of the collection, however, Hibernate correctly minded the @Where annotation. Since the @Where condition I was specifying was actually significant, these two methods returned different results.

Clearly, this isn't ideal behavior. But, since it exists, I imagine that it doesn't come up often. Also, for those paying close attention and asking why I have LazyCollectionOption.EXTRA on any collection that has 2 items, the answer is because it's in early development right now and will not have only 2 items in practice.

Off to file a Hibernate bug report....

Socket Programming with Flex and Apache Mina

Recently, I've been doing a significant amount of socket programming between a flex application and our Java servers. Namely, we were building a game in Flex with enable real-time multiplayer. Since HTTP connections are way too slow for real-time multiplayer games, we needed to use socket layer communications organized into channels. Here's a couple things I learned in the process.

Version 1
The first version I whipped together used standard TCP blocking sockets with pools of acceptor threads, worker threads, and SQS monitoring threads (since the multiplayer servers communicate with the web-application servers via Amazon SQS). This was a straight forward process, with the main difficulty being to guaranteeing synchronized access to resources shared by multiple threads. The result was more complex than desired and could handle a max load of only a few hundred concurrent users. Beyond that, the CPU time would become bogged down by switching contexts between worker threads, leaving little CPU resources for actually doing the work. Not good enough... scrap it, on to round 2.

Version 2
The second version involved using Java NIO non-blocking socket connections, which promised potential scalability to thousands of concurrent users with very few threads, and thus very little context swapping and a lot of CPU resources for actual work. The downside is that writing an NIO server is significantly more complicated and, in the words or a friend, really a pain in the ass. Fortunately, I only spent a few hours designing how I wanted to proceed before the same friend saved me days of work pointed me in the direction of Apache Mina.

Version 3
The third version was implemented with Apache Mina in roughly 2 days and has worked wonderfully thus far. The documentation for Mina is quite good, with plenty of sample applications. Mina is basically a barebones Java NIO server that can be configured to do whatever you need from a non-blocking server. It handles all of the gritty Java NIO and low-level communication and provides the programmer with standard web-application conventions like Sessions and a filter chain. Wow, that was awesome.

Implementing a server based on Mina is done with three components:

1. A protocol encoder
2. a protocol decoder
3. a protocol handler

The first two become part of the filter chain between Mina and the net, and handle the job of encoding/decoding your internal representation of a command to/from the bytes that are sent across the net. The beauty of this is that, internally, your command can be a POJO offering whatever convenience methods you desire, as opposed to the ByteBuffer it is received as. The protocol handler is the class that does all of the work. It includes such functions as sessionCreated, sessionIdle, sessionClosed, and messageReceived (which are called when their names would suggest).

Overall, the Mina version was both the simplest in design and implementation, as well as the best performing. I'm extremely happy with the choice.

On the Flex side, I decided to mimic the Mina structure by implementing a protocol encoder/decoder as well as a handler, which has worked out quite well. Since working with Mina had already involved thinking in that manner, porting the structure to Flex was the most natural procedure.

So, basic intro aside, here are a few of the "gotchas" that I encountered:

Policy File Requests
The first thing Flex will do when opening a Socket connection is send a policy file request to the host and port to which you are trying to conect. You have two options in this scenario. The first is to have your server check for <policy-file-request /> and respond with the a policy file. The second is to have Flex read the policy file from an alternate location via Security.loadPolicyFile("wherever") before attempting to open the socket connection. I found the later to be more convenient since it removed the need for my mulitplayer server to know how to handle policy file requests. There are a few restrictions with this route, though.

See here for the full reference, but the basic rules are these:

1. A policy file served from above port 1024 cannot grant access to a port below 1024
2. Policy files served via an xmlsocket (rather than an Http request to an actual crossdomain.xml) must be terminated by a null byte
3. Policy files served via an xmlsocket must include the to-ports attribute of the allow-access-from tag
4. Access to ports below 1024 can only be granted via a call to loadPolicyFile

An example call would be Security.loadPolicyFile("xmlsocket://localhost:5000"); Upon receiving a valid policy file, Flex will then establish the desired socket connection.

Flushing the Flex Socket

You would think that a call to socket.flush() would be necessary every time the Flex app writes to the socket. In practice this isn't the case, but you should always flush anyway. I only mention this because it is an oddity. Namely, in Mac OS X browsers, Flex commands written to the socket are received fine by the server without ever flushing the socket on in the Flex client. In Windows browsers, however, failing to flush the socket will cause the command not to be sent on all writes to the socket after the first (the first write will go through fine without flushing). Surprised the hell out of me when I discovered this (which I did by writing the app on a Mac and failing to flush the socket, which worked fine, until I tested it on a Windows machine and found out it wasn't working).

2 Messages Sent !(necessarily)= 2 Messages Received

In your protocol decoder (the one that receives incoming requests and converts the bytes into your internal representation of a command), you have to maintain a state to monitor where you are in receiving data. For example, you may receive one full command, only part of a command, or more than a full command at any given time. In short, this is due to Nagle's algorithm to increase network efficiency by decreasing the number of packets sent (full description). Your protocol decoder needs to know how to handle these situations.

The case is simplest when receiving a whole command, since reading all bytes available will result in one full command. However, the other cases are made easier to handle by including the length of the incoming message in it's header. For example, I defined a very lightweight protocol of 1 byte to indicate the type of message, 2 bytes to indicate the length of the message, then the message, which could be of variable size. Thus, when receiving some data, you can check the length to see how much you need, and then handle the incoming data based on how much you have and how much you need. A longer description with good examples is available here on the Mina website. I mention it here because it is definitely worth reading before writing any code, and if you neglect it, you'll end up with code that will behaive unpredictably.

That's about all I can think of right now, I started this post days ago and just came back to it today, so I might add anything else if I remember, or maybe as a new post. Happy socket programming!

J2Free 0.1 Alpha Preview

A couple months back, I wrote a post titled Java Complaints and Introducing the Redemption of Java: J2Free, in which I laid out a few of my complains about Java and introduced a new project, called J2Free, that Arjun and I had begun. Well, development of J2Free had stalled for a month or so while I was out of town and working on revamping an older site. But, much to my great pleasure, we now have the first previews of J2Free to show.

Today's preview is of the entity administration portion of the project, which uses a heavy amount of reflection to analyze the structure of persistent entities and expose a CRUD interface into your persistence layer without you ever having to write a line of code. For now, the only supported persistence provider is Hibernate. After including the J2Free jar, as well as the JSPs/CSS/JavaScript files, you can begin browsing and editing the entities stored in your persistence layer through a web interface.

Here's a view of the J2Free Entity Administration section when first loaded. The entities are listed in tabs across the top.


After clicking on a tab, the available entities of that type will be loaded via an AJAX call and populate a list along the left. Entities are loaded in batches of 100 for now, with an option to load more.


Selecting an entity on the left will open that entity for editing in form that is dynamically generated based on the fields within your entity. Read-only fields are set as such, and x-to-x mappings to other entities are represented by entity 'boxes'.


Editing a single-entity field, or a collection of entities is done via a third column, the entity browser. From there, entities can be dragged and dropped between a field and the entity browser column. Click save will save any changes, and clicking delete will delete the entity.


Any entity can be clicked to display that entities properties in read-only format (to keep confusion about which entity is being edited at a given time to a minimum).


At the top of the left-side entity list is a New button that will create a new instance of any type of entity and present you with the same form used for editing entities to populate the fields. New entities are not persisted until the saved from the edit view.

There's a lot that needs to be done; including improving the visual display of entity associations, editing one-to-many associations, proper display of embedded IDs, and improvements to handle larger amounts of entity types and larger quantities of entities. Additionally, there are plans to auto-generate Lucene indexes (with Hibernate search) on String fields to allow searching of all entities. I also want to include a box for submitting custom queries (in JPQL) that will return the results in the visual entity form.

Got a long ways to go... but it's far enough along now that it has become functionally useful to me! The most exciting part is that the reflection necessary to create an entity administration CRUD interface into a persistence structure with no knowledge of the composition is the basis of a lot of what will follow, including access controlled operations that can be exposed as components within a consumer facing site and a fully-functional REST API. Imagine that, creating an API into your site as easy as setting api.enabled=true (or for more fine-grain control, setting api.enabled.classes=com.mysite.dao.*). Excited yet?

LivePipe UI

I love Prototype, have since the day I discovered it.  Today, I found something equally awesome.  LivePipe UI, where have you been my whole life?

Adobe Flex + J2EE Security - Mismatched SessionID!

So get this.

I have a servlet called FileUploader. It takes a multipart/form-data request and proxies the file to S3. At the same time, it records some data in our database regarding the user, filename, and so forth. Simple enough.

I have a Flex 3 application that acts as a file chooser on the client side. It propogates a FileReference, which is then submitted to my FileUploader servlet via a URLRequest. Again, straight forward.

My Flex application lives in a JSP mapped to /upload. I have used declarative security to protect /upload for all HTTP methods. I have a servlet that sets a User object, containing attributes relating to the current user, to the session a after successful login, so that the User bean will be available to the FileUploader servlet.

Here's where it gets interesting...

I go to /upload and am redirected to the login page. I login and am redirected back to /upload. The JSP at /upload outputs some information from the User bean stored in the session (verifying that the bean is set correctly) as well as the JSESSIONID and the Flex application. Great so far. I use the Flex app to choose a file, which is then submitted to the FileUploader servlet. The FileUploader servlet gets the HttpSession from the request and logs the JSESSIONID. Check it the result:

JSESSIONID from the JSP: f8e64a4e62840010b6067470a346
JSESSIONID from the log: f8ec7389408a60103998c6bb75f0

WHAT!?!?! How did that happen? Is the Flex application identifying itself to my server as a different user than the browser? How can this be? The Flex documentation states that flex applications will use the underlying browser state when interacting with J2EE declarative security.

WTF Adobe? Am I surprised that it is not working according to the documentation? Not in the least.

Whenever development is going well, and especially when ahead of schedule, some Adobe product will ruin the fun. Don't believe it? Google for the Flash External Interface bug where subclass field values will "float" up to their superclass if the superclass and subclass both have a field of the same name... drove me crazy last fall...

Fantasy Congress Redesign

Just pushed out a redesigned Fantasy Congress today... check it out.

Netbeans 6.1 UI Freezes

Just the other day I decided to finally try out Netbeans 6 for real. I've tried out the various betas and release candidates for V6 over the past year or so, but each one has had some major flaw that severely limited by productivity. I figured that by 6.1 the IDE would be stable enough, but alas it is not.

Randomly, the UI will freeze up. It can happen 2 minutes into using the IDE or 6 hours into use. Sometimes it happens when I click on something, other times it happens without my input at all. In all cases, the IDE becomes non-responsive to input, but apparently still responsive enough to not show up as non-responsive in Activity Monitor. I've tried letting the IDE sit for up to 30 minutes to work itself out, but that did not help. I just end up force-quitting randomly throughout the day. Extremely frustrating.

I'm tempted to pull a Windows and revert to Netbeans 5.5, except that I put a great deal of time into customizing Netbeans indentation, syntax highlighting, etc when first setting up 6.1 and I don't want to lose those changes.

Sure, even Netbeans 5.5 was slow at times, but on the whole I find the UI of 6.1 to be far more sluggish and delayed. Ah!