entrepreneurial musings

Pascal(a)'s Triangle

2010-11-30T16:11:00.003-05:00

I've been toying with scala for a bit now. Not as much as I'd like to, and not often for work, but just because I like what little I've seen. As a bit of a college CS course throwback, I wrote a little scala script to print Pascal's triangle. The goal of these exercises is to code in good scala style, using scala as best possible. So, if you happen to be a scalawiz and see some un-scala construct, please point it out and suggest something more scala-ish.

Flex Activate / Deactivate Events on OS X

2010-02-13T14:09:00.003-05:00

In JamLegend, it would be nice to auto-pause the game when the user switches focus to another application. You'd think it would be very simple to detect if Flash player currently has focus, but the recommended methods are unreliable cross-platform.

Findings (so far)

Chrome, Firefox, and Safari receive activate and deactivate events if the browser loads the SWF directly, but NOT if the SWF is loaded inside of an HTML page.
The method of loading the SWF is inconsequential; e.g. JavaScript vs. HTML tags.
When the SWF is loaded directly, activate is not dispatched when re-focusing on the browser by clicking the flex app; a second click is required.

This is really a pain. One suggested hack is to look for the MOUSE_LEAVE event (when the user moves the mouse off of the flex app). That works well when UI elements are draggable and you need to handle the case of the mouse leaving the flex app while dragging, but it isn't a valid indicator of whether the flex app still has focus. For example, in JamLegend, the user can move the mouse anywhere and they can still play as long as the flex app has focus (thus receiving key events).

There might be a big hack somewhere, utilizing activate and deactivate events for Windows, plus a combo of JavaScript focus events and mouse position for OS X, but I haven't figured it out yet...

System Details

OS X 10.6.2
Flash Player 10,0,42,34
Chrome 5.0.307.7 beta
Safari 4.0.4
Firefox 3.5.5.

Resources

Flex Binding "Gotcha!"

2010-01-27T20:32:00.006-05:00

Somewhere online, I saw a comment saying that binding in Flex is "evil magic" and should be avoided except in the simplest of cases. Though I could see where the author was coming from in theory, it wasn't until today that I encountered this in practice.

In short, setting a bindable variable to a field of a bindable class binds the variable to the value referenced by the field, not to the reference in the bindable class. Get it? The whole pass by value vs. pass by reference thing. Here's an example (ignore the use of the default package):

Class A:

package
{
[Bindable]
public class A
{
  public var val : int;
  public var b   : B;

  public function A()
  {
    val = 0;
    b   = new B(0);
  }
}
}

Class B:

package
{
[Bindable]
public class B
{
  public var val : int;

    public function B(v:int)
  {
    this.val = v;
  }
}
}

MXML Test:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical">

<mx:Button click="onClick()" label="Click Me" />
<mx:Text text="A: {myA.val}" />
<mx:Text text="B: {myB.val}" />

<mx:Script>
  <![CDATA[
    [Bindable] var myA:A = new A();
    [Bindable] var myB:B = myA.b;

    function onClick() : void
    {
      myA.val += 1;
      myA.b = new B(3);
    }
  ]]>
</mx:Script>
</mx:Application>

At launch, the text elements both have the value 0. But, after one click of the button, what is the value of myB.val?

I don't think I'm alone on this, but maybe I am. In any case, I expected myB.val to be 3, because myB was assigned to myA.b and A is a [Bindable] class. When I changed the value of myA.b, I expected myB to reference new instance of B that myA.b now referenced.

Turns out it doesn't. After one click of the button, myB.val == 0. Why? Because myB is still bound to the original instance of B referenced by myA.b at the time myB was assigned, rather than whatever instance of B that myA.b might reference in the future.

Bummer. If only myB was a pointer...

Toggle Hibernate "Show SQL" at Runtime

2009-10-29T20:36:00.004-04:00

Okay, the title is a bit deceiving. As far as I know, you cannot modify the value of the configuration property "hibernate.show_sql." However, that does not prohibit you from toggling at runtime whether SQL statement are output to your log files.

Why would you want to do this?

Because turning on "show_sql" will flood your logs with all the SQL generated by hibernate, making it difficult to find a particular section you might be debugging or optimizing. Being able to toggle the output of SQL allows you to enable it just for the sections of code you are investigating.

First, the solution, then I'll get to why:


import org.apache.log4j.*;

...

Logger sqlLogger = Logger.getLogger("org.hibernate.SQL");

sqlLogger.setLevel(Level.DEBUG);

... do your stuff ...

sqlLogger.setLevel(Level.OFF);

So what's going on here?

The key is knowing what "show_sql" does and that it is not the only way to log the SQL statements generated by hibernate. Namely, "show_sql" sends all SQL to the standard output, but hibernate also uses Apache Commons Logging. So, as long as you configure commons logging to use an underlying logging framework that supports changing log levels at runtime, you can toggle the output of SQL by getting a hold of the "org.hibernate.SQL" logger and bending it to your will.

FragmentCache part duex: Cleaning

2009-04-10T21:57:00.002-04:00

I realized that I forgot to address an additional, critical, issue with fragment caching: cleaning out expired fragments. A fragment cache implemented as I described in my last post would update expired fragments when requested, but what about expired fragments that aren't requested?

If you externalized the cache itself, then your cache will likely support LRU or LFU expiration policies. But, if you're using a in-memory cache in the Java heap space, well, that's a memory leak. If you cache a lot of fragments, failing to clean the cache is a recipe for maxing out the old generation space and thus causing Full GC's to run back-to-back, collecting just enough space to limp along, but not do any good, until eventually your application becomes non-responsive. So, an additional requirement for an in-memory fragment cache is that you monitor the usage and have a cleaner task which runs, say, once an hour, to remove fragments that have expired and are not currently locked for update. Doing this efficiently with a large cache requires using a data structure that supports thread-safe concurrent access, so as to avoid locking the whole cache while cleaning and maintain high throughput.

The JamLegend Fragment Cache

2009-04-05T21:10:00.006-04:00

Abstract

The benefits of fragment caching are pretty well-known. If you haven't come across it before, the concept is simple: cache portions of the page to lighten database load by reducing or eliminating redundant queries. Often, this can be the "most bang for your buck" type of cache. How long a fragment should be cached is a function of the expense of generation, frequency of changes, and the frequency of access. Ideal fragments for caching are the most accessed and most expensive to generate.

The fragment cache I'm going to discuss is the implementation of the FragmentCacheTag JSP custom tag, in J2Free, the one-day-to-be-open-source framework on which JamLegend is built (you can find more info on J2Free here, and in a few of my previous blogs).

JamLegend Examples

The "Popular Today" songs lists on JamLegend are a little expensive to generate, shown on all the highest trafficked pages, and don't change too quickly. As such, we cache that list for 5 minutes, saving us a lot of queries but keeping the list rather dynamic. On the other extreme, comments change constantly and need to be much more dynamic. We cache comment threads for 30 seconds. When you have a high number of concurrent users, even a 30 second cache can save you a lot of queries without sacrificing liveliness.

The Code Samples


<cache:fragment timeout="${30 * 1000}">
Simple cache based on a timeout of 30 seconds.
</cache:fragment>


<cache:fragment timeout="-1" condition="${size(items)}">
Simple cache with no timeout, using instead an additional attribute to refresh the cache when the size of the collection "items" has changed since the cache was last updated.  This works well in situations where data changes rarely, and you want it to be available immediately upon change.
</cache:fragment>


<cache:fragment timeout="${30*1000}" disable="${devMode}">
Simple cache based on a timeout of 30 seconds but with an attribute to disable the cache if devMode is flagged.  Here, devMode is a application scope attribute set when the server starts up. It's nice to be able to selectively disable certain caches for testing, and have them enabled live without having to change any code.
</cache:fragment>

The JamLegend FragmentCacheTag also supports an additional application scope attribute to globally disable all caching. One of these days, I'm going to get around to changing the timeout to be specified in seconds, instead of milliseconds; just hasn't happened yet...

Fragment Caching "Gotchas"

Two of the first problems you'll run into with fragment caching are:

logged in vs. logged out users
concurrent cache refreshing

The first is a problem you're going to encounter simply by using a fragment cache. The second, on the other hand, you'll only run into if you're building your own fragment cache; if you use the J2Free FragmentCacheTag described here, it will be taken care of for you.

Adding Dynamic Capabilities to Cached HTML

Sometimes ideal candidates for fragment caching have display differences when users are logged in. For example, caching comment threads: logged in users see a "report" link, whereas logged out users do not. Since a fragment cache holds the generated HTML, as opposed to an object cache or query cache, it isn't easy or efficient to modify the fragment based on logged-in status (though, it's not impossible and I'd appreciate any elegant approaches). Fortunately, JavaScript comes to the rescue.

On JamLegend, a "report" link is included in the generated HTML for comment threads whether the user is logged in or not, with style="display:none;" to hide it initially. Then, if the user is logged in, we use a little JavaScript on page load to show all the report links, otherwise they stay hidden. This is the same technique used by 37Signals (see JavaScript Makes Relative Times Compatible with Caching). Since we also check server-side that a user is logged in before allowing them to mark a comment as spam, there isn't a significant downside to having the hidden spam links for users who are not logged in.

Preventing Concurrent Refreshes

If you're caching a fragment because it's expensive to generate, the last thing you want is a few hundred users concurrently refreshing the fragment triggering a few hundred concurrent expensive queries. So, how do you prevent this? The first question you have to answer is this:

When user A has triggered a cache refresh, what should happen to other users who hit the same fragment?

Should they:

(a) Head over to user A's thread, wait for user A's thread to complete the query, and then ask for the result? or,

(b) Should they just see that user A is refreshing the cache and so grab the last cached fragment and return immediately?

Both are acceptable answers, but which you choose depends on how stale you allow your fragment to be. In case A, you can serve the most recent fragment to an unlimited number of callers while still only generating the fragment once. Since user A triggered the refresh first, that thread is necessarily ahead of the other threads, so the others will still return is less time than if they were the sole thread refreshing the cache. In case B, the subsequent threads can return immediately serving the last version of the fragment while assuming that user A's thread will refresh the cache.

For J2Free's FragmentCacheTag, I chose scenario B on the reasoning that the fragment is cached to begin with because it is not crucial that it be 100% up-to-date. Ergo, what's one more minute with the old fragment? At some point, I want to add this as an optional attribute, something to the effect of:


<cache:fragment timeout="600000" waitOnRefresh="true">
... where waitOnRefresh enabled would indicate that the cache should not server the last fragment while being refreshed, but rather attach itself to the thread conducting the refresh and get the result when complete.
</cache:fragment>

Obviously, the above only applies to multi-threaded servers. There is an additional case, though, that applies to single and multi-threaded servers: what happens in a cluster? How do you prevent concurrent refreshes when you're running multiple servers? A few options are:

Store fragments in a distributed cache, like ehcache, JBoss cache, or memcached, using replication instead of invalidation. However, there is still the possibility of instance A triggering a refresh then instance B triggering a refresh before instance B received the updated fragment from instance A. This window could be decreased by replicating a lock-for-update on a fragment, rather than waiting for the complete fragment, but not closed entirely.
Using a central, synchronized cache would remove the race-condition from situation 1, but at the cost of decreased throughput. Depending on the location and network access to such a cache, it may or may not be worth it. Ping times intra-ec2-zone are only 1-2ms between instances, so it might be worth it to add a few ms to each request to avoid concurrently executing an expensive query.
Use a distributed cache, as in situation #1, but only allow 1 instance to refresh the cache. This may or may not work depending on how well you split requests between instances. You certainly wouldn't want users on instance B to continue to receive stale data because no user on instance A has triggered a cache refresh. This strategy may work well for pages that are guaranteed to be accessed on each instance.

I welcome comments on any additional solutions to preventing concurrent cache refreshing in a cluster.

Another Flex Cross-Browser Incompatibility (OS X ProgressBar display)

2008-11-20T03:26:00.004-05:00

So, Flash is supposed to solve all our cross-browser issues... only it doesn't. My most recent discovery relates to the visual look of a ProgressBar between OS X and Windows when the mode = ProgressBarMode.MANUAL and indeterminate = true.

To illustrate this, we'll start with the following MXML component:


<mx:progressbar x="10" y="10" id="progress" labelplacement="bottom" indeterminate="true" enabled="true" minimum="0" maximum="100" label="" width="210"/>

With some styling applied, the result is this:

Now, let's say that we want to use the ProgressBar for some non-indeterminate (er, determinate) purpose, like loading some data from a server. So, let's do this:


progress.indeterminate = false;
progress.mode = ProgressBarMode.MANUAL;
progress.setProgress(0,100);

someLoader.addEventListener(ProgressEvent.PROGRESS,someProgressListener);

Great. Now, someLoader will load something, and someProgressListener will keep us up-to-date about the progress. So, say we've loaded all the data and now we're doing some processing, the progress of which we don't know since our processing function is synchronous and does not dispatch ProgressEvents. So, we want to set indeterminate back to true so that our users will know something is still happening but not show possibly incorrect progress with a determinate ProgressBar. So, let's do this:


progress.removeEventListener(ProgressEvent.PROGRESS,someProgressListener);
progress.indeterminate = true;

WTF! Now our ProgressBar looks like this:

In Firefox and IE on Windows, the ProgressBar looks identical to before, which is the expected behavior. But in Safari (at least) on OS X, we have this monstrosity. What happened? Shouldn't Flash look identical in any browser? Well, apparently it doesn't.

Fortunately, this isn't unfixable. In fact, it's a very simple fix, though it makes zero sense to me why this changes the visual display. Just set progress.mode = ProgressBarMode.EVENT. Turns out that this ugly displacement only occurs in OS X if (progress.mode == ProgressBarMode.MANUAL && progress.indeterminate). Oh well, something to chalk up on the list of cross-browser differences in Flash.

On another note, remember that progress.setProgress(num,total) will not do anything unless you have set progress.mode = ProgressBarMode.MANUAL. This is another thing I think is ridiculous, but the ProgressBar will ignore any calls to setProgress if mode != ProgressBarMode.MANUAL. I understand that the different ProgressBarMode settings will instruct the ProgressBar where to expect progress changes from, but it still seems odd to fully ignore calls to setProgress WITHOUT throwing any kind of error or warning when mode != ProgressBarMode.MANUAL.

Versions: This was discovered in OS X 10.5, Flash MAC 10.0.12.36 (debug), Safari 3.2.1.

Flex / AS3 Tweening

2008-11-10T04:34:00.002-05:00

I can't believe I haven't given a shout out to TweenLite before. I love it. Absolutely, love it. For the only reason you should need to use TweenLite over Flash's built in Tween, Tweener, ZigoEngine (Fuse), Twease, HydroTween, or gTween, see this speed comparison.

The second great part about TweenLite is that it is dead simple to use.


TweenLite.to(objectToTween, howLongToTween, { propsToTween ... callbacks });

Ridiculously simple to integrate into any project.

I only have one complaint: When I haven't tweened in a few days, I always forget that the duration for a TweenLite is represented in secconds, not milliseconds. Usually, it takes me about 5 minutes to remember, since my Tween didn't happen when I wanted and instead happened 300 seconds later as I was staring at my screen wondering what the hell I messed up. I get that it's a convenience thing, but to me it's inconvenient. If TweenLite durations were specified in milliseconds, I think I'd gain 1000* 60 * 5 * 7 back to my week.

Monitoring AWS Ubuntu Instances using Munin

2008-11-01T20:50:00.005-04:00

Another munin setup blog? Why?

There are way too many blog posts out there with incorrect info on how to setup munin. My process to get it setup infuriated me enough to break from work and write the post. If any blog tells you to run:

/usr/bin/munin-cron --force-root

or any other munin command using --force-root, you should instead run like hell in the opposite direction. Doing so will only cause you hours of hell figuring out why your graphs stopped updating. It turns out that by using --force-root you just hijacked ownership of the .rrd, .png, and .html files that munin uses; for more details, see here. Furthermore, anybody that wrote a blog telling you to do that clearly didn't wait to see whether their setup was successful before posting (or they made the unwise choice of running munin as root all the time).

There also is some bad information on the web about using the setting use_node_name when setting up groups and hosts in the munin master config file. Nearly every blog I found included enabling use_node_name despite the munin documentation saying very specifically,

"You will almost never want to set [use_node_name] to anything, as setting it to "yes" tends to break things." (source)

Doesn't that just scream "Don't do it!" And yet nearly everybody is blindly recommending it. The thing is that by enabling it, you can make a setup work where it otherwise wouldn't have if you screwed up the config by using inconsistent host names between the munin master and munin nodes. So, what should you do? Is it okay to enable use_host_name? My inclination is no, because it's just a naive hack around the fact that you screwed up the config. If the people that build munin say don't use it, then I say don't use it; just learn how to configure munin correctly.

So, without further introduction, here's a more comprehensive guide to setting up a single munin master to monitor multiple munin nodes, and since I did this on AWS, there are some snippets about do's and don'ts on AWS.

On the Munin Master

Quite simple, run:

apt-get install -y munin apache2

There. Munin master is installed. Easy, no? That just setup the appropriate directories, cron jobs, and permissions, so all you really need to do next is edit /etc/munin/munin.conf but we'll wait to do that until after the nodes are ready. Additionally, it installed apache so that you can view munin reports on the web.

On all Munin Nodes

Again, simple, run:

apt-get install -y munin-node

Told you that would be easy. Great, now time to configure each node. There are only a few things we need to configure on the node, and they are all located in /etc/munin/munin-node.conf. The properties to set are:

host_name my.hostname.com allow ^120\.0\.0\.1$

The host_name setting will be the name by which your munin-node will report itself to the munin master. This is very important, though I didn't find it mentioned in any other blog, for reasons I will explain below when I show the master munin.conf config file. The short version is that it doesn't really matter what you call the node, just choose something that makes sense to you and remember it. For a database layer, something rational might be mysql.yourdomain.com for the master, and slave0.yourdomain.com and slave1.yourdomain.com for the slaves.

The allow setting specifies what master host(s) are allowed to connect to this munin node. Due to some perl specifics, this value has to be a regex, though that can come in handy. You can repeat the allow line as much as necessary.

This is where the AWS issue comes in: what value do we put for allow? Do we use the external address, such as ec2-127-0-0-1.compute-1.amazonaws.com? Do we use the internal address, domU-00-00-00-00-00-00.compute-1.internal? Or, do we use the ip address directly, say 127.0.0.1? This will resurface later, too, in how the master will connect to the nodes.

Which one you choose does matter. For example, if you use the ip address or the external AWS address, your AWS account will be charged for all data transfer and you will have to authorize port 4949 to allow the munin master to get through the firewall. Ergo, it really makes sense to use the internal representation. Furthermore, this allows you to keep the firewall closed, in which case you can put a nice little regex in the allow line to allow any of your instances to connect, thus saving yourself headache if your master external address changes, if you decide to switch the master to a different address, or for some reason add another master. So, on AWS, the easiest allow line entry is:

allow ^.*$

If you're not familiar with regex, that literally allows any munin master to connect, which means that it would be ridiculously stupid to do in any situation where your machines are not all behind a common firewall. Make sure to restart each munin-node (/etc/init.d/munin-node restart) after editing the config file to ensure the changes are live. Okay, now repeat that on as many machines/instances as you want to monitor. For more options configuring a node, see the munin documentation. When finished, continue.

Configuring the Munin Master

The config file for the munin master is /etc/munin/munin.conf. The basic idea here is to define groups and hosts. A simple group looks like:

[db.yourdomain.com] address domU-00-00-00-00-00-00.compute-1.internal

And here's what that all means: You defined a group "yourdomain.com" and then the host "db.yourdomain.com" under that group. Then, you said that the munin node located at that address is the host named db.yourdomain.com in the group yourdomain.com. So, what is all this group nonsense? The simple version is that it's just how hosts are organized, and that makes for a nice display on the web. If you want to get more complex, you can configure different behaviors for groups or show totals by group. For more info on this file, see the munin documentation.

So, on to things that matter. The name of the host (db.yourdomain.com) MUST match the value of host_name that the munin node at the address specified will report. If it doesn't, then you won't see any data collected for the node. The exception is if you enable use_node_name, then graphs will show up because that setting tells munin to fetch "all configured plugins associated with the given node, both plugins that report the nodes host_name and none at all," (source). But, since use_node_name is not recommended, just make sure that the values of host_name in munin-node.conf an the name of the host in munin.conf are the same.

Now Wait

You only have to wait 5 minutes at the most. After that, you will be able to see the data collected from the web. This is where a lot of blogs get impatient and tell you to run:

/usr/bin/munin-cron --force-root

which will update munin databases right away, but will also hijack the permissions from user "munin" to the root user, thereby denying the munin user access to those files at a later time, which will stop all updates because the cron job installed in /etc/cron.d/munin runs /usr/bin/munin-cron as munin, not root. You know, if I recommended doing that in my blog, I'd feel like a jerk for leading people astray.

Okay, patience is not one of your virtues and you don't want to wait 5 minutes? Fine. Just do this instead:

root@xxx:/root$ su munin --shell=/bin/bash munin@xxx:/root$ /usr/bin/munin-cron

There, that will let you run munin-cron as "munin", which lets you see your update right away without hijacking file permissions. Just hit ctrl+d when you're done to go back to whatever user you previously were.

Alright, hope that saves somebody some frustration.

JamLegend Invites

2008-10-21T01:20:00.004-04:00

So, it seems like there are a shortage of JamLegend invites on the web... and since I'm one of the Founder's, I figured I could give out a few here : )

So, to sign up for JamLegend, head on over to http://www.jamlegend.com/register/shinealight and start jamming!

p.s. No, I'm not going to review my own product (duh, that's the rest of the world's job!). I'm just handing out a few more invites (around 75 or so with that link). And this is to basically explain where the hell I have been the last month and a half and why I haven't written anything....

JavaMail SMTP Authentication

2008-09-09T13:09:00.006-04:00

There are a couple ways to authenticate yourself when using JavaMail APIs to send e-mail through an SMTP server.

Here's one way:



public void sendMail(String fromAddress, 
                     String recipients, 
                     String subject, 
                     String content, 
                     String contentType, 
                     String smtpHost, 
                     int smtpPort, 
                     String username, 
                     String password) {
  try {

    Properties props = System.getProperties();
    Session  session = Session.getDefaltInstance(props,null);

    MimeMessage message = new MimeMessage(session);

    message.setFrom(fromAddress);
    message.setRecipients(Message.RecipientType.TO, InternetAddress.parse(recipients, false));

    message.setSubject(subject);
    message.setContent(content,contentType);
    message.setSentDate(new Date());

    Transport transport = session.getTransport("smtp");
    transport.connect(smtphost,smtpPort,username,password);
    transport.sendMessage(message,message.getAllRecipients());
    transport.close();

  } catch (AddressException e) {
    e.printStackTrace();
  } catch (MessagingException e) {
    e.printStackTrace();
  }
}

I don't like that way, because it's no good for anything but sending via an SMTP server with password based authentication. So, if I want to send e-mail through an SMTP server that has, say, a white-listed IP policy so I don't need to provide authentication, I have to write a whole new function that will duplicate a significant part of that code. Also, that long list of arguments to the function is just ugly, but it's what you're stuck with if you abstract a function that's hard-coded for SMTP password authentication.

Here's a better way, that is abstracted nicely:


public void sendMail(Properties props, 
                     Authenticator authenticator, 
                     String fromAddress, 
                     String recipients, 
                     String subject, 
                     String content, 
                     String contentType) {
  try {

    Session session = Session.getDefaultInstance(props, authenticator);

    Message message = new MimeMessage(session);

    message.setFrom(fromAddress);
    message.setRecipients(Message.RecipientType.TO, InternetAddress.parse(recipients, false));

    message.setSubject(subject);
    message.setContent(content,contentType);
    message.setSentDate(new Date());

    Transport.send(message);

  } catch (AddressException e) {
    e.printStackTrace();
  } catch (MessagingException e) {
    e.printStackTrace();
  }
}

The function above can be used to send e-mail with all sorts of configurations. All you have to do is pass it the Properties and, if you need to, an instance of a class extending javax.mail.Authenticator. Here's an example to send an e-mail identical to the first example above:


Properties props = System.getProperties();
props.put("mail.smtp.host","smtpHost");
props.put("mail.smtp.port","smtpPort");
props.put("mail.smtp.auth","true");

Authenticator auth = new Authenticator() {
  public PasswordAuthentication getPasswordAuthentication() {
    return new PasswordAuthentication("username","password");
  }
};

sendMail(props,auth,"no-reply@mysite.com","user@site.com","Hello World","Wow, big world.","text/plain");

And, to send an e-mail without authentication (as in the IP white-list policy):


Properties props = System.getProperties();
props.put("mail.smtp.host","smtpHost");
props.put("mail.smtp.port","smtpPort");

sendMail(props,null,"no-reply@mysite.com","user@site.com","Hello World","Wow, big world.","text/plain");

The great part is that the function sending the e-mail doesn't need to know anything about how it's going to do it, since the configuration is entirely passed in as arguments via the session and the optional authenticator. There is, however, a catch that I've noticed has tripped people up all over the internet. The values you set to your Properties object must all be of class String. For example, this works


Properties props = System.getProperties();
props.put("mail.smtp.host","mail.mysite.com");
props.put("mail.smtp.port","587");
props.put("mail.smtp.auth","true");

Whereas, this does not:


Properties props = System.getProperties();
props.put("mail.smtp.host","mail.mysite.com");
props.put("mail.smtp.port",587);
props.put("mail.smtp.auth",true);

It doesn't matter that the values for mail.smtp.port and mail.smtp.auth are an int and a boolean, respectively. If they are not specified as a String, Transport.sendMessage(message) will fail.

* As a disclaimer, I typed the above code, rather than copy and paste from a program I've compiled, so I apologize if there's a typo anywhere up there that prevents it from compiling.

Socket Programming with Flex and Apache Mina (Part 2)

2008-09-07T01:50:00.004-04:00

I was responding to narup's comment on my previous post, Socket Programming with Flex and Apache Mina, and my response got so long that I decided it was worth a follow-up post. The previous post focused mainly on the parts involved in socket layer communications and some common pitfalls. This post will focus more on decisions when defining a protocol to use in socket layer communications. narup asked two things:

Whether I used binary or XML sockets, and
Some general clues on the whole architecture

The most important thing is to define your protocol. If you choose a binary protocol, some important decisions to make are,

Whether to have a header and, if so, what it will contain
Whether a command will have a fixed or variable length and, if variable, how to know the length
Whether to have the header indicate the type of command and, if so, how many types you might implement, which will help determine how many bytes you allocate for the command.

I'm a strong believer that it's impossible to anticipate every case you'll ever need, so I like to leave myself room to expand without having to change the protocol. This inclines me toward variable length commands.

Command Length

Reading a fixed length command is easy. If your protocol is 8 bytes of data, you read 8 bytes. Variable length commands, by their nature, are of an unpredictable length. Ergo, you can't just read bytes blindly. Each command must somehow transmit with it an indicator for the size of the command. Some options are terminating with a null byte, a particular sequence of bytes (like an EOF character), or indicating the length before the command.

Terminating with a null byte can be problematic if for some reason you have a null byte in your command. I don't want to rule out that possibility, so I rule out that strategy. I also don't like reading ahead without any idea of how far I'm going, so I'm just not a big fan of terminating with a sequence. This can be especially tricky when your command doesn't show up in its unique entirety, but rather comes in pieces or along with another command. Knowing how far to read by prefacing the command with byte or four indicating the length, however, is clean and simple.

I really like variable length commands because its easily expandable if you decide to 10x the size of the data on which you're operating, but it doesn't require every message to be that 10x size, especially if your data is 1/100th of that when you start out.

Headers

When you have the length prefacing the command, you might as well call it a header. I like headers. They separate operations from data. I like sending a command with a header that indicates what to do and the length of the data upon which to act. Thus, the command body contains only the data. Simple. But now we're back to the length issue again... is our header variable or fixed? I actually like fixed header lengths, because otherwise you end up with the same prefacing or termination issues I just discussed for the body. Since a header is more defined (specific to the protocol than the data) and less likely to undergo volatile changes as your development progresses (unless you change your protocol, in which case this is all out the window), it's easy to define a fixed length header with ample room for growth.

For example, if in all the planning I can only define 10 command types, I'm still likely to allocate a whole byte to specifying the command despite the fact that I could fit my 10 known types into 4 bits with 6 as yet unknown types to spare. The fact of the matter is that 8 bits vs. 4 bits is a miniscule difference but if months later I needed 17 types, I certainly won't want to have to rewrite my encoders/decoders. Same goes for the number of bytes to allocate toward indicating the length of the command; maybe you only need 2 bytes for now, but why not use a full integer just in case. Those 2 additional bytes aren't going to break the bank.

The Right Socket for Your Protocol
This is a bit of a square-peg, round-hole point. It's not usually that definite, but once you've defined your protocol, the decision about how to carry out the transmissions will mainly depend on how much you value human-readability vs. markup overhead, as well as what you're most comfortable with.

I used binary, since the server would be relaying thousands of commands a second and I wanted to be as lightweight as possible. That said, reading and writing byte arrays to process your command is a recipe for obscure bugs you'll spend hours tracking down. By writing protocol encoders/decoders on both the mina and flex sides, and logic testing those independently of the system, I was able to handle the commands in an OO fashion within my programs, but still transmit the command in a super lightweight fashion.

In my case, since I had confirmed the correctness of the encoder/decoder logic, I really didn't care at all about the human readability that XML would have provided. Plus, the overhead of the markup was undesirable both in terms of network transfer and parsing time.

Additional Note
I should probably put a disclaimer that I also prefer sending and receiving JSON over XHR instead of XML, as well as properties files instead of XML config files, haha. I did use XML sockets when writing a Mina-based server to act solely as a policy file server, though, and I could think of cases where it makes sense.

Connecting to EC2 from iSSH on the iPhone

2008-09-05T17:01:00.004-04:00

I've been debating buying an SSH app for my iPhone for some time, but wanted to wait until I found one that supported key based authentication so I could connect to EC2 instances. After careful evaluation, I decided that iSSH looked like the best one to try out, and so I purchased it from the iPhone app store yesterday.

iSSH has a great little feature to generate it's own key and then transfer that key to any machine you want, but it requires that you know the password for that machine. When it comes to EC2 instances, I've never used a password and to my knowledge, there isn't a user account with a password on them. Problem.

After posting on the iSSH google group asking if there was going to be a way to transfer a key to iSSH, Chris Jones responded that I might be able to connect if I could associate multiple keys with my AWS account. I looked into this and discovered that, no, Amazon does not let you associate multiple X.509 certificates with an AWS account. However, in this process, I realized that there wasn't anything stopping me from adding iSSH's public key to the authorized_keys file on the machine I wanted to connect to.

Here's the steps:

If you don't have one already, install a SSH server on you computer. I have a macbook pro running OS X 10.5, so all I had to do was enable remote login from the System Preferences, which enables the SSH server under the hood.
Use iSSH's transfer function to transfer the iSSH key to your computer.
Copy the iPhone iSSH key from ~/.ssh/authorized_keys (if you have multiple, it will be indicated with a comment following the key of the type "iphone-rsa-key-<SOME_NUMBER>")
Add the key to the ~/.ssh/authorized_keys file of the EC2 instance you want to connect to.

Viola! Key-based authentication allowing iSSH to connect to an EC2 machine, or any machine for that matter. If you chose to have iSSH use a password/passphrase when creating its' key, you'll have to enter this whenever you connect, but that's probably a good thing in case you lose your phone...

Glassfish: Dynamic Reconfiguration

2008-09-04T21:06:00.002-04:00

There is an option in the Glassfish server config for "dynamic reconfiguration". I've stumbled across many a forum post asking,

Why does glassfish not reflect my configuration changes? I have dynamic-reconfiguration enabled and I see the change reflected in the domain.xml, but the config changes aren't live!

The answer is simple, but not intuitive: That's not what dynamic-reconfiguration means. From the engineering document, dynamic-reconfiguration is defined as the

Ability to be able to control the output of statistical data while the server is running.

Thus, dynamic-reconfiguration does not mean that Glassfish and the JVM are dynamically reconfigurable, but rather that monitoring levels can be changed without having to restart the server. This especially useful if you want to enable monitoring when your server is bogged down to find out why, but don't want the overhead of monitoring running all the time.

So, in short, love dynamic-reconfiguration for what it is, but don't think it's a catch all for changing anything and seeing the change right away.

Sending Large Amazon SQS Messages with Typica in Java

2008-09-04T02:24:00.008-04:00

Here's a real-world case I ran into building JamLegend (http://jamlegend.com)...

Part of JamLegend is in the web-tier and part is in multiple socket-layer servers built on Apache Mina's awesome non-blocking technology. When you have only one of either, communication is easy, but when you have multiple of each, then you really need a system to communicate effectively no matter how many of each, or which particular of each, happen to be running at any given moment in time. Fortunately, Amazon SQS comes to the rescue and, combined with Typica (http://code.google.com/p/typica/) makes it brilliantly simply to distribute jobs amongst any size group of workers.

But SQS has it's own limitation. From the documentation,

Amazon SQS messages can contain up to 8 KB of text data, including XML, JSON and unformatted text.

But that isn't the whole story. More specifically, SQS messages can contain up to 8KB of UTF-16 encoded text data, which makes a big difference in the max character length of your message. So, now we have a constraint to work with, but it's not hard to imagine a case where sending larger messages is quite useful.

Ergo, a code tutorial on chunking SQS messages and some brief discussion of other considerations.

I use Typica, as a mentioned above, which is an awesome SQS utility for Java. Assume for the following that your internal protocol for SQS messages is:

header;content

Given that, chunking a large SQS message into multiple smaller messages is pretty straightforward. First, that header is going to need to accompany all of the messages. Then, we'll repeatedly form messages of the maximum size until we've sent all the data. In the following code, message would be the raw text of the SQS message you wish to send.

Some helpers:


public static final String UTF_16 = "UTF-16";

public static final int SQS_MAX_MESSAGE_SIZE = 8192;

Now onto the logic:


// Connect to the Queue
MessageQueue queue = SQSUtils.connectToQueue(SQS_QUEUE_NAME,AWS_ACCESS_KEY_ID,AWS_SHARED_SECRET);

// if message is small enough to be sent as one message, do it
if (message.getBytes(UTF_16).length <= SQS_MAX_MESSAGE_SIZE) {
  queue.sendMessage(message);
} else {
                
  // if it's too big for one message, chunk it and send as multiple

  // split the message according to the protocol, but limit it
  // in case the content portion contains a semi-colon
  String[] parts = message.split(";",2);
                
  // break off the header, this is needed for each chunk
  byte[] header = (parts[0] + ";").getBytes(UTF_16);
                
  // get the content as a byte[]
  byte[] content = parts[1].getBytes(UTF_16);
                
  // figure out how much content can be in each chunk
  int chunkSize = SQS_MAX_MESSAGE_SIZE - header.length;
                
  // create a byte[] for our max message size
  // we're going to repeatedly fill this and send the message while
  // content remains.
  byte[] bytes = new byte[SQS_MAX_MESSAGE_SIZE];
                
  // copy the header into the byte[], we'll only do this once
  System.arraycopy(header,0,bytes,0,header.length);
                
  // while there is content left, send a message
  for (int i = 0; i < content.length; i += chunkSize) {
            
  // copy the smaller of the remaining bytes or the max chunkSize chunk
  // of content into the message array, then send the message.  Form the 
  // message String from the appropriate portion of the array
  if (content.length - i < chunkSize) {
                        
    System.arraycopy(content,i,bytes,header.length,content.length - i);
    message = new String(bytes,0,header.length + content.length - i,UTF_16);
                        
  } else {
                        
    System.arraycopy(content,i,bytes,header.length,chunkSize);
    message = new String(bytes,UTF_16);
                        
  }
  // send the chunk
  queue.sendMessage(message);
}

Since the max size is a measure of UTF-16 bytes, I found it easiest to just deal with everything as a byte[], but you could do it other ways if you felt like it. The bigger thing to note is that this is really only half the solution, since now you face the task of handling chunked SQS messages on the other side. Possible solutions include:

Ensuring each message is executable alone, and thus the desired effect is the sum of the executions of each individual message. This has the benefit of not caring which of the workers receive the message, and was the solution I chose to implement.

Ensure each message is received by a particular worker, which will then reassemble the messages and execute the whole. For this to work, you really should implement an additional part to the header indicating how many chunks compose a whole and have each chunk identify itself as chunk x of y. That way, the worker will know when it has received the entirety of the message and may execute its task.

JavaScript Scrolling with Prototype or Script.aculo.us

2008-08-18T23:11:00.002-04:00

Just discovered that both Prototype's Element.scrollTo('id'), or $('id').scrollTo(), and Script.aculo.us' Effect.ScrollTo('id') will fail if the desired element is not currently visible on the page. I didn't try with visibility:hidden; but trying to programmatically scroll to an element with display:none; will result in nothing happening. Just something to remember...

JavaScript Date Formatting

2008-07-31T12:47:00.002-04:00

Discovered this awesome script for formatting dates in JavaScript. I always wanted a DateFormat in JavaScript similar to the DateFormat in Java, and now I do; exciting discovery!

Hibernate Bug when counting Extra Lazy Collections mapped with @Where

2008-07-23T11:20:00.003-04:00

For a quick background, the @Where annotation can be put on a Collection to specify additional SQL conditions. The @LazyCollection annotation can be used to specify when and how to load a collection. For example, using LazyCollectionOption.EXTRA tells hibernate to prefer extra queries to overfilling eagerly. This means go beyond Hibernate's normal lazy loading and is useful when dealing with very large collections, since it will cause hibernate to execute COUNT statements when accessing the size of an unloaded collection, rather than load the whole collection normally. If you know that you frequently query the size of a large collection without needing any of the actual elements, this boosts performance.

However, today I had an odd situation. I have a collection that was mapped with both a @Where annotation and @LazyCollection(LazyCollectionOption.EXTRA). When querying the size of my collection, I was getting a result of 2, yet when I attempted to iterate the collection, the size was returning 0. Excuse me? How is that?

Well, after enabling SQL logging and taking a look at what was going on, I discovered that when executing a COUNT query as specified by @LazyCollection(LazyCollectionOption.EXTRA), Hibernate ignored the @Where annotation. When loading the elements of the collection, however, Hibernate correctly minded the @Where annotation. Since the @Where condition I was specifying was actually significant, these two methods returned different results.

Clearly, this isn't ideal behavior. But, since it exists, I imagine that it doesn't come up often. Also, for those paying close attention and asking why I have LazyCollectionOption.EXTRA on any collection that has 2 items, the answer is because it's in early development right now and will not have only 2 items in practice.

Off to file a Hibernate bug report....

Socket Programming with Flex and Apache Mina

2008-07-20T18:18:00.024-04:00

Recently, I've been doing a significant amount of socket programming between a flex application and our Java servers. Namely, we were building a game in Flex with enable real-time multiplayer. Since HTTP connections are way too slow for real-time multiplayer games, we needed to use socket layer communications organized into channels. Here's a couple things I learned in the process.

Version 1
The first version I whipped together used standard TCP blocking sockets with pools of acceptor threads, worker threads, and SQS monitoring threads (since the multiplayer servers communicate with the web-application servers via Amazon SQS). This was a straight forward process, with the main difficulty being to guaranteeing synchronized access to resources shared by multiple threads. The result was more complex than desired and could handle a max load of only a few hundred concurrent users. Beyond that, the CPU time would become bogged down by switching contexts between worker threads, leaving little CPU resources for actually doing the work. Not good enough... scrap it, on to round 2.

Version 2
The second version involved using Java NIO non-blocking socket connections, which promised potential scalability to thousands of concurrent users with very few threads, and thus very little context swapping and a lot of CPU resources for actual work. The downside is that writing an NIO server is significantly more complicated and, in the words or a friend, really a pain in the ass. Fortunately, I only spent a few hours designing how I wanted to proceed before the same friend saved me days of work pointed me in the direction of Apache Mina.

Version 3
The third version was implemented with Apache Mina in roughly 2 days and has worked wonderfully thus far. The documentation for Mina is quite good, with plenty of sample applications. Mina is basically a barebones Java NIO server that can be configured to do whatever you need from a non-blocking server. It handles all of the gritty Java NIO and low-level communication and provides the programmer with standard web-application conventions like Sessions and a filter chain. Wow, that was awesome.

Implementing a server based on Mina is done with three components:

A protocol encoder
a protocol decoder
a protocol handler

The first two become part of the filter chain between Mina and the net, and handle the job of encoding/decoding your internal representation of a command to/from the bytes that are sent across the net. The beauty of this is that, internally, your command can be a POJO offering whatever convenience methods you desire, as opposed to the ByteBuffer it is received as. The protocol handler is the class that does all of the work. It includes such functions as sessionCreated, sessionIdle, sessionClosed, and messageReceived (which are called when their names would suggest).

Overall, the Mina version was both the simplest in design and implementation, as well as the best performing. I'm extremely happy with the choice.

On the Flex side, I decided to mimic the Mina structure by implementing a protocol encoder/decoder as well as a handler, which has worked out quite well. Since working with Mina had already involved thinking in that manner, porting the structure to Flex was the most natural procedure.

So, basic intro aside, here are a few of the "gotchas" that I encountered:

Policy File Requests
The first thing Flex will do when opening a Socket connection is send a policy file request to the host and port to which you are trying to conect. You have two options in this scenario. The first is to have your server check for <policy-file-request /> and respond with the a policy file. The second is to have Flex read the policy file from an alternate location via Security.loadPolicyFile("wherever") before attempting to open the socket connection. I found the later to be more convenient since it removed the need for my mulitplayer server to know how to handle policy file requests. There are a few restrictions with this route, though.

See here for the full reference, but the basic rules are these:

A policy file served from above port 1024 cannot grant access to a port below 1024
Policy files served via an xmlsocket (rather than an Http request to an actual crossdomain.xml) must be terminated by a null byte
Policy files served via an xmlsocket must include the to-ports attribute of the allow-access-from tag
Access to ports below 1024 can only be granted via a call to loadPolicyFile

An example call would be Security.loadPolicyFile("xmlsocket://localhost:5000"); Upon receiving a valid policy file, Flex will then establish the desired socket connection.

Flushing the Flex Socket

You would think that a call to socket.flush() would be necessary every time the Flex app writes to the socket. In practice this isn't the case, but you should always flush anyway. I only mention this because it is an oddity. Namely, in Mac OS X browsers, Flex commands written to the socket are received fine by the server without ever flushing the socket on in the Flex client. In Windows browsers, however, failing to flush the socket will cause the command not to be sent on all writes to the socket after the first (the first write will go through fine without flushing). Surprised the hell out of me when I discovered this (which I did by writing the app on a Mac and failing to flush the socket, which worked fine, until I tested it on a Windows machine and found out it wasn't working).

2 Messages Sent !(necessarily)= 2 Messages Received

In your protocol decoder (the one that receives incoming requests and converts the bytes into your internal representation of a command), you have to maintain a state to monitor where you are in receiving data. For example, you may receive one full command, only part of a command, or more than a full command at any given time. In short, this is due to Nagle's algorithm to increase network efficiency by decreasing the number of packets sent (full description). Your protocol decoder needs to know how to handle these situations.

The case is simplest when receiving a whole command, since reading all bytes available will result in one full command. However, the other cases are made easier to handle by including the length of the incoming message in it's header. For example, I defined a very lightweight protocol of 1 byte to indicate the type of message, 2 bytes to indicate the length of the message, then the message, which could be of variable size. Thus, when receiving some data, you can check the length to see how much you need, and then handle the incoming data based on how much you have and how much you need. A longer description with good examples is available here on the Mina website. I mention it here because it is definitely worth reading before writing any code, and if you neglect it, you'll end up with code that will behaive unpredictably.

That's about all I can think of right now, I started this post days ago and just came back to it today, so I might add anything else if I remember, or maybe as a new post. Happy socket programming!

J2Free 0.1 Alpha Preview

2008-06-30T14:43:00.005-04:00

A couple months back, I wrote a post titled Java Complaints and Introducing the Redemption of Java: J2Free, in which I laid out a few of my complains about Java and introduced a new project, called J2Free, that Arjun and I had begun. Well, development of J2Free had stalled for a month or so while I was out of town and working on revamping an older site. But, much to my great pleasure, we now have the first previews of J2Free to show.

Today's preview is of the entity administration portion of the project, which uses a heavy amount of reflection to analyze the structure of persistent entities and expose a CRUD interface into your persistence layer without you ever having to write a line of code. For now, the only supported persistence provider is Hibernate. After including the J2Free jar, as well as the JSPs/CSS/JavaScript files, you can begin browsing and editing the entities stored in your persistence layer through a web interface.

Here's a view of the J2Free Entity Administration section when first loaded. The entities are listed in tabs across the top.

After clicking on a tab, the available entities of that type will be loaded via an AJAX call and populate a list along the left. Entities are loaded in batches of 100 for now, with an option to load more.

Selecting an entity on the left will open that entity for editing in form that is dynamically generated based on the fields within your entity. Read-only fields are set as such, and x-to-x mappings to other entities are represented by entity 'boxes'.

Editing a single-entity field, or a collection of entities is done via a third column, the entity browser. From there, entities can be dragged and dropped between a field and the entity browser column. Click save will save any changes, and clicking delete will delete the entity.

Any entity can be clicked to display that entities properties in read-only format (to keep confusion about which entity is being edited at a given time to a minimum).

At the top of the left-side entity list is a New button that will create a new instance of any type of entity and present you with the same form used for editing entities to populate the fields. New entities are not persisted until the saved from the edit view.

There's a lot that needs to be done; including improving the visual display of entity associations, editing one-to-many associations, proper display of embedded IDs, and improvements to handle larger amounts of entity types and larger quantities of entities. Additionally, there are plans to auto-generate Lucene indexes (with Hibernate search) on String fields to allow searching of all entities. I also want to include a box for submitting custom queries (in JPQL) that will return the results in the visual entity form.

Got a long ways to go... but it's far enough along now that it has become functionally useful to me! The most exciting part is that the reflection necessary to create an entity administration CRUD interface into a persistence structure with no knowledge of the composition is the basis of a lot of what will follow, including access controlled operations that can be exposed as components within a consumer facing site and a fully-functional REST API. Imagine that, creating an API into your site as easy as setting api.enabled=true (or for more fine-grain control, setting api.enabled.classes=com.mysite.dao.*). Excited yet?

LivePipe UI

2008-06-27T03:39:00.002-04:00

I love Prototype, have since the day I discovered it. Today, I found something equally awesome. LivePipe UI, where have you been my whole life?

Adobe Flex + J2EE Security - Mismatched SessionID!

2008-06-19T02:44:00.002-04:00

So get this.

I have a servlet called FileUploader. It takes a multipart/form-data request and proxies the file to S3. At the same time, it records some data in our database regarding the user, filename, and so forth. Simple enough.

I have a Flex 3 application that acts as a file chooser on the client side. It propogates a FileReference, which is then submitted to my FileUploader servlet via a URLRequest. Again, straight forward.

My Flex application lives in a JSP mapped to /upload. I have used declarative security to protect /upload for all HTTP methods. I have a servlet that sets a User object, containing attributes relating to the current user, to the session a after successful login, so that the User bean will be available to the FileUploader servlet.

Here's where it gets interesting...

I go to /upload and am redirected to the login page. I login and am redirected back to /upload. The JSP at /upload outputs some information from the User bean stored in the session (verifying that the bean is set correctly) as well as the JSESSIONID and the Flex application. Great so far. I use the Flex app to choose a file, which is then submitted to the FileUploader servlet. The FileUploader servlet gets the HttpSession from the request and logs the JSESSIONID. Check it the result:

JSESSIONID from the JSP: f8e64a4e62840010b6067470a346
JSESSIONID from the log: f8ec7389408a60103998c6bb75f0

WHAT!?!?! How did that happen? Is the Flex application identifying itself to my server as a different user than the browser? How can this be? The Flex documentation states that flex applications will use the underlying browser state when interacting with J2EE declarative security.

WTF Adobe? Am I surprised that it is not working according to the documentation? Not in the least.

Whenever development is going well, and especially when ahead of schedule, some Adobe product will ruin the fun. Don't believe it? Google for the Flash External Interface bug where subclass field values will "float" up to their superclass if the superclass and subclass both have a field of the same name... drove me crazy last fall...

Fantasy Congress Redesign

2008-06-18T18:05:00.002-04:00

Just pushed out a redesigned Fantasy Congress today... check it out.

Netbeans 6.1 UI Freezes

2008-06-18T02:23:00.003-04:00

Just the other day I decided to finally try out Netbeans 6 for real. I've tried out the various betas and release candidates for V6 over the past year or so, but each one has had some major flaw that severely limited by productivity. I figured that by 6.1 the IDE would be stable enough, but alas it is not.

Randomly, the UI will freeze up. It can happen 2 minutes into using the IDE or 6 hours into use. Sometimes it happens when I click on something, other times it happens without my input at all. In all cases, the IDE becomes non-responsive to input, but apparently still responsive enough to not show up as non-responsive in Activity Monitor. I've tried letting the IDE sit for up to 30 minutes to work itself out, but that did not help. I just end up force-quitting randomly throughout the day. Extremely frustrating.

I'm tempted to pull a Windows and revert to Netbeans 5.5, except that I put a great deal of time into customizing Netbeans indentation, syntax highlighting, etc when first setting up 6.1 and I don't want to lose those changes.

Sure, even Netbeans 5.5 was slow at times, but on the whole I find the UI of 6.1 to be far more sluggish and delayed. Ah!

Prototype DOM Extension Browser Quirks

2008-04-23T10:37:00.006-04:00

Documentation is really helpful. Seriously. In fact, anybody who failed to read the section entitled 'Native Extensions' in the article on How Prototype extends the DOM would be surprised to find that much of their code would not work in IE while working fine in many browsers. Until recently, I was one of those people that had not read that article. This is an easy pitfall for developers whose primary development browser is Firefox, or even Opera.

The problem is that Firefox allows adding methods to prototype of native objects, such as HTMLElement. Thus, all DOM extensions are are available on all elements, including dynamically created elements. However, this is not the case with IE. The syntactic shorthand DOM extensions I'm referring to are constructs such as myDiv.getWidth() instead of the longer Element.getWidth(myDiv).

IE does not let anyone touch HTMLElement.prototype, thus DOM extensions are not available by default on elements in IE. To get around this, Prototype has two constructs to enable DOM extensions on an element:

Element.extend('my_div'), or
$('my_div')

Prototype's $ function automatically extends any element passed through it, while still being smart enough to not doubly extend any element. Thus, assuming that you access all elements through $('elem'), you won't have to worry about the Element extensions not being available on a given object.

The simplest, though more annoying solution, is to not use syntactic extensions and use Element methods directly. This means always using Element.doSomething(myElem,...) rather than myElem.doSomething(...). That's not an ideal solution, and we can do better assuming we can remember to extend elements ourselves, which is as simple as calling Element.extend(myElem), or wrapping access to the element in a $ call, such as $(myElem).

There are three sticky situations to watch out for: (1) dynamically created objects, (2) going up or down the DOM chain from an extended element to non-extended elements, and (3) browser objects.

Dynamically Created Elements
In IE, dynamically created objects are not extended by default, unlike Firefox. In Firefox, the following will work:


var myDiv = document.createElement('div');
myDiv.setStyle({width: '50px'});

Note the DOM extension .setStyle({}) is available by default. IE requires an additional step to achieve the same:


var myDiv = document.createElement('div');
Element.extend(myDiv);
myDiv.setStyle({width: '50px'});

It's easy to forget the call to Element.extend(myDiv) when dynamically creating elements, especially because the code will work just fine without that in a few more developer friendly browsers.

Traversing the DOM
Extending an element does not confer that extension upon elements referenced from said element. parentNode is a big one to watch out for. It's really easy to want to do:


$('myDiv').parentNode.hide();

That will work fine in Firefox, but only because all elements are extended by default (as described above). In IE, you'll have to remember to extend the parent element as well:


$($('myDiv').parentNode).hide();

I agree, it is annoying. On the good side, since $ is smart enough not to re-extend a previously extended element, we don't have to worry about browsers that extent all elements by default running into problems from the constructs necessary to make IE behave nicely.

Browser Objects
document is an object, and thus must be extended to enable the syntactic shortcuts. In Firefox, the following will work fine:


document.body.insert(myDiv);

In IE, that will result in the error message, "Object doesn't support this property of method." To make it work in IE, just wrap the call:


$(document.body).insert(myDiv);

On the whole, I like the syntactic shortcuts, but as everyone stresses, you have to test the code in multiple browsers. If my code using Prototype works in Firefox but not IE, these are the first three things I look for. Happy scripting!

Beer Quest: 22 Down, 478 to go

2008-04-22T17:09:00.002-04:00

For my birthday last weekend, a friend gave me a copy of Michael Jackon's Great Beer Guide to 500 Classic Brews. This Michael Jackson is the beer aficionado, not the singer. And so, I'm going to use the book as a checklist to expand my beer knowledge. I have had 22 of the 500 beers before receiving the book, and #23 is currently in the fridge. I am thoroughly looking forward to the next 478 beers!

Here's the current list of beers in the book that I have tried:

Alaskan Amber
Alaskan Smoked Porter
Bitburger Premium Pils
Boon Geuze
Budweiser Budvar (sold in the US under the name Chechvar)
Fanziskaner Dunkel Hefe-Weissbier
Gorden Biersch Marzen
Gouden Carolus
Guinness Extra Stout
Hoegaarden Witbier
Huyghe Delirium Tremens
Kasteel Bier Donker
Konig Pilsener
Kostitzer Schwarzbier
Newcastle Brown
North Coast Old Rasputin Russian Imperial Stout
Ommegang
Paulaner Hefe-Weizen
Pilsner Urquell
Red Hook ESB
Saison Dupont
Schloss Eggenberg Urbock 23
Singha

I may not have mentioned it before, but my housemates and I also brew our own beer. We started last summer and it has been an awesome time. So far we're brewed an English Brown, an Irish Red, a German Altbier, and an IPA. Currently underway is a Belgian Triple, and just last night we kegged a Germain Weizenbier. Next up is a Maibock. Home-brewing is certainly a lot more work than going to the store and picking up a nice 6-pack, but it's really enjoyable. Kind of like how steak houses have great food, but sometimes it's way more fun to stand outside in the rain and grill that steak yourself. So, in addition to tech posts and random thoughts, I think there will be an increased number of posts relating to beer in the future.

Parallels Desktop Config Recovery

2008-04-15T22:45:00.004-04:00

Do not run OS X Disk Utility's Erase Free Space feature while Parallels Desktop is running. I did that today, and my Parallels VM config file disappeared. Details follow.

Erasing free space is accomplished in a rather interesting way. Disk Utility creates one really huge file to fill up the remaining space on your hard drive, then proceeds to write over that space with 0's as many times as you choose (1-pass, 7-pass, or 35-pass). Thus, at the peak of the process, your system will indicate that your hard drive is full and suggest that you clear space (the free space alert has no awareness of Disk Utilities activities and treats the new large file like any other file). The technique involved is actually a pretty creative way of letting the operating system unconsciously collect all the free space without having to explicitly write a new routine for collecting free space. After all, allocating space for a file is just a matter of finding enough free space for said file.

Anyhow, I had Parallels running while this process was going on and I had forgotten that Parallels requires a certain amount of free space to operate. When the remaining free space dropped below the required amount (in my case 512MB, though it may vary depending on your VM configuration), the VM alerted me and presented two options: clear enough space and continue operating, or shutdown the VM. I chose the latter since I wasn't actually using right then. And that was the final blow...

When I went to restart Parallels later, I was prompted with the screen to install a new VM. WTF Parallels? What happened to the VM I already had installed and have been using for months? After some investigating, I discovered that the config file for my VM was empty. That's right, empty. The file remained, but its contents had disappeared. Not sure why, but it was disappointing to say the least.

The fact that I am a Mac user should suggest that my tolerance for Windows XP installs is rather low, so clearly I didn't want to reinstall the VM... if only I had somebody else's config file...

At this point, I had no idea what the structure of a Parallels VM config file was, but I figured that investigating that would be far quicker than an XP install. Fortunately Arjun also uses Parallels, so he sent over his config file, and I broke out the mad doctor goggles and dove in.

Turns out the VM config file is a simple properties file with [] section headers exactly like a my.cnf MySQL server/client config file. Furthermore, there are very few properties in there that really matter in terms of the VM running or not, the rest just deal with how it runs. Here they are:

App Version = 3.0.5582
...
Disk 0:0 image = winxp.hdd

The first could be problematic because the config file is going to need to list an App Version compatible with the version of Parallels you have installed. For example, if you upgraded to Parallels version 3.x.y from version 2.x.y, the installer reformatted your existing VMs to work with version 3.x.y. If you have version 2.x.y of Parallels, but your config file indicates version 3.x.y, I'd suspect you'd run into some problems...

The other important property there is just a reference to the name of the Parallels .hdd disk image. Since Arjun has the same version of Parallels as me, all I had to change was the name of the disk image.

The only remaining changes were smaller, but I suspect would have caused problems if I hadn't changed them too. Namely, in the [Shared folders] section, there can be references to shared folders, in which case you'll need to change the location to match your file structure. I had the following, but yours may be different.

Folder0 path = /Users/Ryan
...
Folder1 path = /Users/Ryan/Documents

Also, if you remove shared folder mappings, make sure to adjust this property accordingly:

Shared folders count = 2

After making those changes, I fired up Parallels and was back up and running just like before! Overall, it was a remarkably painless process to recover the VM config file if you happen to have lost it. At the very least, those few changes will get your VM back up and running, and then you can use the config GUIs to restore your specific preferences.

As a disclaimer for comments, no, I did not GLOG or RTFM for this, so if it's out there already somewhere, sorry for the duplication. In all, it was probably faster to try than it would have been to find the answer online anyway.

Java Complaints and Introducing the Redemption of Java: J2Free

2008-04-02T20:44:00.003-04:00

I'm a big Java fan, so this list is by no means a declaration of war against Java. There are, however, a few things about the language that I find very annoying:

1) Strongly typed. I enjoy types, I really do. Sometimes I even love the strongly typed nature because of the kick ass things that can be done with generics and reflection. Other times, I crave the flexibility of loosely typed languages, such as javascript. I've found that loose typing can lend itself to more elegant solutions.

2) No anonymous functions. If Java treated functions as first-class objects and allowed dynamic creation of functions at runtime, I would be in heaven. Sometimes, I find scenarios where the most elegant, and quickest to implement, solution would make use of an anonymous function. Other times I really wish I could pass functions around as arguments. This would make my life more happy.

3) == vs. .equals() for equality comparison. I'm not a big fan of pointers in C and C++. I prefer array notation in Java, for example. However, C and C++ pointers make equality comparison very easy and clear. Namely, it is quite obvious when you are comparing by memory location and when you are comparing by value. In Java, it is easy for this to escape me momentarily and cause me to spend 5 minutes debugging something so simple as an equality comparison. For example, comparing int primitive data types with == compares by value. However, if you happen to be using the int wrapper class, java.lang.Integer, then equality comparisons with == default to comparison by memory location since Integer is an Object. To compare two Integers by value, you have to use the .equals() method of Integer. I would much prefer if == in Java defaulted to comparison by value, and .equals() (or something like .objectEquals()) was used for memory location comparison. I think it would be more straight forward at least when it comes to consistency with the Java primitive data types.

4) I can't ever find any hosting providers that support Java application servers. This is a pain because I end up setting up all of our servers, taking me away from tasks I find more interesting, challenging, engaging and, in general, worthwhile.

In all fairness, I still think the reasons against Java out there are highly exaggerated. The "experiments" that show Java is less agile, clunkier, and slower to develop all seem to be written by people that don't know Java. The first time you're trying any framework it's confusing as hell, simple as that. Java has some basic configuration required for any web-application. I've been doing it long enough now that I've developed my own framework-ish libraries, which allow me to skip right past all the parts of Java people complain about and jump right into the creation aspect.

On that note, I've begun development of a consolidation of my tools with the help of one of my Co-Founders at Publi.us, Arjun. We are calling the project J2Free. It's goal: redemption of Java when compared with frameworks such as Rails & Django. We started out by thinking up all of the things that annoy us when starting a new Java web-application. The first step in the J2Free is a smooth process by which users can bypass the annoying crap and jump straight into development.

J2Free itself will utilize J2EE + JPA (with Hibernate as the persistence provider) + MySQL in a classic MVC architecture. As a disclaimer, I despise JSF with it's massive xml configuration files and odd String return values from controller classes. So, don't expect any of that here. J2Free will feature time saving procedures such as auto-generation of an admin section and fun little tidbits such as taglibs for pagination of datasets, templating, caching, and display formatting (Date formatting, quote escaping, etc).

Additionally, J2Free will make it easy to automatically open controller (or subsets of controllers, such as read-only functions) classes via API. Want an API? "Click. Done." Since I am a big fan of DWR, I plan on including that same "Click. Done." ease for enabling AJAX via DWR.

Clearly, this is all very early, but we're getting started with planning so now is the best time to throw out ideas! When complete, we plan on making J2Free available freely and we will be open sourcing the project at some point as well. Anyway, something to look forward to!

Interesting 30-second Snippet for Emailing Websites to Friends

2008-04-01T14:00:00.006-04:00

So, one of my co-founders was telling me today about a startup out there that basically is del.icio.us but designed for sending articles to your friends. I realized that I could implement 95% of the "sending articles to your friends" functionality in 30 seconds. The code:


<a href="javascript:location.href='mailto:?subject=Check Out: \'' + document.title + '\'&body=Thought you might like this: ' + window.location.href;">Send to Friends</a>

It's just a Del.icio.us-style bookmarklet that opens your mail client with the current page title in the subject and the link to the current page in the body. Simple, clean, done. Now I can just click that when I want to email a page I'm on to friends.

For simplicity, you can drag this link to your bookmarks bar to add the feature yourself.

E-mail to Friends

PHOTOREE Image Recommendation System

2008-02-27T13:40:00.002-05:00

For your mindless clicking enjoyment...

http://www.photoree.com

This is a fun little site I discovered the other day. You give the thumbs up or down to photos and it recommends other photos that it thinks you might like. Sounds like straight forward user-based collaborative filtering, though maybe it's more technically advanced and has some sort of image analysis system and is actually using item-based collaborative filtering...

MacBook Pro Multi-touch Trackpad

2008-02-27T12:28:00.006-05:00

I have an ongoing debate with Arjun over the new MacBook Pro with the Multi-Touch trackpad (the kind first introduced on the MacBook Air).

The Question
Are the additional multi-touch input gestures in the new MBP a result of a software/firmware or a hardware upgrade?

My Position
The hardware of the trackpad must be different in the new MBP, otherwise Apple would have released the upgrade as a firmware/software update, benefiting a larger audience, rather than as a new feature on a new machine.

Arjun's Position
The hardware is the same, and Apple chose not to release the new feature as a firmware/software update so as to exaggerate the "new" features of the new MBP and further distinguish the new model from the current model.

Admittedly, my view holds Apple's integrity in high-esteem. If I'm wrong, that esteem will no longer be so high. I hope that Arjun is right, if only because that would mean Apple could (would?) eventually release the same firmware/software for the last-gen MBP, or some 3rd party could attempt to do so.

The Evidence
First, from pictures, the trackpad on the new and last-gen models are visually the same, as opposed to the MacBook Air trackpad which is visually larger. Although not conclusive either way, it suggests a similarity in hardware.

Second, the last-gen model trackpad can sense multiple sources of input; two-finger tapping is distinguished from one-finger tapping for right and left click respectively, and two-finger scrolling is recognized for page scrolling.

As further evidence (and the most convincing in terms of Arjun's argument), the multi-touch capability of the last-gen models is independent of finger position, meaning that a two-finger tap with fingers side by side is recognized just as well as a two-finger tap with fingers spread to opposite sides of the trackpad. This position indifference holds true for two-finger scrolling; dragging two spread fingers works exactly the same as dragging two touching side by side fingers.

The logical inference from the above is that if two fingers spread wide can be dragged in a straight line for scrolling, then those same two fingers spread wide should be able to be rotated in a circular motion for, say, rotating a photo in iPhoto. Because of the hardware necessary to detect two, spread fingers moving simultaneously is already in place in the last-gen model, it isn't too big a jump to suspect the additional gestures added in the new MBP are a result of upgraded firmware/software.

However, one caveat to note is the nature of multi-touch input in the last-gen modes. Namely, all supported multi-touch gestures involve parallel motion; (1) two-finger tapping is recognized only when both fingers tap simultaneously, as opposed to having one finger in contact with the trackpad and then tapping with the other, and (2) two-finger scrolling involves both contacts with the hardware moving in the same direction.

The exception to my parallel inference in the last-gen models is that two fingered scrolling does not require both fingers to be moving, but works just fine if one finger is in contact with the trackpad and the other is moving.

The gestures introduced in the new MBP break the need for parallel motion; 'pinching' is convergent motion, 'spreading' is divergent, 'rotating' is inverse motion, etc. The exception is three fingered 'swiping', though the ability to sense more than 2 sources of simultaneous input may be a hardware improvement in the new MBP.

At this point, it's all speculative, though I hope that somebody with more knowledge in this subject will chime in resolve the debate. Otherwise, maybe we'll hear from Apple some day.

Why I Don't Use My Del.icio.us Account

2008-02-22T14:40:00.011-05:00

Introducing the Need
I have a del.icio.us account. I've used it sparingly from time to time and I actually think the concept is very good. Depending on my mood and task at hand, I'll browse the web with Firefox, Camino, or Safari. I wouldn't use Firefox except for the Firebug plugin, so that's mainly for developing. The problem is that there's no predicting = what browser I'm going to be using when I stumble across a website that I want to bookmark. On the flip side, there are certain sites I want to have bookmarked in all my browsers. As a result, the browsers share certain bookmarks but each have their own unique finds and so when recalling a site I want to revisit, I can never remember which browser I was using at the time, and thus have to check all three to find the bookmark again.

That is why I wish del.icio.us did not suck so much. If not for the forthcoming problems, it would solve my bookmark woes.

Now to the Problems
First, I dislike the homepage. It assaults my eyes with so much web-safe blue that it all blends together and I feel like I'm looking at a lava-lamp with white liquid and blue lava. The addition of pictures smoothed it out a bit, but it's still a painful experience.

On it's own, that would not be enough to keep me away, which brings me to my second complaint.

Exhibit A:

#9999FF is not a good font-color on top of various shades of pink. Is it too much to ask for a little contrast? I agree, it is a very cool effect to calculate the color of some element based on collected data (I've even done it for a project). Nevertheless, there is always one overriding concern: Make sure that text in the variable color element is still readable.

Even so, the contrast issues wouldn't keep me away, especially since I don't particularly care how many other people have saved my bookmark. My third complaint is the real reason I don't use my del.icio.us account.

Fatal Flaw
The bookmarklet is useless for what I want to do with it.

Sure, I can just drop it in my bookmarks bar and click it when I'm on a page I want to add. But what does this do? It takes me to del.icio.us where I have to sign into my account to save the bookmark. This is a hassle. When I use the bookmarklet, it's because I don't want to go to del.icio.us!

I never visit del.icio.us, so I'm never signed in. Furthermore, I don't want to have to sign in just to save a bookmark. I don't want to use del.icio.us right then and there. I just want to throw something at them real quick, and then come back to it later.

One of the problems is that browser security constraints disallow sending an asynchronous HTTP request by clicking on a bookmarklet. This would be the ideal solution, since it would not even require opening a new window or moving away from the current page. However, as it is not plausible, I'll focus on the sign-in issue.

The fact of the matter is there exists a simple, safe, secure alternative. All the bookmarklet does is redirect the browser to del.icio.us's page for submitting links and passes along information about where you just were (specifically, the url and the window title). The base dilemma is that the bookmarklet is a generic script-- that is, each user has the exact same code in their bookmarklet. The solution is to make the bookmarklet specific to each user.

Del.icio.us could tack on two more parameters to the query string above that would contain (1) either your user ID or username (both public information, so no security issues yet), and (2) a hash that would be (a) unpredictable, and (b) unique to each user. Basically, it's like having a secret piece of information that only you and del.icio.us know and that is only valid for one user and for one-way communication to del.icio.us. For those familiar with public/private key security, this should sound similar. That way, when clicking the bookmarklet, del.icio.us already knows who you are and doesn't have to ask you to sign in first.

Potential Problems
Alright, so what about multi-user environments? First, del.icio.us bookmarklets are probably not going to be found on massively public computers such as libraries and schools. To accommodate the rare case, del.icio.us could keep the generic version available. Second, in the home, different user accounts should accompany different users, thus the operating system manages user settings. Third, what if somebody is on my computer and clicks my "post to del.icio.us" bookmarklet? Just add a "Not userA?" link on the post page that allows a user to login to a different account, problem solved.

Short Version
I like the idea of del.icio.us, but it's just as much of a hassle as synchronizing my local browser bookmarks until there is a solution that does not require me to go to del.icio.us to submit a bookmark.

Amazon.com's Design Usability Problems

2008-02-13T12:48:00.003-05:00

I have issues with Amazon.com. Namely, it always takes me ridiculously longer than necessary to accomplish anything on the site.

The Sign In Link
At the top of the page is a line reading "Hello. Sign in to get personalized recommendations." For some reason, the part of that line that links to the login page is "personalized recommendations" rather than "Sign in" or even the whole sentence. My reasons for being on the site had nothing to do with personalized recommendations. Ergo, I stayed away from the mysterious "personalized recommendations" link and was completely frustrated by the time I finally clicked it out of desperation and accidentally stumbled upon the login page.

Gift Card Purchases
I wanted to use a gift card that I had applied to my account to buy a song. For the life of me , I could not find the option to use the gift card for the purchase. I was signed in. I found the song. I clicked purchase. There was no option to select a gift card from my account as the payment method. There was an option to apply a new gift card, but none to use a previously applied gift card. Eventually I said, "f*** this, it's only $1, I'll just use my credit card."

My credit card, however, was never charged. In investigating why, I discovered that gift cards on your account are automatically applied to purchases before whatever payment form you provide.

That leads to a few thoughts:

Why not let me choose to use a gift card, or choose not to? Seems odd to force me to use a gift card just because I applied it to my account for safe-keeping.
If they're going to force me to use the gift card first, why not check to see if I have one before making me input my credit card information, thus saving me the hassle of filling out an unnecessary form.

Well, that's the shortlist of my recent complains regarding Amazon.com's usability...

Update

I was recently told this was, "because you're not dumb enough to appreciate Amazon's idiotic structure." I'll take that as a compliment.

Why AugmentedFragments.com for the domain?

2008-02-13T11:30:00.003-05:00

I thought that the reasons behind the domain name deserved a post. I meant to do this a while ago to christen the blog, but just didn't get around to it, so here we go.

First the Why
A few trivial reasons:

My name is common enough that every combination I could justify owning as a domain name was taken long before the idea occurred to me
I wanted something unique, reflective of me, not reflective of the times
I wanted a .com domain
AugmentedFragments.com was available (likely because of it's obscurity, length, and likelihood of misspelling)

Now the How
At the most basic level, this blog is a partial record of my thoughts. Fragments, you could say, of my thought process. To take it a step further, I don't always stop to explain my thoughts to myself, but in this blog I do attempt to clarify and analyze. Ergo, the fragments are modified, expounded upon, hopefully improved.

From "improved" I inferred progress in the upward direction and then my background as a musician took over. Up translated as a higher pitch. Fragments became chords, as a piece of music may be digested down to the building blocks underlying the complexity of the melody, rhythm, intonation, and color.

So what happens if you take a chord and "move it up"? If you maintain the same chord, then "moving up" would probably just be playing the first or second inversion. That definition doesn't fit well, since essentially it means reordering the notes within the chord. All the same notes are there, nothing has changed but the order.

I view my posts as more than a reordering of my thoughts, so I needed something that modified the root chord, and yet still "moved it up." And thus I arrived on augmented chords. An augmented triad is a chord with a raised fifth (basically, the high note of the chord is a little higher than usual) resulting in two symmetrical intervals, between the first and third, and the third and fifth, that split the octave evenly.

Augmented chords are dissonant, unresolved, but that's not the implication I am trying to convey. An additional feature of an augmented chord is that, beginning with a major augmented triad in an arbitrary (I'll explain why arbitrary shortly) root position, the first inversion is a root position augmented triad beginning on the third and the second inversion is a root position augmented triad beginning on the fifth. Thus, the root of an augmented chord is ambiguous since it could be any one of the three notes forming the triad. This may seem problematic, but is actually quite wonderful if you happen to be an accompanying musician since it increases the available chord shapes which will "sound good."

Thus, augmented fragments are supposed to be portions of my thought process which have been both clarified and obscured so that they are easily understood and readily applicable to a variety of situations.

I'm not going to give you a fish here, but hopefully you'll come away with a few more hints on how to fish.

Compressing and Obfuscating JavaScript (with Prototype & Script.aculo.us)

2008-01-18T09:48:00.000-05:00

Abstract
My methods, trials and eventual solution for optimizing the delivery of JavaScript files via minification, obfuscation, and compression, including examples with notoriously problematic to minify libraries such as Prototype and Script.aculo.us. Some of my explanation will work on any platform, although my eventual sweet solution is Java specific and relies on a JSP taglib.

Introduction
Last fall, my company developed an election game for the primaries and caucuses called Kingmaker, which we launched in the middle of December. We built portions of the site in Adobe Flex. Although this provided some cool functionality, it increased our page sizes significantly. So, I began my search for ways to shrink down everything else on the site to compensate for the tremendous increase caused by Flex apps.

First, we compressed the images as far as possible before losing too much quality. Next, I saw to the shrinking of our JavaScript files. This presented a series of problems:

I like Prototype. I wanted to keep it. But it's huge (124K uncompressed).
I like Script.aculo.us. I wanted to keep that too. Again, it's huge (I only use effects, controls, slider, and dragdrop... 111K uncompressed)
My list of known, and highly used JavaScript compressors were known not to be able to compress Prototype or Script.aculo.us correctly.

Why Minify/Obfuscate/Compress Anyway?
At the most basic level, regardless of connection speeds, smaller files translate into shorter download times. Sure, web browsers cache static content such as stylesheets and JavaScript files, but there is no justification for unnecessarily slowing the initial download, or relying on the client when you can do something about it quickly and easily.

As for obfuscation, you can't ever really protect JavaScript like you can server-side code/bytecode, and you shouldn't be writing JavaScript that make it easy for a malicious user to manipulate your system. Bottom line: a security risk is still a security risk when obfuscated and you should avoid it all together. That said, with a system as easy to use as the one I'll outline shortly, there's no reason not make a malicious user's goals just a little bit more difficult. On another level, perhaps there are some people you want to share your code with, and some people you don't. Obfuscation gives you more control, since obfuscated code is a major pain to read, understand, and modify safely. Plus, the substitution of tiny meaningless variable names also means a smaller size file (minimally, but smaller nonetheless).

What Didn't Work
Dean Edwards packer is a pretty common tool for packing JavaScript files. There's an online (JavaScript) version, as well as offline versions in .NET, perl, and PHP. Packer, however, uses regex, has strict syntax requirements for it to work that Prototype is known to break. Also, even after alleviating Prototype's syntax issues, the online version of packer still chokes on a few properties, though the offline versions do not. It's reported that there is an online version of the PHP version that does work, located here, but it didn't work when I was trying it. Besides, you still have to fix up the syntax first, and even then there's multiple better solutions.

JSMin is also regex based, so it has the same issues.

Fixing Prototype's Syntax
I really shouldn't say fixing... because technically (as in, according to JavaScript interpreters) what Prototype does is 100% OK. Nevertheless, regex based parsers need stricter syntax than a JavaScript interpreter. Fortunately, the guys over at Dojo had the wonderful insight to ditch the regex all together and use (you guessed it) a JavaScript interpreter! Specifically, they use a custom version of Rhino, which is an open source JavaScript interpreter used in Mozilla applications (Firefox, Camino, Thunderbird, Seamonkey, etc).

Rhino is written in Java, and as a JavaScript interpreter has more context about what's going on in a JavaScript file than a rigid regex does. Hence, Dojo ShrinkSafe came into the world with the ability to safely compress JavaScript files without additional strict syntax requirements. By running files through Dojo ShrinkSafe first, they could then be packed/obfuscated additionally by regex packers because the post Rhino treatment was syntactically 'fixed' for a regex packer.

But there was a remaining problem: If you're using the Dojo Toolkit, then your files are automatically compressed using ShrinkSafe, but I am not and I didn't want to have to go through those steps to manually re-minify a JavaScript file every time I edited it... so my search continued.

A Changing Solution
Shortly, I discovered the pack:tag JSP Tag Library, which seemed to be what I was looking for. The pack:tag had many benefits:

Simultaneous minification and obfuscation
gzip compression
Combination of static resources to minimize round-trips from client to server
Caching of compressed static resources via a memory (Servlet) or file cache, and thus minification, obfuscation, and compression at resource request time (as opposed to compile time)
Pluggable compression algorithms (and an implementable interface for creating your own packing strategy)
Configuration via a .properties file

Four packing strategies are included: 2 for CSS (Isaac Shlueter's CSS Compressor, and the iBloom CSS Compressor), and 2 for JavaScript(JSMin, and the YUI Compressor).

My favorite additional benefits are as follows:

First, the pack:tag allows you to edit your JavaScript files in their un-minified and un-obfuscated form, thus relinquishing me from having to either edit a minified file or use Dojo Shrinksafe and the perl version of Dean Edward's packer every time I changed a file.

Second, the pack:tag checks to ensure that a resource is not included more than once in the same page, automatically ignoring subsequent requests, which can accidentally happen quite easily when using multiple JSP includes (used in multiple places) to dynamically build a page.

Third, the pack:tag allows you to keep your uncompressed (and thus easy to read and edit) JavaScript files within the WEB-INF directory of a web application, where they are protected from prying eyes.

Using the pack:tag as simple as including the JSP Taglib declaration in your JSP, and then using the following instead of normal <script> tags to include JavaScript files:


<pack:script src="/my/file.js" />

Or, to combine multiple resources:


<pack:script>
<src>/my/file1.js</>
<src>/my/file2.js</>
</pack:script>

pack:tag Refined
The only remaining problem with the pack:tag was the same one I discussed earlier: JSMin and other regex based packers do not safely minify Prototype. Fortunately, pluggable compressor strategies come to the rescue! As of version 2.2, the YUICompressor can safely compress Prototype (thanks to the fact that it, like Dojo ShrinkSafe, is implemented using Rhino). This was an amazing development because I now had a solution whereby I could hide my JavaScript files, edit them with complete clarity, and have them automatically combined, minified, obfuscated, and gzip compressed at request time, then cached for the next request!

An added benefit of using the YUICompressor is a significant amount of logging output that complains to you're doing something stupid. Unfortunately, I haven't found a way to turn off that output, so it does fill up the log a bit when compressing files wherein I reference functions that are declared in other files (which it has no context of, unless the files are combined before processing).

Statistical Results
For testing, all downloaded file size measurements were taken using the Firefox plugin Firebug, version 1.05.

I wanted large, commonly used libraries for testing. Since I happen to be a big fan of Prototype and Script.aculo.us, and since Prototype has been my resident problem in this description, I decided those two libraries would be sufficient. I use the same technique outlined above on my own JavaScript files with wonderful results. Since I mostly use Script.aculo.us for the effects, dragdrop, and slider, and thus never include all the components at once, I decided to measure each file individually rather than Script.aculo.us as a whole package.

Terminology
Minification - refers to both minifying and obfuscating the JavaScript
Individual - Each JavaScript file was included separately in the page; for the base case via :


<script type="text/javascript" src="/js/prototype.js"><script>

Or for the other scenarios using pack:tag to include each with:


<pack:script src="/js/prototype.js" />

Grouped - The JavaScrpt files were included together in one file. For the case with minification and gzip via:


<pack:tag>
<src>/js/prototype.js</src>
<src>/js/effects.js</src>
<src>/js/controls.js</src>
<src>/js/slider.js</src>
<src>/js/builder.js</src>
<src>/js/dragdrop.js</src>
<src>/js/sound.js</src>
</pack:tag>

For the minified test without gzip, I created a new file (total.js) with the output of the above grouping and included it via:


<script type="text/javascript" src="/js/total.js"></script>

Versions used in testing
Prototype 1.6
Script.aculo.us 1.7.1 Beta 3

Individual Results

Grouped Results

Conclusion
Compression in some form can significantly decrease file size and, implicitly, download times. Different methods of compression yield varying degrees of benefit. Namely, Gzip (which, by the way, is Prototype's 'officially supported' compression method) yields the greatest initial benefit, with minification yielding additional benefit. On the whole, I now have a setup that I don't have to do anything. I create my JavaScript files as if I were not compressing, include them with slightly different syntax, and the pack:tag with the YUICompressor strategy takes care of everything else.

Adendum
As a testament to this conclusion, more compression options have become available lately. John-David Dalton has released a collection of compressed Prototype and Script.aculo.us versions on Google Code. He explained his process in an Axajian post last December:

I format the code manually, fixing semi-colons and fixing references to $super. I run them through a compressor with quotes around the $super vars so they aren’t changed then fix their method arguments. I use Dean Edward’s Packer because it creates the smallest files. From there you can use a server side solution to gzip/version/and deploy the file. I use Prado (www.pradosoft.com) and their asset publishing capabilities.
I have a Blog but it’s currently in the early stages, I never have time to work on it.

Basically it's the process I described above, except manually fixing the syntax 'problems' rather than using a JavaScript interpreter to do it for you. It's a great solution for non-Java server environments where the pack:tag is not an option.

Update and things to come

2008-01-10T10:11:00.000-05:00

Been a while since I've posted. To update, I was swamped working on my latest project, Kingmaker, but there should be a flurry of blog posts soon as I have a backlog of ideas I want to write about.

In other good news, I was introduced to an awesome program for OS X called Journler, which has become one of my most frequently used programs due to its versatility and thus my ability to use it practically however I want. It is now my second most recommended download for OS X following Quicksilver.

I finally upgraded my Apple MacBook Pro to 3GB of RAM. I have had 2GB since I first got it. Although that was fine for a while, I discovered that attempting to run Netbeans, Adobe Flex Builder, and Photoshop with the usual accompaniment of Apple Mail, web browsers, iChat, and iTunes resulted in a ridiculously high degree of paging and significantly decreased my productivity as I sat there waiting to switch between programs or waiting for programs to start and exit.

Adding the additional RAM made an incredible difference and my productivity has been much the better for it. I only wish that I could have 4GB... which brings to mind that I debated putting in a matched pair of 2GB sticks knowing that my MBP actually supports up to 3.3GB. Ultimately, I decided the extra 0.3GB and having a matched pair was not worth the price of the stick. I had debated moving to 3GB knowing that doing so would mean not having a matched pair and thus losing the 6-8% performance increase recorded for using a matched pair in MBPs. I reasoned that the decrease in paging to move from 2GB to 3GB would significantly outweigh the performance increase from a matched pair and my experience thus far with 3GB has confirmed this.

Blogs to look forward to soon:

1) The continuation of my series on setting up a glassfish cluster and mysql cluster.

2) Effectively and safely packing JavaScript files (including Prototype and Scriptaculous) using the pack:tag JSP-Taglib.

3) My attempts to use HA-JDBC on my glassfish cluster as a way of load balancing requests to the SQL nodes in my MySQL cluster.

4) Using Quartz for scheduling in Glassfish.

5) A Glassfish Admin GUI JDBCRealm bug that affects the containers ability to detect and pick up changes to the realm security.

6) Prototype specific IE6 JavaScript quirks.

7) Minifying DWR

8) Clustered second-level caching with JPA and Hibernate in my glassfish cluster. My 24-hour attempt to use JBoss TreeCache, and the subsequent 30-minute solution using Ehcache.

... But not necessarily in that order.

Project, Part 2: Glassfish cluster

2007-11-09T18:58:00.003-05:00

I came across a few tutorials, but none of them worked out of the box for me. So here's my version of how to set up a 2 machine glassfish cluster. Note, this process was completed on 2 Fedora 7 machines with Glassfish V2 final (aka b58g Promoted Build). If you're using a different operating system or a different build of glassfish, you might need to tweak the process.

For the rest of this post, here will be 2 machines: glassfish1 and glassfish2. Glassfish1 will be the domain administration server (DAS) and will house g1-a1 (glassfish1, node-agent 1) as well as g1-i1 (glassfish1, instance1). Glassfish2 will only have g2-a1 (glassfish2, node-agent 1) and g2-i1 (glassfish2, instance2). In all of the commands below, I reference the machines by their hostname (glassfish1 or glassfish2). I am able to do this because I have edited /etc/hosts on both machines to provide a mapping from the hostnames glassfish1 and glassfish2 to the physical IP for those machines. If you have not done this, and if you do not have another setup for DNS resolution, you'll need to refer to the machines by their physical IP.

You'll need to grab the glassfish installer from https://glassfish.dev.java.net//downloads/v2-b58g.html with wget or FTP it onto both glassfish1 and glassfish2.

If you don't have Sun's Java JDK 5 or 6 installed on your machines, you'll need that first. If you're running Fedora, you can find a guide in my post 'Project, Part 1: Fedora + Sun Java' for installing Sun's Java JDK 1.5 build 13 on Fedora 7.

Time to setup glassfish1:

Copy the installer jar file to /opt

cp <glassfish-installer>.jar /opt

Unpack the jar

java -Xmx256m -jar <glassfish-installer>.jar

Add the execute permission to the ant stuff

chmod -R +x glassfish/lib/ant/bin

Setup the cluster (this will create the DAS)

glassfish/lib/ant/bin/ant -f setup-cluster.xml

Add the glassfish bin directory to the path so we can call asadmin from anywhere

emacs /etc/profile
In Fedora, there is a nice function pathmunge, already in /etc/profile, which will allow you to add directories to the path without accidentally creating duplicates, so add the following line before PATH is exported:

pathmunge /opt/glassfish/bin

Start the default domain (domain1) with:

asadmin start-domain

Create the first node-agent

asadmin create-node-agent --host glassfish1 --port 4848 g1-a1

Create the cluster

asadmin create-cluster --host glassfish1 --port 4848 cluster1

Create the first instance

asadmin create-instance --host glassfish1 --port 4848 --nodeagent g1-a1 --cluster cluster1 g1-i1

Start the node-agent

asadmin start-node-agent --syncinstances=true g1-a1
Note: the syncinstances argument will start the associated instances and clusters as well, so you'll want to remember this for later.

If you made it that far, now it's time to setup glassfish2:

cp <glassfish-installer>.jar /opt
java -Xmx256m -jar <glassfish-installer>.jar
chmod -R +x glassfish/lib/ant/lib
glassfish/lib/ant/bin/ant -f setup-cluster.xml

Note: the above will create a DAS on glassfish2... which we don't want, so you can remove the DAS after the script finishes with:
rm -rf /opt/glassfish/domains/domain1
If you want, you can avoid creating the DAS on glassfish2 in the first place by using the following command instead of the one above (#4):
glassfish/lib/ant/bin/ant -f setup-cluster.xml \ create-node-agent --Ddas.host=glassfish1
That will create a node-agent with the default name "na". If you want to specify the name of the node-agent, you'll need to edit setup-cluster.xml before running the above command. Find the property "nodeagent.name" and change the value to whatever name you want:
<property name="nodeagent.name" value="my-nodeagent-name" />
Either way, running setup-cluster.xml, which creates the DAS, then removing the DAS and creating a node agent will provide the same result as running setup-cluster.xml with the option to only create the node-agent.

Add /opt/glassfish/bin to the PATH as on glassfish1
Create the second node-agent (If you didn't in step 4)

asadmin create-node-agent --host glassfish1 --port 4848 g2-a1

Create a new instance

asadmin create-instance --host glassfish1 --port 4848 --nodeagent g2-a1 --cluster cluster1 g2-i1

Start the second node agent

asadmin start-node-agent --syncinstances=true g2-a1

That's it! Well... sort of, there are a few things to note:

Make sure that the firewall/SELinux/iptables rules are set to allow ports that are needed by Glassfish. These are, at the minimum:
- 4848 (admin HTTP)
- 8080 (HTTP)
- 8181 (HTTPS)
- 8686 (RMI)
- 3700 (ORB)
- 7676 (IMQ)
and possible others if those are already in use. If your node-agents are having trouble communicating, try disabling the firewall & SELinux, and set iptables to accept all incoming connections. DO NOT DO THIS AS A LONG TERM SOLUTION. IT IS ONLY MEANT TO RULE OUT PORT COMMUNICATIONS AS A PROBLEM. Depending on your needs, you may be altering the port configurations anyway. I decided to implement a white-listing policy, where I deny all incoming connections except the following:
- On glassfish1:
- On glassfish2:
Thus, my servers are free to communicate with each other on any port they desire, but outside users may only connect to glassfish1 on 8080 and 8181, and only I can connect to glassfish1 on 4848. If you're messing with iptables, and you're not too familiar with *nix environments, you should probably have a *nix sys-admin check out your setup before it goes into production to make sure you haven't inadvertently created any security holes.

If you're having trouble starting your instances, especially g2-i1 on glassfish2 you might have to edit your /etc/hosts file... I did. This is something that you should check if the following shows up in your server.log:

[#|2007-11-05T12:19:03.832-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NAGT
0038:Executing Synchronization for node-agent With DAS|#]

[#|2007-11-05T12:19:12.920-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.tools.synchronization|_ThreadID=10;_ThreadNam
e=main;|SYNC001: Unable to communicate with Domain Administration Server. Skipping synchronization.|#]

[#|2007-11-05T12:19:12.922-0500|SEVERE|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NA
GT0035:The NodeAgent failed to complete the intial synchronization with the DAS. Please make sure the DAS is running and is a
ccessible from the NodeAgents server|#]

[#|2007-11-05T12:19:14.925-0500|WARNING|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|N
AGT0013:Stopping Node Agent...|#]

Or, if the following shows up in your instance server.log:

[#|2007-11-07T02:25:31.352-0500|WARNING|sun-appserver9.1|javax.jms|_ThreadID=10;_ThreadName=main;_RequestID=efb5ac8e-778b-412b-ae9e-3af1e95a8ee2;|[C4003]: Error occurred on connection creation [localhost:37676]. - cause: java.net.ConnectException: Connection refused|#]

[#|2007-11-07T02:25:31.363-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=10;_ThreadName=main;_RequestID=efb5ac8e-778b-412b-ae9e-3af1e95a8ee2;|java.lang.RuntimeException: MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
at com.sun.messaging.jms.ra.LocalBrokerRunner.start(LocalBrokerRunner.java:366)
at com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:447)
at com.sun.enterprise.connectors.ActiveInboundResourceAdapter$1.run(ActiveInboundResourceAdapter.java:135)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.enterprise.connectors.ActiveInboundResourceAdapter.(ActiveInboundResourceAdapter.java:131)
at com.sun.enterprise.connectors.system.ActiveJmsResourceAdapter.(ActiveJmsResourceAdapter.java:247)
at com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(ActiveRAFactory.java:107)
at com.sun.enterprise.connectors.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:300)
at com.sun.enterprise.connectors.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:445)
at com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(ConnectorRuntime.java:224)
at com.sun.enterprise.jms.JmsProviderLifecycle.onStartup(JmsProviderLifecycle.java:428)
at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:442)
at com.sun.enterprise.server.ondemand.OnDemandServer.onStartup(OnDemandServer.java:120)
at com.sun.enterprise.server.PEMain.run(PEMain.java:411)
at com.sun.enterprise.server.PEMain.main(PEMain.java:338)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.enterprise.server.PELaunch.main(PELaunch.java:412)
|#]

What's happening here is that the node-agent is not starting because the instance is failing due to an MQ startup problem As it turns out, the MQ needs the hostname to point to an absolute address, not 127.0.0.1 or localhost. If you edit your /etc/hosts file on glassfish1 to add:

xxx.xxx.xxx.xxx glassfish1

And on glassfish2 to add:

xxx.xxx.xxx.xxx glassfish2

then the problem should go away. Shalini, was kind enough to point that out to me and he has a blog post here that also mentions the problem. My original guide for doing this also came from Shalini's blog entry on setting up a 2 machine glassfish cluster using CLI.

A response to "Have you seen the news?"

2007-11-07T23:59:00.000-05:00

I normally reserve this blog for technology posts, but this is an exception.

Over the past day, I've received a couple emails with links to news stories, regarding the tragic death of a British girl studying abroad in Italy, accompanied by the question, "Have you seen this?" I'm asked because I happened to attend the same high school (within a couple years) as one of the suspects. Too that question, here is my answer:

Yes, I've heard, so have you. Now let's stop endorsing the unrestrained sensationalism of the media.

If you browse around for a while, you'll see that various articles paint different pictures slandering different individuals. In situations like this, what normally is viewed as typical college debauchery is recast as criminally indicative behavior. After all, profit motivated media needs to sell headlines, and in the realm of "Buy this paper!", nothing beats a sexual deviant with a murderous dark side. The deplorable part is that it doesn't matter if the individual fits the headline, only that the headline sells the paper.

Italian police are under enormous pressure, which the strong media attention only increases. They can't move too quickly, lest they appear hasty or negligent. They can't be too methodical, lest they appear incapable or overwhelmed. In either case, it is a tightrope to avoid having their qualifications called into question. Too many want the answer now, and too few want the right answer. It's a terrible situation all around and I feel for each person caught in this dragnet of media attention. I sympathize with all, each for a unique reason.

At this point, available information is limited to media sensationalism with underlying culture-stereotypic allegations redolent of long existing cultural partisanship and prejudice. The most important thing now is to acknowledge the havoc and irreparable change that has and will continue to affect the individuals and their families. The court of public opinion has no jurisdiction here, and we've all had a time when we deserved more privacy and respect than we received from overly eager interested parties. If out of nothing more than respect for the families, I hope that the hype will calm down.

And no, I'm not going to post links to any of the articles. If you have no qualms about supporting sensationalistic media, then I'm sure your Google skills are already top notch.

Project, Part 1: Fedora + Sun Java

2007-11-05T17:46:00.000-05:00

Glassfish needs Java JDK 5, or 6. Fedora doesn't come with the Sun JDK, but rather GNU Java. Additionally, unlike my experiences with Ubuntu, Fedora does not have a recent implementation of the JDK available from a repository. So I was confronted with this choice:

Try to force Glassfish to work with GNU Java, the default in Fedora, or
Try to force Fedora to use Sun's Java

I decided on the later after reading that none of the java language features new in Java 1.5 are implemented yet in gcj. At the same time, the blog consensus was that installing Sun's Java on Fedora was not straight forward. First, the different java versions could interfere with each other; and second, if installed incorrectly, Sun's java could be accidentally, and unintentionally, 'upgraded' to gcj when installing a system update.

For the most part, I followed a combination of these blogs: blog1, blog2, blog3, blog4. Alas, none of them worked despite following step by step, but in doing so, I learned a lot about what was going on.

The first problem seemed to be related to a dependency difference between JDK 1.5.0 build 11 or 12 and JDK 1.5.0 build 13, which is what I was using. Although each blog listed certain dependencies, none listed the dependency problem I was having. I still don't know what libXtst is, but without it I was unable to install the JDK. With it, I was able to install just fine.

The second problem had to do with the order of installing RPMs. If I tried to install them all at once with a yum -y --nogpgcheck install /usr/src/redhat/RPMS/i586/java*.rpm, it would fail because of an error in the SPEC file dealing with font package locations (detailed here, in the Current Issues section). However, if I installed only java-1.5.0-sun-1.5.0.13-1jpp.i586.rpm first, and then installed the rest, I ended up with a working version (though the font problem remained).

A third problem was that blog1, referenced above, which involved rebuilding the java RPMs via rpmbuild -ba java-1.x.0-sun.spec resulted in a working JRE, but no JDK.

Here is the final version of what worked for me:

Some prerequisites
1. yum -y install rpmdevtools jpackage-utils rpm-build libXp unixODBC unixODBC-devel libXtst yum-priorities
2. rpm -Uvh http://www.fedorafaq.org/yum http://rpm.livna.org/livna-release-7.rpm
Get the JPackage Java RPM with
wget http://mirrors.dotsrc.org/jpackage/1.7/generic/SRPMS.non-free/java-1.5.0-sun-1.5.0.13-1jpp.nosrc.rpm (lists at http://mirrors.dotsrc.org/jpackage/1.7/generic/SRPMS.non-free/)

Download the right JDK binary from Sun (not the self-extracting RPM) for the java version and build version of the JPackage RPM into /usr/src/redhat/SOURCES/

recreate the java rpm
1. rpmbuild --rebuild java-1.5.0-sun*src.rpm
2. should create new rpms in /usr/src/redhat/RPMS/i586/
install new java rpms (need the --nogpg option because locally created rpms are not signed)
1. yum -y --nogpgcheck install /usr/src/redhat/RPMS/i586/java-1.5.0-sun-1.5.0.13-1jpp.i586.rpm
2. yum -y --nogpgcheck install /usr/src/redhat/RPMS/i586/java*.rpm
check java version with java -version, and that the JDK is installed with javac --help. If the version is not right, or it didn't work, change the default Java system to Sun's with alternatives --config java, and select the right jvm

Set the JAVA_HOME environment variable
1. Add to /etc/profile
  JAVA_HOME="/usr/lib/jvm/java" export JAVA_HOME
2. run source /etc/profile to read in profile, and check it worked with echo $JAVA_HOME

That's it, now for Glassfish...

Fedora 7 + Sun Java + Glassfish + MySQL

2007-11-05T17:30:00.000-05:00

Start:
6 - Servers with a base install of Fedora 7
? - Time and effort

End result:
1 - 2 machine, load-balanced, application server cluster with failover and in-memory replication
1 - 4 machine, load-balanced, MySQL NDB database cluster with 2 storage nodes, 1 sql / load balancer node, and 1 management / load balancer node

The process will be described in a serious of forthcoming posts.

Odd JSP EL behaviour

2007-10-30T20:48:00.001-04:00

I just discovered an oddity in JSP Expression Language. I'm a huge fan of EL and its many uses. This is hardly a problem, but is something to keep in mind because it is an unexpected behavior.

Take:


int a = 5;
int b = 2;

System.out.println(a / b);

The println statement will output 2. Thus in EL (and JSTL) in a JSP you would expect the same integer division behavior from the following:


<var="a" value="5" />
<var="b" value="2" />

${a / b}

In the above, ${a / b} yields 2.5, behaving like floating point division rather than integer division. I can see this having its uses, but it surprised me, especially since I've been working almost exclusively in Java web technologies for the past few years. For future purposes I created an EL function so I can just call the following when I want to force integer division (where f is whatever prefix is specified for the custom taglib):


${f:integerDivision(a,b)}

Why I'm not moving to Leopard

2007-10-29T15:53:00.000-04:00

Although I've been looking forward to OS X Leopard for some time now, I just can't make the switch yet. As it turns out, there's one major hurdle to be overcome before Leopard is an option for developers of Java EE 5 applications to be deployed on Glassfish. Here's the problem:

Glassfish encounters an exception that prevents domain startup when connected to the internet via Wireless, not connected at all, or in any other way not connected by the ethernet port (Stack trace below). If, however, I'm connected via the ethernet port, everything is okay. I develop applications to deploy in Glassfish, but I can't limit my development to when I'm tied down by the ethernet cable. So until there's a solution, I'm sticking with Tiger.

Anyone with a solution, please let me know. I haven't seen anything like this since fixing PCs with strange APIPA problems...

Full Exception:

[#|2007-10-29T15:28:48.086-0400|SEVERE|sun-appserver9.1|
  javax.enterprise.system.core|
  _ThreadID=10;
  _ThreadName=main;com.sun.enterprise.admin.server.core.JmxConnectorLifecycle@328cbf;
    com.sun.appserv.server.ServerLifecycleException: Cannot bind to URL
    [rmi://169.254.161.207:8686/management/rmi-jmx-connector]:
      javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException:
      Connection refused to host: 169.254.161.207; nested exception is:
      java.net.ConnectException: Operation timed out
    ];
  _RequestID=69db80be-8146-46bb-a879-157fedee2f87;|
  Service com.sun.enterprise.admin.server.core.JmxConnectorLifecycle@328cbf cannot be started!:
  com.sun.appserv.server.ServerLifecycleException: Cannot bind to URL
  [rmi://169.254.161.207:8686/management/rmi-jmx-connector]:
    javax.naming.ServiceUnavailableException
    [Root exception is java.rmi.ConnectException:
      Connection refused to host: 169.254.161.207; nested exception is:
      java.net.ConnectException: Operation timed out
    ]|
#]

Facebook Platform

2007-10-25T15:56:00.000-04:00

Sometimes FB developers suck. Namely, when I have to add an application just to see what somebody else has done with it. It's the equivalent annoyance of having to login to comment, or to rate something on a site. I see the app on their profile, and there are options to interact with it. I try to do something, but they require me to add the app to do anything. So, of course, I decline the app and never see what I was trying to see.

It's just really frustrating that many developers force you to add the app to see anything about it, and it seems a bit backwards. I'm not going to add an app without seeing inside it first, but if I have to add it to see inside, I'll never add it. Now, this doesn't seem to be troubling many developers, as they are racking up users quite easily. Nevertheless, I'd appreciate it if developers would stop trying to build mini walled gardens inside the walled FB estate.

On a larger note, I'm still waiting for the FB app to beat all FB apps in terms of actual usefulness. Maybe I just haven't found it yet, but I really don't want gimmicky things on my profile and I think the FB platform has much better uses than trying to build up vampire points.

On a smaller note, the abundance of crappy looking FB apps being launched has reaffirmed my opinion originally derived from MySpace profiles that most people out there have absolutely zero sense of design.

Gmail IMAP woes

2007-10-25T15:12:00.000-04:00

I was really excited yesterday when I found out that my Gmail for domains account had IMAP access. I've used Apple Mail and my Blackjack to access my Gmail accounts via POP for a while now. It's not that I have anything against the Gmail web interface, but I've just never particularly liked having to open a web browser to view my email.

POP access had its downfalls, of course, though by using "recent:username@gmail.com" as the username, I was at least able to make sure that all my various devices accessing the account were able to get new emails. When, I saw IMAP access had been added, I thought all my troubles with accessing Gmail accounts from multiple sources would be gone for good. Not quite so.

It's pretty close though, I only have one complaint: Accessing Gmail via IMAP from Apple Mail stores drafts of messages in Gmails Sent Mail folder, rather than the Drafts folder. I sent a couple emails last night, and when I woke up this morning I saw 8 copies of one of the emails in my Sent Messages folder. Before I fired off an apology to the recipient for accidentally spamming their inbox, I figured I'd do a little testing, during the course of which I confirmed that I did not actually send the email 8 times, but rather that drafts were being stored on Gmail as sent messages.

I don't know whose fault this is, Apple Mail or Gmail, but it's a little annoying. For now, I've solved the problem by unchecking the box to "Store draft messages on the server" in Apple Mail's preferences for the Gmail account. I suspect it may have something to do with folder mappings, but I don't know of a way in Apple Mail to change the folder on the server that is associated with a particular type of email, i.e. a draft.

SimpleCaptcha in Glassfish and Tomcat

2007-10-09T16:15:00.000-04:00

I'm a big fan of SimpleCaptcha. It has ample configuration options and, after some tweaking, it looks good.

It's clean and easy for the user to read, but still accomplishes the purpose. I especially like that you can specify the characters to be used, thereby eliminating possibly confusing characters, such as 1 and l (that's the number one, and a lower case L).

I've used SimpleCaptcha in Fantasy Congress, running on Tomcat 5.5, and Glassfish v2. Although SimpleCaptcha is great, there were a couple quirks that I had to work around. By far the best thing about SimpleCaptcha is that the source is available, which I've needed both times to get it to work and which allows you to really change whatever you want.

1.) If you use SimpleCaptcha on a server running on some version of *nix without a GUI, you'll need to decapitate AWT. There are a lot of blogs describing how to set the system property java.awt.headless=true, but with limited success. I never got that to work and from the comments I've seen, I'm not alone. Another solution, as this blog describes, is to fake a GUI with the X window interface framebuffer server. I didn't see any reason to resort to that. My solution: I noticed that the problem was being caused in the renderWord(String word, int width, int height) method of nl.captcha.text.imp.DefaultWordRenderer at:


GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
      
GraphicsDevice gd = ge.getDefaultScreenDevice();
GraphicsConfiguration gc = gd.getDefaultConfiguration();

The interesting part is that these objects are created, but never referenced, so I just commented them out and it worked fine. I later discovered an article on Java Boutique explaining why this was.

2.) I didn't like the line across the text that SimpleCaptcha generated by default. I know that it increases the effectiveness of a captcha but it simply wasn't necessary for my purposes. So I cracked open the source again and removed the line by commenting out these lines in nl.captcha.obscurity.imp.WaterRiple:


//draw line over the iamge and/or text
NoiseProducer noise = (NoiseProducer)Helper.ThingFactory.loadImpl(
                  Helper.ThingFactory.NOICEIMP, props);
noise.makeNoise(imageDistorted, .1f, .1f, .25f, .25f);
noise.makeNoise(imageDistorted, .1f, .25f, .5f, .9f);

Note: I set cap.obscurificator to nl.captcha.obscurity.imp.WaterRiple in the servlet declaration. If you're using a different class for obscurity, you'll need to modify the relevant class.

3.) Dropping the SimpleCaptcha setup I used in Tomcat into Glassfish resulted in a dump of nonsensical text to the display. I remedied this by editing nl.captcha.servlet.CaptchaServlet and changing the doGet method from:


String capText = captchaProducer.createText();
req.getSession().setAttribute(Constants.SIMPLE_CAPCHA_SESSION_KEY, capText);

String simpleC =(String) req.getSession().getAttribute(Constants.SIMPLE_CAPCHA_SESSION_KEY);

captchaProducer.createImage(resp.getOutputStream(), capText);

to:


String capText = captchaProducer.createText();
req.getSession().setAttribute(Constants.SIMPLE_CAPCHA_SESSION_KEY, capText);

String simpleC =(String) req.getSession().getAttribute(Constants.SIMPLE_CAPCHA_SESSION_KEY);

resp.setContentType("image/jpeg");
captchaProducer.createImage(resp.getOutputStream(), capText);

It seemed logical that because the servlet is outputting an image (and in my case is mapped to /Captcha.jpg) that the content type should be an image, specifically "image/jpeg". Because adding the line resp.setContentType("image/jpeg"); solved the problem, I'm led to believe that the contentType was not being set before (though I haven't checked). However, that presents the interesting question of why SimpleCaptcha works in Tomcat without setting the contentType. I think this might have something to with the default mime types declared in the server-wide deployment descriptor in Tomcat. I'm not familiar enough with Glassfish yet to say whether it has an equivalent. I'd appreciate some insight if you have an explanation. For now, it works, and I don't have time to troubleshoot what works : )

EIP meet DWR, DWR meet EIP

2007-10-09T12:32:00.001-04:00

My favorite AJAX framework out there is DWR as I've found it tremendously useful, developer friendly, and a breeze to implement. I originally used AJAX by writing my own non-abstractable-pseudo-framework and for the longest time I avoided AJAX frameworks because they were not extensible enough, didn't provide the functionality I needed, or just seemed to stick their grubby little paws into my code in ways I didn't desire. I finally adopted a framework when I discovered DWR, and it was my most wonderful day with AJAX. Thus, I've become quite attached to DWR.

I've toyed around with edit-in-place functionality in a few projects, but only on a small scale, and without using any 3rd party scripts. (I generally approach technologies in that manner, I like to know how it works before I start using it.) I wanted to thoroughly integrate an in-place editing system in a project I'm currently developing, and since by now I had experimented enough to know how the edit-in-place wheel rolls, I started looking around for a suitable wheel that somebody else had already built. The best I stumbled across is EIP, written by Joseph Scott (check out his blog at http://joseph.randomnetworks.com/).

EIP seemed to be a nice system, but it had one obstacle that needed to be overcome for it to be a viable option for me: it uses Prototype for AJAX, and I wanted DWR integration. Fortunately, this was relatively straight forward and there were no huge barriers to making DWR and EIP play well together. The only stumbling block was how to enable EIP to call methods opened up by DWR with variable numbers of arguments. A cool thing about javascript is that essentially every method accepts a variable number of arguments, despite what the method header (which is basically just a way to assign names to arguments) may declare. The tricky part was that somehow, EIP needed to be able to take:


dwr_params: {
p0: 0,
p1: 1
}

...

dwr_params: {
p0: 0,
p1: 1,
p2: 2
}

and then call:


DwrExposedClass.dwrExposedMethod(p0,p1,dwrCallback);
DwrExposedClass.dwrExposedMethod(p0,p1,p2,dwrCallback);

respectively. But there is no way to construct an argument list in a loop, except for to create the entire method call as a String and then use eval() to make it happen. I'm not a big fan of eval() (for reasons that may show up in a different post), and although I occasionally see it as necessary, I only resort to it after exhausting the possibility of other options.

The first thing that came to mind was currying, which (for the unfamiliar) lets you turn a method that takes multiple arguments into a method that takes one argument. This seemed like a good starting point because if I could call any DWR exposed method with only one argument, then it didn't matter how many additional arguments needed to be included. At that point I figured I could either curry an array of arguments down to one, or recursively curry until I arrived at a single argument method.

When playing around with how to implement such a solution, I discovered a much simpler way in rediscovering the apply() method of the javascript Function object. Essentially, arguments are passed as an array in javascript, evidenced by the fact that any number of arguments can be passed to any method, even one with zero declared parameters, and then referenced within the method by treating the local variable arguments as an array. Thus, at a basic level, the apply() method of a function can be used to apply an array of arguments to the function.


var args = [0,1,2,3];
DwrExposedClass.dwrExposedMethod.apply({},args);

Problem solved, I can now make any item editable and use DWR for AJAX with:


EditInPlace.makeEditable({
 id: 'editableItemId',
 ... any other EIP params...
 use_dwr: true,
 dwr_method: DwrExposedClass.dwrExposedMethod,
 dwr_params: { // any other data you want to pass
     p0: 0,
     p1: 'foo'
 }
});

Feel free to use the script. You'll need DWR and Prototype.

Download the script here.