FragmentCache part duex: Cleaning

I realized that I forgot to address an additional, critical, issue with fragment caching: cleaning out expired fragments.  A fragment cache implemented as I described in my last post would update expired fragments when requested, but what about expired fragments that aren't requested?  

If you externalized the cache itself, then your cache will likely support LRU or LFU expiration policies. But, if you're using a in-memory cache in the Java heap space, well, that's a memory leak.  If you cache a lot of fragments, failing to clean the cache is a recipe for maxing out the old generation space and thus causing Full GC's to run back-to-back, collecting just enough space to limp along, but not do any good, until eventually your application becomes non-responsive.  So, an additional requirement for an in-memory fragment cache is that you monitor the usage and have a cleaner task which runs, say, once an hour, to remove fragments that have expired and are not currently locked for update.  Doing this efficiently with a large cache requires using a data structure that supports thread-safe concurrent access, so as to avoid locking the whole cache while cleaning and maintain high throughput.  

The JamLegend Fragment Cache

Abstract

The benefits of fragment caching are pretty well-known.  If you haven't come across it before, the concept is simple:  cache portions of the page to lighten database load by reducing or eliminating redundant queries.  Often, this can be the "most bang for your buck" type of cache. How long a fragment should be cached is a function of the expense of generation, frequency of changes, and the frequency of access.  Ideal fragments for caching are the most accessed and most expensive to generate.

The fragment cache I'm going to discuss is the implementation of the FragmentCacheTag JSP custom tag, in J2Free, the one-day-to-be-open-source framework on which JamLegend is built (you can find more info on J2Free here, and in a few of my previous blogs).  

JamLegend Examples

The "Popular Today" songs lists on JamLegend are a little expensive to generate, shown on all the highest trafficked pages, and don't change too quickly.  As such, we cache that list for 5 minutes, saving us a lot of queries but keeping the list rather dynamic.  On the other extreme, comments change constantly and need to be much more dynamic.  We cache comment threads for 30 seconds.  When you have a high number of concurrent users, even a 30 second cache can save you a lot of queries without sacrificing liveliness.

The Code Samples


<cache:fragment timeout="${30 * 1000}">
Simple cache based on a timeout of 30 seconds.
</cache:fragment>

<cache:fragment timeout="-1" condition="${size(items)}">
Simple cache with no timeout, using instead an additional attribute to refresh the cache when the size of the collection "items" has changed since the cache was last updated. This works well in situations where data changes rarely, and you want it to be available immediately upon change.
</cache:fragment>

<cache:fragment timeout="${30*1000}" disable="${devMode}">
Simple cache based on a timeout of 30 seconds but with an attribute to disable the cache if devMode is flagged. Here, devMode is a application scope attribute set when the server starts up. It's nice to be able to selectively disable certain caches for testing, and have them enabled live without having to change any code.
</cache:fragment>

The JamLegend FragmentCacheTag also supports an additional application scope attribute to globally disable all caching.  One of these days, I'm going to get around to changing the timeout to be specified in seconds, instead of milliseconds; just hasn't happened yet... 

Fragment Caching "Gotchas"

Two of the first problems you'll run into with fragment caching are:

1. logged in vs. logged out users
2. concurrent cache refreshing

The first is a problem you're going to encounter simply by using a fragment cache.  The second, on the other hand, you'll only run into if you're building your own fragment cache; if you use the J2Free FragmentCacheTag described here, it will be taken care of for you.

Adding Dynamic Capabilities to Cached HTML 

Sometimes ideal candidates for fragment caching have display differences when users are logged in.  For example, caching comment threads:  logged in users see a "report" link, whereas logged out users do not.  Since a fragment cache holds the generated HTML, as opposed to an object cache or query cache, it isn't easy or efficient to modify the fragment based on logged-in status (though, it's not impossible and I'd appreciate any elegant approaches).  Fortunately, JavaScript comes to the rescue.  

On JamLegend, a "report" link is included in the generated HTML for comment threads whether the user is logged in or not, with style="display:none;" to hide it initially.  Then, if the user is logged in, we use a little JavaScript on page load to show all the report links, otherwise they stay hidden.  This is the same technique used by 37Signals (see JavaScript Makes Relative Times Compatible with Caching).  Since we also check server-side that a user is logged in before allowing them to mark a comment as spam, there isn't a significant downside to having the hidden spam links for users who are not logged in.

Preventing Concurrent Refreshes

If you're caching a fragment because it's expensive to generate, the last thing you want is a few hundred users concurrently refreshing the fragment triggering a few hundred concurrent expensive queries.  So, how do you prevent this?  The first question you have to answer is this:

When user A has triggered a cache refresh, what should happen to other users who hit the same fragment?

Should they:

(a) Head over to user A's thread, wait for user A's thread to complete the query, and then ask for the result?  or,

(b) Should they just see that user A is refreshing the cache and so grab the last cached fragment and return immediately?

Both are acceptable answers, but which you choose depends on how stale you allow your fragment to be.  In case A, you can serve the most recent fragment to an unlimited number of callers while still only generating the fragment once.  Since user A triggered the refresh first, that thread is necessarily ahead of the other threads, so the others will still return is less time than if they were the sole thread refreshing the cache.  In case B, the subsequent threads can return immediately serving the last version of the fragment while assuming that user A's thread will refresh the cache.

For J2Free's FragmentCacheTag, I chose scenario B on the reasoning that the fragment is cached to begin with because it is not crucial that it be 100% up-to-date.  Ergo, what's one more minute with the old fragment?  At some point, I want to add this as an optional attribute, something to the effect of:


<cache:fragment timeout="600000" waitOnRefresh="true">
... where waitOnRefresh enabled would indicate that the cache should not server the last fragment while being refreshed, but rather attach itself to the thread conducting the refresh and get the result when complete.
</cache:fragment>


Obviously, the above only applies to multi-threaded servers.  There is an additional case, though, that applies to single and multi-threaded servers:  what happens in a cluster?  How do you prevent concurrent refreshes when you're running multiple servers?  A few options are: 

1. Store fragments in a distributed cache, like ehcache, JBoss cache, or memcached, using replication instead of invalidation.  However, there is still the possibility of instance A triggering a refresh then instance B triggering a refresh before instance B received the updated fragment from instance A.  This window could be decreased by replicating a lock-for-update on a fragment, rather than waiting for the complete fragment, but not closed entirely.
2. Using a central, synchronized cache would remove the race-condition from situation 1, but at the cost of decreased throughput.  Depending on the location and network access to such a cache, it may or may not be worth it.  Ping times intra-ec2-zone are only 1-2ms between instances, so it might be worth it to add a few ms to each request to avoid concurrently executing an expensive query.
3. Use a distributed cache, as in situation #1, but only allow 1 instance to refresh the cache.  This may or may not work depending on how well you split requests between instances.  You certainly wouldn't want users on instance B to continue to receive stale data because no user on instance A has triggered a cache refresh.  This strategy may work well for pages that are guaranteed to be accessed on each instance.

I welcome comments on any additional solutions to preventing concurrent cache refreshing in a cluster.

Another Flex Cross-Browser Incompatibility (OS X ProgressBar display)

So, Flash is supposed to solve all our cross-browser issues... only it doesn't.  My most recent discovery relates to the visual look of a ProgressBar between OS X and Windows when the mode = ProgressBarMode.MANUAL and indeterminate = true.

To illustrate this, we'll start with the following MXML component:


<mx:progressbar x="10" y="10" id="progress" labelplacement="bottom" indeterminate="true" enabled="true" minimum="0" maximum="100" label="" width="210"/>

With some styling applied, the result is this:

Now, let's say that we want to use the ProgressBar for some non-indeterminate (er, determinate) purpose, like loading some data from a server. So, let's do this:

progress.indeterminate = false;
progress.mode = ProgressBarMode.MANUAL;
progress.setProgress(0,100);

someLoader.addEventListener(ProgressEvent.PROGRESS,someProgressListener);

Great. Now, someLoader will load something, and someProgressListener will keep us up-to-date about the progress. So, say we've loaded all the data and now we're doing some processing, the progress of which we don't know since our processing function is synchronous and does not dispatch ProgressEvents. So, we want to set indeterminate back to true so that our users will know something is still happening but not show possibly incorrect progress with a determinate ProgressBar. So, let's do this:

progress.removeEventListener(ProgressEvent.PROGRESS,someProgressListener);
progress.indeterminate = true;

WTF! Now our ProgressBar looks like this:

In Firefox and IE on Windows, the ProgressBar looks identical to before, which is the expected behavior. But in Safari (at least) on OS X, we have this monstrosity. What happened? Shouldn't Flash look identical in any browser? Well, apparently it doesn't.

Fortunately, this isn't unfixable.  In fact, it's a very simple fix, though it makes zero sense to me why this changes the visual display.  Just set progress.mode = ProgressBarMode.EVENT. Turns out that this ugly displacement only occurs in OS X if (progress.mode == ProgressBarMode.MANUAL && progress.indeterminate). Oh well, something to chalk up on the list of cross-browser differences in Flash.

On another note, remember that progress.setProgress(num,total) will not do anything unless you have set progress.mode = ProgressBarMode.MANUAL. This is another thing I think is ridiculous, but the ProgressBar will ignore any calls to setProgress if mode != ProgressBarMode.MANUAL. I understand that the different ProgressBarMode settings will instruct the ProgressBar where to expect progress changes from, but it still seems odd to fully ignore calls to setProgress WITHOUT throwing any kind of error or warning when mode != ProgressBarMode.MANUAL.

Versions:  This was discovered in OS X 10.5, Flash MAC 10.0.12.36 (debug), Safari 3.2.1.

Flex / AS3 Tweening

I can't believe I haven't given a shout out to TweenLite before.  I love it.  Absolutely, love it.  For the only reason you should need to use TweenLite over Flash's built in Tween, Tweener, ZigoEngine (Fuse), Twease, HydroTween, or gTween, see this speed comparison.  

The second great part about TweenLite is that it is dead simple to use.


TweenLite.to(objectToTween, howLongToTween, { propsToTween ... callbacks });


Ridiculously simple to integrate into any project.

I only have one complaint:  When I haven't tweened in a few days, I always forget that the duration for a TweenLite is represented in secconds, not milliseconds.  Usually, it takes me about 5 minutes to remember, since my Tween didn't happen when I wanted and instead happened 300 seconds later as I was staring at my screen wondering what the hell I messed up.  I get that it's a convenience thing, but to me it's inconvenient.  If TweenLite durations were specified in milliseconds, I think I'd gain 1000* 60 * 5 * 7 back to my week.

Monitoring AWS Ubuntu Instances using Munin

Another munin setup blog? Why?

There are way too many blog posts out there with incorrect info on how to setup munin.  My process to get it setup infuriated me enough to break from work and write the post.  If any blog tells you to run:

/usr/bin/munin-cron --force-root

or any other munin command using --force-root, you should instead run like hell in the opposite direction.  Doing so will only cause you hours of hell figuring out why your graphs stopped updating.  It turns out that by using --force-root you just hijacked ownership of the .rrd, .png, and .html files that munin uses; for more details, see here.  Furthermore, anybody that wrote a blog telling you to do that clearly didn't wait to see whether their setup was successful before posting (or they made the unwise choice of running munin as root all the time).

There also is some bad information on the web about using the setting use_node_name when setting up groups and hosts in the munin master config file.  Nearly every blog I found included enabling use_node_name despite the munin documentation saying very specifically,

"You will almost never want to set [use_node_name]  to anything, as setting it to "yes" tends to break things." (source)

Doesn't that just scream "Don't do it!"  And yet nearly everybody is blindly recommending it.  The thing is that by enabling it, you can make a setup work where it otherwise wouldn't have if you screwed up the config by using inconsistent host names between the munin master and munin nodes.  So, what should you do?  Is it okay to enable use_host_name?  My inclination is no, because it's just a naive hack around the fact that you screwed up the config.  If the people that build munin say don't use it, then I say don't use it;  just learn how to configure munin correctly.

So, without further introduction, here's a more comprehensive guide to setting up a single munin master to monitor multiple munin nodes, and since I did this on AWS, there are some snippets about do's and don'ts on AWS.  

On the Munin Master

Quite simple, run:

apt-get install -y munin apache2

There.  Munin master is installed. Easy, no? That just setup the appropriate directories, cron jobs, and permissions, so all you really need to do next is edit /etc/munin/munin.conf but we'll wait to do that until after the nodes are ready.  Additionally, it installed apache so that you can view munin reports on the web.

On all Munin Nodes

Again, simple, run:

apt-get install -y munin-node

Told you that would be easy.  Great, now time to configure each node.  There are only a few things we need to configure on the node, and they are all located in /etc/munin/munin-node.conf.  The properties to set are:

host_name my.hostname.com
allow ^120\.0\.0\.1$

The host_name setting will be the name by which your munin-node will report itself to the munin master.  This is very important, though I didn't find it mentioned in any other blog, for reasons I will explain below when I show the master munin.conf config file.  The short version is that it doesn't really matter what you call the node, just choose something that makes sense to you and remember it.  For a database layer, something rational might be mysql.yourdomain.com for the master, and slave0.yourdomain.com and slave1.yourdomain.com for the slaves.

The allow setting specifies what master host(s) are allowed to connect to this munin node.  Due to some perl specifics, this value has to be a regex, though that can come in handy.  You can repeat the allow line as much as necessary.

This is where the AWS issue comes in:  what value do we put for allow?  Do we use the external address, such as ec2-127-0-0-1.compute-1.amazonaws.com?  Do we use the internal address, domU-00-00-00-00-00-00.compute-1.internal?  Or, do we use the ip address directly, say 127.0.0.1?  This will resurface later, too, in how the master will connect to the nodes.

Which one you choose does matter.  For example, if you use the ip address or the external AWS address, your AWS account will be charged for all data transfer and you will have to authorize port 4949 to allow the munin master to get through the firewall.  Ergo, it really makes sense to use the internal representation.  Furthermore, this allows you to keep the firewall closed, in which case you can put a nice little regex in the allow line to allow any of your instances to connect, thus saving yourself headache if your master external address changes, if you decide to switch the master to a different address, or for some reason add another master.  So, on AWS, the easiest allow line entry is:

allow ^\.*$

If you're not familiar with regex, that literally allows any munin master to connect, which means that it would be ridiculously stupid to do in any situation where your machines are not all behind a common firewall.  Make sure to restart each munin-node (/etc/init.d/munin-node restart) after editing the config file to ensure the changes are live.  Okay, now repeat that on as many machines/instances as you want to monitor.  For more options configuring a node, see the munin documentation.  When finished, continue.

Configuring the Munin Master

The config file for the munin master is /etc/munin/munin.conf.  The basic idea here is to define groups and hosts.  A simple group looks like:


[db.yourdomain.com]
     address domU-00-00-00-00-00-00.compute-1.internal


And here's what that all means:  You defined a group "yourdomain.com" and then the host "db.yourdomain.com" under that group.  Then, you said that the munin node located at that address is the host named db.yourdomain.com in the group yourdomain.com.  So, what is all this group nonsense?  The simple version is that it's just how hosts are organized, and that makes for a nice display on the web.  If you want to get more complex, you can configure different behaviors for groups or show totals by group.  For more info on this file, see the munin documentation.  

So, on to things that matter.  The name of the host (db.yourdomain.com) MUST match the value of host_name that the munin node at the address specified will report.  If it doesn't, then you won't see any data collected for the node.  The exception is if you enable use_node_name, then graphs will show up because that setting tells munin to fetch "all configured plugins associated with the given node, both plugins that report the nodes host_name and none at all," (source).  But, since use_node_name is not recommended, just make sure that the values of host_name in munin-node.conf an the name of the host in munin.conf are the same. 

Now Wait

You only have to wait 5 minutes at the most.  After that, you will be able to see the data collected from the web.  This is where a lot of blogs get impatient and tell you to run:

/usr/bin/munin-cron --force-root

which will update munin databases right away, but will also hijack the permissions from user "munin" to the root user, thereby denying the munin user access to those files at a later time, which will stop all updates because the cron job installed in /etc/cron.d/munin runs /usr/bin/munin-cron as munin, not root.  You know, if I recommended doing that in my blog, I'd feel like a jerk for leading people astray.  

Okay, patience is not one of your virtues and you don't want to wait 5 minutes?  Fine.  Just do this instead:


root@xxx:/root$ su munin --shell=/bin/bash
munin@xxx:/root$ /usr/bin/munin-cron


There, that will let you run munin-cron as "munin", which lets you see your update right away without hijacking file permissions.  Just hit ctrl+d when you're done to go back to whatever user you previously were.

Alright, hope that saves somebody some frustration.

JamLegend Invites

So, it seems like there are a shortage of JamLegend invites on the web... and since I'm one of the Founder's, I figured I could give out a few here : )

So, to sign up for JamLegend, head on over to http://www.jamlegend.com/register/shinealight and start jamming!

p.s.  No, I'm not going to review my own product (duh, that's the rest of the world's job!).  I'm just handing out a few more invites (around 75 or so with that link).  And this is to basically explain where the hell I have been the last month and a half and why I haven't written anything....

JavaMail SMTP Authentication

There are a couple ways to authenticate yourself when using JavaMail APIs to send e-mail through an SMTP server.

Here's one way:

public void sendMail(String fromAddress, 
                     String recipients, 
                     String subject, 
                     String content, 
                     String contentType, 
                     String smtpHost, 
                     int smtpPort, 
                     String username, 
                     String password) {
  try {

    Properties props = System.getProperties();
    Session  session = Session.getDefaltInstance(props,null);

    MimeMessage message = new MimeMessage(session);

    message.setFrom(fromAddress);
    message.setRecipients(Message.RecipientType.TO, InternetAddress.parse(recipients, false));

    message.setSubject(subject);
    message.setContent(content,contentType);
    message.setSentDate(new Date());

    Transport transport = session.getTransport("smtp");
    transport.connect(smtphost,smtpPort,username,password);
    transport.sendMessage(message,message.getAllRecipients());
    transport.close();

  } catch (AddressException e) {
    e.printStackTrace();
  } catch (MessagingException e) {
    e.printStackTrace();
  }
}

I don't like that way, because it's no good for anything but sending via an SMTP server with password based authentication. So, if I want to send e-mail through an SMTP server that has, say, a white-listed IP policy so I don't need to provide authentication, I have to write a whole new function that will duplicate a significant part of that code. Also, that long list of arguments to the function is just ugly, but it's what you're stuck with if you abstract a function that's hard-coded for SMTP password authentication.

Here's a better way, that is abstracted nicely:

public void sendMail(Properties props, 
                     Authenticator authenticator, 
                     String fromAddress, 
                     String recipients, 
                     String subject, 
                     String content, 
                     String contentType) {
  try {

    Session session = Session.getDefaultInstance(props, authenticator);

    Message message = new MimeMessage(session);

    message.setFrom(fromAddress);
    message.setRecipients(Message.RecipientType.TO, InternetAddress.parse(recipients, false));

    message.setSubject(subject);
    message.setContent(content,contentType);
    message.setSentDate(new Date());

    Transport.send(message);

  } catch (AddressException e) {
    e.printStackTrace();
  } catch (MessagingException e) {
    e.printStackTrace();
  }
}

The function above can be used to send e-mail with all sorts of configurations. All you have to do is pass it the Properties and, if you need to, an instance of a class extending javax.mail.Authenticator. Here's an example to send an e-mail identical to the first example above:

Properties props = System.getProperties();
props.put("mail.smtp.host","smtpHost");
props.put("mail.smtp.port","smtpPort");
props.put("mail.smtp.auth","true");

Authenticator auth = new Authenticator() {
  public PasswordAuthentication getPasswordAuthentication() {
    return new PasswordAuthentication("username","password");
  }
};

sendMail(props,auth,"no-reply@mysite.com","user@site.com","Hello World","Wow, big world.","text/plain");

And, to send an e-mail without authentication (as in the IP white-list policy):

Properties props = System.getProperties();
props.put("mail.smtp.host","smtpHost");
props.put("mail.smtp.port","smtpPort");

sendMail(props,null,"no-reply@mysite.com","user@site.com","Hello World","Wow, big world.","text/plain");

The great part is that the function sending the e-mail doesn't need to know anything about how it's going to do it, since the configuration is entirely passed in as arguments via the session and the optional authenticator. There is, however, a catch that I've noticed has tripped people up all over the internet. The values you set to your Properties object must all be of class String. For example, this works

Properties props = System.getProperties();
props.put("mail.smtp.host","mail.mysite.com");
props.put("mail.smtp.port","587");
props.put("mail.smtp.auth","true");

Whereas, this does not:

Properties props = System.getProperties();
props.put("mail.smtp.host","mail.mysite.com");
props.put("mail.smtp.port",587);
props.put("mail.smtp.auth",true);

It doesn't matter that the values for mail.smtp.port and mail.smtp.auth are an int and a boolean, respectively. If they are not specified as a String, Transport.sendMessage(message) will fail.

* As a disclaimer, I typed the above code, rather than copy and paste from a program I've compiled, so I apologize if there's a typo anywhere up there that prevents it from compiling.