Project, Part 2: Glassfish cluster

I came across a few tutorials, but none of them worked out of the box for me. So here's my version of how to set up a 2 machine glassfish cluster. Note, this process was completed on 2 Fedora 7 machines with Glassfish V2 final (aka b58g Promoted Build). If you're using a different operating system or a different build of glassfish, you might need to tweak the process.

For the rest of this post, here will be 2 machines: glassfish1 and glassfish2. Glassfish1 will be the domain administration server (DAS) and will house g1-a1 (glassfish1, node-agent 1) as well as g1-i1 (glassfish1, instance1). Glassfish2 will only have g2-a1 (glassfish2, node-agent 1) and g2-i1 (glassfish2, instance2). In all of the commands below, I reference the machines by their hostname (glassfish1 or glassfish2). I am able to do this because I have edited /etc/hosts on both machines to provide a mapping from the hostnames glassfish1 and glassfish2 to the physical IP for those machines. If you have not done this, and if you do not have another setup for DNS resolution, you'll need to refer to the machines by their physical IP.

You'll need to grab the glassfish installer from https://glassfish.dev.java.net//downloads/v2-b58g.html with wget or FTP it onto both glassfish1 and glassfish2.

If you don't have Sun's Java JDK 5 or 6 installed on your machines, you'll need that first. If you're running Fedora, you can find a guide in my post 'Project, Part 1: Fedora + Sun Java' for installing Sun's Java JDK 1.5 build 13 on Fedora 7.

Time to setup glassfish1:

1. Copy the installer jar file to /opt
• cp <glassfish-installer>.jar /opt
2. Unpack the jar
   
• java -Xmx256m -jar <glassfish-installer>.jar
3. Add the execute permission to the ant stuff
• chmod -R +x glassfish/lib/ant/bin

4. Setup the cluster (this will create the DAS)
• glassfish/lib/ant/bin/ant -f setup-cluster.xml
5. Add the glassfish bin directory to the path so we can call asadmin from anywhere
• emacs /etc/profile
• In Fedora, there is a nice function pathmunge, already in /etc/profile, which will allow you to add directories to the path without accidentally creating duplicates, so add the following line before PATH is exported:
  
• pathmunge /opt/glassfish/bin
6. Start the default domain (domain1) with:
• asadmin start-domain
7. Create the first node-agent
• asadmin create-node-agent --host glassfish1 --port 4848 g1-a1
8. Create the cluster
• asadmin create-cluster --host glassfish1 --port 4848 cluster1
9. Create the first instance
• asadmin create-instance --host glassfish1 --port 4848 --nodeagent g1-a1 --cluster cluster1 g1-i1
10. Start the node-agent
• asadmin start-node-agent --syncinstances=true g1-a1
• Note: the syncinstances argument will start the associated instances and clusters as well, so you'll want to remember this for later.
  

If you made it that far, now it's time to setup glassfish2:

1. cp <glassfish-installer>.jar /opt
2. java -Xmx256m -jar <glassfish-installer>.jar
3. chmod -R +x glassfish/lib/ant/lib
4. glassfish/lib/ant/bin/ant -f setup-cluster.xml
   
   Note: the above will create a DAS on glassfish2... which we don't want, so you can remove the DAS after the script finishes with:
   
rm -rf /opt/glassfish/domains/domain1

   If you want, you can avoid creating the DAS on glassfish2 in the first place by using the following command instead of the one above (#4):
   
glassfish/lib/ant/bin/ant -f setup-cluster.xml \
create-node-agent --Ddas.host=glassfish1

   That will create a node-agent with the default name "na".  If you want to specify the name of the node-agent, you'll need to edit setup-cluster.xml before running the above command.  Find the property "nodeagent.name" and change the value to whatever name you want:
   
<property name="nodeagent.name" value="my-nodeagent-name" />

   Either way, running setup-cluster.xml, which creates the DAS, then removing the DAS and creating a node agent will provide the same result as running setup-cluster.xml with the option to only create the node-agent.


5. Add /opt/glassfish/bin to the PATH as on glassfish1
6. Create the second node-agent (If you didn't in step 4)
• asadmin create-node-agent --host glassfish1 --port 4848 g2-a1
7. Create a new instance
• asadmin create-instance --host glassfish1 --port 4848 --nodeagent g2-a1 --cluster cluster1 g2-i1
  
  
8. Start the second node agent
• asadmin start-node-agent --syncinstances=true g2-a1

That's it! Well... sort of, there are a few things to note:

1. Make sure that the firewall/SELinux/iptables rules are set to allow ports that are needed by Glassfish. These are, at the minimum:
   
   
   • 4848 (admin HTTP)
   • 8080 (HTTP)
   • 8181 (HTTPS)
   • 8686 (RMI)
   • 3700 (ORB)
   • 7676 (IMQ)
     
   
   and possible others if those are already in use. If your node-agents are having trouble communicating, try disabling the firewall & SELinux, and set iptables to accept all incoming connections. DO NOT DO THIS AS A LONG TERM SOLUTION. IT IS ONLY MEANT TO RULE OUT PORT COMMUNICATIONS AS A PROBLEM. Depending on your needs, you may be altering the port configurations anyway. I decided to implement a white-listing policy, where I deny all incoming connections except the following:
   
   
   • On glassfish1:
   • connections on port 8080 (for HTTP) from any IP
     
   • connections on port 8181 (HTTPS) from any IP
     
   • connections on port 4848 (HTTP admin GUI) from only my IP
     
   • connections on any port from the IP of glassfish2
   • On glassfish2:
   • connections on any port from the IP of glassfish1
     
   
   Thus, my servers are free to communicate with each other on any port they desire, but outside users may only connect to glassfish1 on 8080 and 8181, and only I can connect to glassfish1 on 4848. If you're messing with iptables, and you're not too familiar with *nix environments, you should probably have a *nix sys-admin check out your setup before it goes into production to make sure you haven't inadvertently created any security holes.
   


2. If you're having trouble starting your instances, especially g2-i1 on glassfish2 you might have to edit your /etc/hosts file... I did. This is something that you should check if the following shows up in your server.log:
   
   [#|2007-11-05T12:19:03.832-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NAGT
   0038:Executing Synchronization for node-agent With DAS|#]
   
   [#|2007-11-05T12:19:12.920-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.tools.synchronization|_ThreadID=10;_ThreadNam
   e=main;|SYNC001: Unable to communicate with Domain Administration Server. Skipping synchronization.|#]
   
   [#|2007-11-05T12:19:12.922-0500|SEVERE|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NA
   GT0035:The NodeAgent failed to complete the intial synchronization with the DAS. Please make sure the DAS is running and is a
   ccessible from the NodeAgents server|#]
   
   [#|2007-11-05T12:19:14.925-0500|WARNING|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|N
   AGT0013:Stopping Node Agent...|#]
   
   Or, if the following shows up in your instance server.log:
   
   [#|2007-11-07T02:25:31.352-0500|WARNING|sun-appserver9.1|javax.jms|_ThreadID=10;_ThreadName=main;_RequestID=efb5ac8e-778b-412b-ae9e-3af1e95a8ee2;|[C4003]: Error occurred on connection creation [localhost:37676]. - cause: java.net.ConnectException: Connection refused|#]
   
   [#|2007-11-07T02:25:31.363-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=10;_ThreadName=main;_RequestID=efb5ac8e-778b-412b-ae9e-3af1e95a8ee2;|java.lang.RuntimeException: MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
   at com.sun.messaging.jms.ra.LocalBrokerRunner.start(LocalBrokerRunner.java:366)
   at com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:447)
   at com.sun.enterprise.connectors.ActiveInboundResourceAdapter$1.run(ActiveInboundResourceAdapter.java:135)
   at java.security.AccessController.doPrivileged(Native Method)
   at com.sun.enterprise.connectors.ActiveInboundResourceAdapter.(ActiveInboundResourceAdapter.java:131)
   at com.sun.enterprise.connectors.system.ActiveJmsResourceAdapter.(ActiveJmsResourceAdapter.java:247)
   at com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(ActiveRAFactory.java:107)
   at com.sun.enterprise.connectors.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:300)
   at com.sun.enterprise.connectors.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:445)
   at com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(ConnectorRuntime.java:224)
   at com.sun.enterprise.jms.JmsProviderLifecycle.onStartup(JmsProviderLifecycle.java:428)
   at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:442)
   at com.sun.enterprise.server.ondemand.OnDemandServer.onStartup(OnDemandServer.java:120)
   at com.sun.enterprise.server.PEMain.run(PEMain.java:411)
   at com.sun.enterprise.server.PEMain.main(PEMain.java:338)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:585)
   at com.sun.enterprise.server.PELaunch.main(PELaunch.java:412)
   |#]
   
   What's happening here is that the node-agent is not starting because the instance is failing due to an MQ startup problem As it turns out, the MQ needs the hostname to point to an absolute address, not 127.0.0.1 or localhost. If you edit your /etc/hosts file on glassfish1 to add:
   
   xxx.xxx.xxx.xxx glassfish1
   
   And on glassfish2 to add:
   
   xxx.xxx.xxx.xxx glassfish2
   
   then the problem should go away. Shalini, was kind enough to point that out to me and he has a blog post here that also mentions the problem. My original guide for doing this also came from Shalini's blog entry on setting up a 2 machine glassfish cluster using CLI.
   

A response to "Have you seen the news?"

I normally reserve this blog for technology posts, but this is an exception.

Over the past day, I've received a couple emails with links to news stories, regarding the tragic death of a British girl studying abroad in Italy, accompanied by the question, "Have you seen this?"  I'm asked because I happened to attend the same high school (within a couple years) as one of the suspects.  Too that question, here is my answer:

Yes, I've heard, so have you.  Now let's stop endorsing the unrestrained sensationalism of the media.  

If you browse around for a while, you'll see that various articles paint different pictures slandering different individuals. In situations like this, what normally is viewed as typical college debauchery is recast as criminally indicative behavior. After all, profit motivated media needs to sell headlines, and in the realm of "Buy this paper!", nothing beats a sexual deviant with a murderous dark side.  The deplorable part is that it doesn't matter if the individual fits the headline, only that the headline sells the paper.

Italian police are under enormous pressure, which the strong media attention only increases. They can't move too quickly, lest they appear hasty or negligent. They can't be too methodical, lest they appear incapable or overwhelmed. In either case, it is a tightrope to avoid having their qualifications called into question. Too many want the answer now, and too few want the right answer. It's a terrible situation all around and I feel for each person caught in this dragnet of media attention. I sympathize with all, each for a unique reason.

At this point, available information is limited to media sensationalism with underlying culture-stereotypic allegations redolent of long existing cultural partisanship and prejudice. The most important thing now is to acknowledge the havoc and irreparable change that has and will continue to affect the individuals and their families. The court of public opinion has no jurisdiction here, and we've all had a time when we deserved more privacy and respect than we received from overly eager interested parties. If out of nothing more than respect for the families, I hope that the hype will calm down.

And no, I'm not going to post links to any of the articles.  If you have no qualms about supporting sensationalistic media, then I'm sure your Google skills are already top notch.

Project, Part 1: Fedora + Sun Java

Glassfish needs Java JDK 5, or 6. Fedora doesn't come with the Sun JDK, but rather GNU Java. Additionally, unlike my experiences with Ubuntu, Fedora does not have a recent implementation of the JDK available from a repository. So I was confronted with this choice:

1. Try to force Glassfish to work with GNU Java, the default in Fedora, or
2. Try to force Fedora to use Sun's Java

I decided on the later after reading that none of the java language features new in Java 1.5 are implemented yet in gcj. At the same time, the blog consensus was that installing Sun's Java on Fedora was not straight forward. First, the different java versions could interfere with each other; and second, if installed incorrectly, Sun's java could be accidentally, and unintentionally, 'upgraded' to gcj when installing a system update.

For the most part, I followed a combination of these blogs: blog1, blog2, blog3, blog4. Alas, none of them worked despite following step by step, but in doing so, I learned a lot about what was going on.

The first problem seemed to be related to a dependency difference between JDK 1.5.0 build 11 or 12 and JDK 1.5.0 build 13, which is what I was using. Although each blog listed certain dependencies, none listed the dependency problem I was having. I still don't know what libXtst is, but without it I was unable to install the JDK. With it, I was able to install just fine.

The second problem had to do with the order of installing RPMs. If I tried to install them all at once with a yum -y --nogpgcheck install /usr/src/redhat/RPMS/i586/java*.rpm, it would fail because of an error in the SPEC file dealing with font package locations (detailed here, in the Current Issues section). However, if I installed only java-1.5.0-sun-1.5.0.13-1jpp.i586.rpm first, and then installed the rest, I ended up with a working version (though the font problem remained).

A third problem was that blog1, referenced above, which involved rebuilding the java RPMs via rpmbuild -ba java-1.x.0-sun.spec resulted in a working JRE, but no JDK.

Here is the final version of what worked for me:

1. Some prerequisites
   
   1. yum -y install rpmdevtools jpackage-utils rpm-build libXp unixODBC unixODBC-devel libXtst yum-priorities
   2. rpm -Uvh http://www.fedorafaq.org/yum http://rpm.livna.org/livna-release-7.rpm
   
   
2. Get the JPackage Java RPM with
   wget http://mirrors.dotsrc.org/jpackage/1.7/generic/SRPMS.non-free/java-1.5.0-sun-1.5.0.13-1jpp.nosrc.rpm (lists at http://mirrors.dotsrc.org/jpackage/1.7/generic/SRPMS.non-free/)


3. Download the right JDK binary from Sun (not the self-extracting RPM) for the java version and build version of the JPackage RPM into /usr/src/redhat/SOURCES/


4. recreate the java rpm
   
   1. rpmbuild --rebuild java-1.5.0-sun*src.rpm
   2. should create new rpms in /usr/src/redhat/RPMS/i586/
   
   
5. install new java rpms (need the --nogpg option because locally created rpms are not signed)
   
   1. yum -y --nogpgcheck install /usr/src/redhat/RPMS/i586/java-1.5.0-sun-1.5.0.13-1jpp.i586.rpm
   2. yum -y --nogpgcheck install /usr/src/redhat/RPMS/i586/java*.rpm
   
   
6. check java version with java -version, and that the JDK is installed with javac --help. If the version is not right, or it didn't work, change the default Java system to Sun's with alternatives --config java, and select the right jvm


7. Set the JAVA_HOME environment variable
   
   
   
   1. Add to /etc/profile
JAVA_HOME="/usr/lib/jvm/java"
export JAVA_HOME
   
   
   2. run source /etc/profile to read in profile, and check it worked with echo $JAVA_HOME
      
   
   

That's it, now for Glassfish...

Fedora 7 + Sun Java + Glassfish + MySQL

Start:
6 - Servers with a base install of Fedora 7
? - Time and effort


End result:
1 - 2 machine, load-balanced, application server cluster with failover and in-memory replication
1 - 4 machine, load-balanced, MySQL NDB database cluster with 2 storage nodes, 1 sql / load balancer node, and 1 management / load balancer node


The process will be described in a serious of forthcoming posts.