Friday, November 9, 2007

Project, Part 2: Glassfish cluster

I came across a few tutorials, but none of them worked out of the box for me. So here's my version of how to set up a 2 machine glassfish cluster. Note, this process was completed on 2 Fedora 7 machines with Glassfish V2 final (aka b58g Promoted Build). If you're using a different operating system or a different build of glassfish, you might need to tweak the process.

For the rest of this post, here will be 2 machines: glassfish1 and glassfish2. Glassfish1 will be the domain administration server (DAS) and will house g1-a1 (glassfish1, node-agent 1) as well as g1-i1 (glassfish1, instance1). Glassfish2 will only have g2-a1 (glassfish2, node-agent 1) and g2-i1 (glassfish2, instance2). In all of the commands below, I reference the machines by their hostname (glassfish1 or glassfish2). I am able to do this because I have edited /etc/hosts on both machines to provide a mapping from the hostnames glassfish1 and glassfish2 to the physical IP for those machines. If you have not done this, and if you do not have another setup for DNS resolution, you'll need to refer to the machines by their physical IP.

You'll need to grab the glassfish installer from https://glassfish.dev.java.net//downloads/v2-b58g.html with wget or FTP it onto both glassfish1 and glassfish2.

If you don't have Sun's Java JDK 5 or 6 installed on your machines, you'll need that first. If you're running Fedora, you can find a guide in my post 'Project, Part 1: Fedora + Sun Java' for installing Sun's Java JDK 1.5 build 13 on Fedora 7.

Time to setup glassfish1:
  1. Copy the installer jar file to /opt
    • cp <glassfish-installer>.jar /opt
  2. Unpack the jar
    • java -Xmx256m -jar <glassfish-installer>.jar
  3. Add the execute permission to the ant stuff
    • chmod -R +x glassfish/lib/ant/bin
  4. Setup the cluster (this will create the DAS)
    • glassfish/lib/ant/bin/ant -f setup-cluster.xml
  5. Add the glassfish bin directory to the path so we can call asadmin from anywhere
    • emacs /etc/profile
    • In Fedora, there is a nice function pathmunge, already in /etc/profile, which will allow you to add directories to the path without accidentally creating duplicates, so add the following line before PATH is exported:
      • pathmunge /opt/glassfish/bin
  6. Start the default domain (domain1) with:
    • asadmin start-domain
  7. Create the first node-agent
    • asadmin create-node-agent --host glassfish1 --port 4848 g1-a1
  8. Create the cluster
    • asadmin create-cluster --host glassfish1 --port 4848 cluster1
  9. Create the first instance
    • asadmin create-instance --host glassfish1 --port 4848 --nodeagent g1-a1 --cluster cluster1 g1-i1
  10. Start the node-agent
    • asadmin start-node-agent --syncinstances=true g1-a1
    • Note: the syncinstances argument will start the associated instances and clusters as well, so you'll want to remember this for later.
If you made it that far, now it's time to setup glassfish2:
  1. cp <glassfish-installer>.jar /opt
  2. java -Xmx256m -jar <glassfish-installer>.jar
  3. chmod -R +x glassfish/lib/ant/lib
  4. glassfish/lib/ant/bin/ant -f setup-cluster.xml

    Note: the above will create a DAS on glassfish2... which we don't want, so you can remove the DAS after the script finishes with:

    rm -rf /opt/glassfish/domains/domain1

    If you want, you can avoid creating the DAS on glassfish2 in the first place by using the following command instead of the one above (#4):

    glassfish/lib/ant/bin/ant -f setup-cluster.xml \
    create-node-agent --Ddas.host=glassfish1

    That will create a node-agent with the default name "na".  If you want to specify the name of the node-agent, you'll need to edit setup-cluster.xml before running the above command.  Find the property "nodeagent.name" and change the value to whatever name you want:

    <property name="nodeagent.name" value="my-nodeagent-name" />

    Either way, running setup-cluster.xml, which creates the DAS, then removing the DAS and creating a node agent will provide the same result as running setup-cluster.xml with the option to only create the node-agent.

  5. Add /opt/glassfish/bin to the PATH as on glassfish1
  6. Create the second node-agent (If you didn't in step 4)
    • asadmin create-node-agent --host glassfish1 --port 4848 g2-a1
  7. Create a new instance
    • asadmin create-instance --host glassfish1 --port 4848 --nodeagent g2-a1 --cluster cluster1 g2-i1

  8. Start the second node agent
    • asadmin start-node-agent --syncinstances=true g2-a1

That's it! Well... sort of, there are a few things to note:
  1. Make sure that the firewall/SELinux/iptables rules are set to allow ports that are needed by Glassfish. These are, at the minimum:

    • 4848 (admin HTTP)
    • 8080 (HTTP)
    • 8181 (HTTPS)
    • 8686 (RMI)
    • 3700 (ORB)
    • 7676 (IMQ)

    and possible others if those are already in use. If your node-agents are having trouble communicating, try disabling the firewall & SELinux, and set iptables to accept all incoming connections. DO NOT DO THIS AS A LONG TERM SOLUTION. IT IS ONLY MEANT TO RULE OUT PORT COMMUNICATIONS AS A PROBLEM. Depending on your needs, you may be altering the port configurations anyway. I decided to implement a white-listing policy, where I deny all incoming connections except the following:

    • On glassfish1:
      • connections on port 8080 (for HTTP) from any IP
      • connections on port 8181 (HTTPS) from any IP
      • connections on port 4848 (HTTP admin GUI) from only my IP
      • connections on any port from the IP of glassfish2
    • On glassfish2:
      • connections on any port from the IP of glassfish1

    Thus, my servers are free to communicate with each other on any port they desire, but outside users may only connect to glassfish1 on 8080 and 8181, and only I can connect to glassfish1 on 4848. If you're messing with iptables, and you're not too familiar with *nix environments, you should probably have a *nix sys-admin check out your setup before it goes into production to make sure you haven't inadvertently created any security holes.

  2. If you're having trouble starting your instances, especially g2-i1 on glassfish2 you might have to edit your /etc/hosts file... I did. This is something that you should check if the following shows up in your server.log:

    [#|2007-11-05T12:19:03.832-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NAGT
    0038:Executing Synchronization for node-agent With DAS|#]

    [#|2007-11-05T12:19:12.920-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.tools.synchronization|_ThreadID=10;_ThreadNam
    e=main;|SYNC001: Unable to communicate with Domain Administration Server. Skipping synchronization.|#]

    [#|2007-11-05T12:19:12.922-0500|SEVERE|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NA
    GT0035:The NodeAgent failed to complete the intial synchronization with the DAS. Please make sure the DAS is running and is a
    ccessible from the NodeAgents server|#]

    [#|2007-11-05T12:19:14.925-0500|WARNING|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|N
    AGT0013:Stopping Node Agent...|#]

    Or, if the following shows up in your instance server.log:

    [#|2007-11-07T02:25:31.352-0500|WARNING|sun-appserver9.1|javax.jms|_ThreadID=10;_ThreadName=main;_RequestID=efb5ac8e-778b-412b-ae9e-3af1e95a8ee2;|[C4003]: Error occurred on connection creation [localhost:37676]. - cause: java.net.ConnectException: Connection refused|#]

    [#|2007-11-07T02:25:31.363-0500|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=10;_ThreadName=main;_RequestID=efb5ac8e-778b-412b-ae9e-3af1e95a8ee2;|java.lang.RuntimeException: MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
    at com.sun.messaging.jms.ra.LocalBrokerRunner.start(LocalBrokerRunner.java:366)
    at com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:447)
    at com.sun.enterprise.connectors.ActiveInboundResourceAdapter$1.run(ActiveInboundResourceAdapter.java:135)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.enterprise.connectors.ActiveInboundResourceAdapter.(ActiveInboundResourceAdapter.java:131)
    at com.sun.enterprise.connectors.system.ActiveJmsResourceAdapter.(ActiveJmsResourceAdapter.java:247)
    at com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(ActiveRAFactory.java:107)
    at com.sun.enterprise.connectors.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:300)
    at com.sun.enterprise.connectors.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:445)
    at com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(ConnectorRuntime.java:224)
    at com.sun.enterprise.jms.JmsProviderLifecycle.onStartup(JmsProviderLifecycle.java:428)
    at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:442)
    at com.sun.enterprise.server.ondemand.OnDemandServer.onStartup(OnDemandServer.java:120)
    at com.sun.enterprise.server.PEMain.run(PEMain.java:411)
    at com.sun.enterprise.server.PEMain.main(PEMain.java:338)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at com.sun.enterprise.server.PELaunch.main(PELaunch.java:412)
    |#]

    What's happening here is that the node-agent is not starting because the instance is failing due to an MQ startup problem As it turns out, the MQ needs the hostname to point to an absolute address, not 127.0.0.1 or localhost. If you edit your /etc/hosts file on glassfish1 to add:

    xxx.xxx.xxx.xxx glassfish1

    And on glassfish2 to add:

    xxx.xxx.xxx.xxx glassfish2

    then the problem should go away. Shalini, was kind enough to point that out to me and he has a blog post here that also mentions the problem. My original guide for doing this also came from Shalini's blog entry on setting up a 2 machine glassfish cluster using CLI.

13 comments:

The Curious said...

HI..

a NICE ARTICLES..

i want to edit the 9th step in machine 1 and 7th step in machine 2. you wrote: --node-agent

it should be : --nodeagent..

Thank you

The Curious
From Indonesia

Ryan said...

So it is. Thanks for noticing that! It's updated to reflect the change.

Alessandro said...

"I don't know if there is a way to specify the port and name for the node-agent at the time of creation"

I found it!!

In the setup-cluster.xml

property name="nodeagent.name" value="na"

Change the value before the command

./ant -f setup-cluster.xml create-node-agent -Ddas.host=hostname

;-)

Ryan said...

@alessandro

Great find! i figured there had to be a way but had never investigated it. Thanks for letting us all know! I've updated the post to note your comment.

Anonymous said...

./ant -f setup-cluster.xml create-node-agent -Ddas.host=hostname -Dnodeagent.name=your-na

should also do the trick but without the need to edit the buildfile.

Xie said...

Hi,Ryan Wilson,

I follow your's step deploy to debian 4 in VMware,but the final step :"asadmin start-node-agent --syncinstances=true g2-a1",when i run it,the system return :"Node Agent g2-a1 failed to startup. Please check the server log for more details.
CLI137 Command start-node-agent failed. "

I try it 6 times,every result is same.The password I never change it.

Please give me a advise,thank you very much.

Jason said...

Ryan,

Nice article and very helpful. One question I have about ports in this setup is the following:

Since your DAS is currently started, isn't it using the defaults ports (8080, 8081, 7676, 3700...) and your cluster instance is just using free ports. So any cluster instance deployed apps would not run on the default ports? How do you suggest allowing the cluster instance to use the default ports but still keep the admin console up and running on the default domain server? Thanks

Jason

Ryan said...

Jason,

Cluster instances do not run on the same ports as glassfish would with the development profile (8080,8081), instead, in my experience, they run on ports 38080, 38081, etc. The admin server runs on 4848, but the cluster instances won't use that port so there's no conflict there.

The cluster setup assumes a load balancer in front of this whole thing that distributes requests among the cluster instances (i.e. http://your.domain.com:80/ is actually a load balancer, not any of the glassfish instances). So, basically, the cluster instances aren't intended to be accessed directly (though, I have found that accessing them directly via their ip:port can be quite useful for testing specific instances).

That said, there might be a way to make the glassfish cluster instances run on 8080, instead of 38080, but I haven't ever tried so I'm sure how you might do that. I'd say reconsider why you want to do that, because having users go directly to a glassfish instance without a load balancer distributing requests kind of defeats the purpose of the cluster because at that point one instance is handling everything. Perhaps I misunderstood the question? Otherwise, I hope that helps.

aaa kitty20101122 said...

pandora charms sale
chrome hearts online store
burberry scarf
adidas nmd
michael kors outlet
Kanye West shoes
yeezy boost
true religion jeans
michael kors handbags
asics running shoes

mahesh said...

Excellent post! We will be linking to this great article on our website. Keep up the good writing. funny fish names , funny pet fish names

Games 2 Girls said...

I understand what you bring it very meaningful and useful, thanks.
sushi-cat2.com

mary Brown said...

Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java EE Online Training from India. or learn thru Java EE Online Training from India . Nowadays Java has tons of job opportunities on various vertical industry.

Anonymous said...

can dogs have tylenol
can i give my dog tylenol
tylenol dosing for dogs
tylenol for dogs dose
what can you give a dog for pain