Friday, July 8, 2011

Synchronous Mule Service - exception handling - Mule 3

In a previous post I outlined a solution to have an exception send back when doing a synchronous service call over JMS in Mule. In the intro of that post you can find the exact problem description. This post is about the exact same topic. The only difference in setup is that we'll use Mule 3.

Upgrading the exception handler from Mule 2 to 3
All the facets of migrating to a new version of Mule deserves a separate post, so I'll stick to the exception handling problem described before.

Version 3 of the Mule DefaultServiceExceptionStrategy deprecates the defaultHandler method. The new one to use is doHandleException. This new method gives you direct access to the MuleEvent so that's an improvement. In my newer implementation I can also get the endpoint more easy from the muleEvent and it doesn't require some Mule registry lookup.

Getting a hold of the replyTo object has become a bit more difficult and it forced me to narrow the usage of this class to JMS only. This is because there is no longer a convenient method like muleEvent.getMessage().getReplyTo(). Now you need to resolve it from a property, which is specific to the JMS connector.

For those interested I included all the code at the bottom of this post.

Using flow
Even though I was pretty pleased about the code I wrote, I still wondered if the new Flow concept in Mule 3 wouldn't allow me to get rid of the custom exception handler. After all it is not so strange that you want to be able to respond to a JMS request in case something goes wrong, not?

The thing I actually found was that using a Flow makes it worse, because by default flow doesn't reply at all to a JMS request: not in case of success or error. There is actually a bug open for this on JIRA: http://www.mulesoft.org/jira/browse/MULE-5307

After fiddling around a bit I did find that everything does work as expected with a VM endpoint! So in case you have an inbound-vm endpoint that is marked as exchange-pattern="request-response", you will always get a response (success and error cases).
And what is more: it also works if you have an inbound JMS endpoint and forward the request to a VM inpoint of a flow. It looks like the configuration below. Notice that you still need a response transformer that will transform the Mule exception payload to whatever you need. If you do not have a transformer you will get an empty message on your JMS response queue. This is because the JMS connector ignores the exception payload and takes the normal payload, which is null.

<mule>
    <jms:endpoint name="jms.queue.request" queue="${mq.queue.request.in}"
                  transformer-refs="JmsToObject"
                  responseTransformer-refs="exPayloadToResponse objectToJms"/>
    <vm:endpoint name="vm.request" address="vm://vm.request"/>

    <model>
        <service name="requestViaJMS">
            <inbound>
                <inbound-endpoint ref="jms.queue.request"/>
            </inbound>
            <outbound>
                <pass-through-router>
                    <outbound-endpoint ref="vm.request" exchange-pattern="request-response"/>
                </pass-through-router>
            </outbound>
        </service>
    </model>

    <flow name="requestViaVM">
        <inbound-endpoint ref="vm.request" exchange-pattern="request-response"/>
        <enricher ... />
        <transformer ... />
        <component ... />
    </flow>
</mule>

You would be right to argue that the above is exactly what is provided by the Mule Bridge pattern (also new in version 3). However, the bridge implementation suffers the same defect as the flow, so this does not bring a solution.

Conclusion
The most elegant solution in Mule 3 for exception handling on synchronous JMS flows is to hide them after a VM endpoint and simply pass through the JMS message.

Custom exceptionStrategy code
public final class ReqRepServiceExceptionStrategy extends DefaultServiceExceptionStrategy {
    public static final String REQ_REP_SERVICE_EXCEPTION_STRATEGY_REPLY_SENT = "REQ_REP_SERVICE_EXCEPTION_STRATEGY_REPLY_SENT";
    private final Logger logger = LoggerFactory.getLogger(ReqRepServiceExceptionStrategy.class);

    @Override
    protected void doHandleException(Exception e, MuleEvent muleEvent) {
        super.doHandleException(e, muleEvent);
        final ImmutableEndpoint inboundEp = muleEvent.getEndpoint();
        final boolean isReqRep = MessageExchangePattern.REQUEST_RESPONSE.equals(inboundEp.getExchangePattern());
        //only process replies for jms endpoints
        if (!isEventAlreadyProcessed(muleEvent) & inboundEp.isProtocolSupported(JmsConnector.JMS) & isReqRep) {
            final MuleMessage replyMessage = new DefaultMuleMessage(null, muleContext);
            replyMessage.setExceptionPayload(new DefaultExceptionPayload(e));
            try {
                final Object replyTo = getReplyTo(muleEvent.getMessage());
                final ReplyToHandler replyToHandler = getReplyToHandler(inboundEp);
                processReplyTo(muleEvent, replyMessage, replyToHandler, replyTo);
            } catch (MuleException me) {
                logger.error("Cannot reply from Exception Strategy.", me);
            }
        } else {
            logger.info("MuleEvent already processed once by this handler, not replying again.");
        }
    }

    private boolean isEventAlreadyProcessed(final MuleEvent muleEvent) {
        boolean eventAlreadyProcessed = false;
        final Object replyAlreadySent = muleEvent.getSession().getProperty(REQ_REP_SERVICE_EXCEPTION_STRATEGY_REPLY_SENT);
        if (replyAlreadySent != null && Boolean.class.isInstance(replyAlreadySent)) {
            eventAlreadyProcessed = Boolean.class.cast(replyAlreadySent);
        }
        return eventAlreadyProcessed;
    }

    private Object getReplyTo(final MuleMessage message) throws MuleException {
        final Object replyTo = message.getOutboundProperty(JmsConstants.JMS_REPLY_TO);
        if (replyTo == null) {
            throw new DefaultMuleException(MessageFactory.createStaticMessage(
                    "There is no jms-reply-to specified on this endpoint"));
        }
        return replyTo;
    }

    private ReplyToHandler getReplyToHandler(final ImmutableEndpoint endpoint) throws MuleException {
        final ReplyToHandler replyToHandler = ((AbstractConnector) endpoint.getConnector()).getReplyToHandler(endpoint);
        if (replyToHandler == null) {
            throw new DefaultMuleException(MessageFactory.createStaticMessage(
                    "There is no replyToHandler specified on this endpoint"));
        }
        final List responseTransformers = endpoint.getResponseTransformers();
        if (responseTransformers != null && responseTransformers.size() > 0) {
            replyToHandler.setTransformers(responseTransformers);
        }
        return replyToHandler;
    }

    private void processReplyTo(final MuleEvent event, final MuleMessage result, final ReplyToHandler replyToHandler,
                                final Object replyTo) throws MuleException {
        final String requestor = result.getOutboundProperty(MuleProperties.MULE_REPLY_TO_REQUESTOR_PROPERTY);
        if (((requestor != null && !requestor.equals(event.getFlowConstruct().getName())) || requestor == null)) {
            replyToHandler.processReplyTo(event, result, replyTo);
            event.getSession().setProperty(REQ_REP_SERVICE_EXCEPTION_STRATEGY_REPLY_SENT, Boolean.TRUE);
            logger.info("Reply send for this MuleEvent to " + replyTo.toString());
        }
    }
}

Wednesday, March 9, 2011

Synchronous Mule Service - exception handling

The problem

We have a mule service that is defined as synchronous. This means that it will handle incoming messages in the same thread as the endpoint that receives it. A high level overview of this is in the picture below.
What we found is that there is different behaviour depending on how the service is called, especially when it comes to exception handling. During our initial tests we called this service from the mule client in a synchronous way. In this case the default exception handler made sure we got an answer even if an exception was thrown.
In a production environment however we call the service asynchronously via JMS. In that use case the client timed out.

Below is a schematic representation of what happens in Mule in case everything goes well and with exception handling. Note that the client calls the service in an async way and that Mule executes the service in a sync way.



The behaviour we want is that Mule always returns a response even in the case of exception. This avoids the service clients to timeout while waiting for a response. To achieve this we need to write a custom exception strategy for Mule.

The solution ingredients

1) A custom exception strategy, some of the code is below. This code will make sure that a reply is send always. Since the exception strategy can be called multiple times, we'll store in the MuleSession if a reply is send already. There is also some specific handling needed for global endpoints and normal endpoints.
public final class ReqRepServiceExceptionStrategy extends DefaultServiceExceptionStrategy {
...
@Override
protected void defaultHandler(Throwable t) {
super.defaultHandler(t);
final MuleEvent muleEvent = RequestContext.getEvent();
if (!isEventAlreadyProcessed(muleEvent)) {
final MuleMessage replyMessage = new DefaultMuleMessage(null);
replyMessage.setExceptionPayload(new DefaultExceptionPayload(t));
ReplyToHandler replyToHandler;
try {
replyToHandler = getReplyToHandler(muleEvent.getMessage(), retrieveInboundEndpoint(muleEvent));
processReplyTo(muleEvent, replyMessage, replyToHandler, muleEvent.getMessage().getReplyTo());
} catch (MuleException e) {
logger.error("Cannot reply from Exception Strategy.", e);
}
} else {
logger.info("MuleEvent already processed once by this handler, not replying again.");
}
}

private ImmutableEndpoint retrieveInboundEndpoint(final MuleEvent muleEvent) throws MuleException {
InboundEndpoint inboundEndpoint;
String originatingAddress = (String) muleEvent.getMessage().getProperty(MuleProperties.MULE_ORIGINATING_ENDPOINT_PROPERTY);
//if not a global endpoint, will start with prefix
if (originatingAddress != null && originatingAddress.startsWith(ENDPOINT_PREFIX)) {
originatingAddress = originatingAddress.substring(ENDPOINT_PREFIX.length());
originatingAddress = originatingAddress.replaceFirst("\\.", "://");
}
inboundEndpoint = muleEvent.getMuleContext().getRegistry().lookupEndpointFactory().getInboundEndpoint(originatingAddress);
if (inboundEndpoint == null) {
throw new DefaultMuleException(MessageFactory.createStaticMessage("Cannot find original inbound endpoint for this message."));
}
return inboundEndpoint;
}

private boolean isEventAlreadyProcessed(final MuleEvent muleEvent) {
boolean eventAlreadyProcessed = false;
final Object replyAlreadySent = muleEvent.getSession().getProperty(REQ_REP_SERVICE_EXCEPTION_STRATEGY_REPLY_SENT);
if (replyAlreadySent != null && Boolean.class.isInstance(replyAlreadySent)) {
eventAlreadyProcessed = Boolean.class.cast(replyAlreadySent);
}
return eventAlreadyProcessed;
}
...
}

2) Add this exception handler to the Mule configuration. The configuration below is from an integration test: it will always throw a runtime exception in the component. This allows us to test the new exception strategy.
<service name="customHandler">
<inbound>
<vm:inbound-endpoint address="vm://custom.request" synchronous="true"/>
</inbound>
<component class="be.i8c.mule.service.RuntimeExComponent">
<custom-exception-strategy class="be.i8c.mule.service.ReqRepServiceExceptionStrategy"/>
</component>
</service>

3) Test the new exception handler and see it work.
@Test
public void testAsyncCustomExceptionStrategy() throws Exception {
final MuleMessage request = createRequest("UNIT TEST MESSAGE","vm://custom.reply");
final MuleMessage response = doAsyncReqRep(request, "vm://custom.request","vm://custom.reply");

assertNotNull(response);
assertNotNull(response.getExceptionPayload());
assertTrue(response.getExceptionPayload().getException().getCause().getMessage().indexOf(RuntimeExComponent.EX_MSG) >= 0);
}



Wednesday, February 23, 2011

JMS speed test: ActiveMQ vs HornetQ

A while ago I was asked by a client to evaluate different open source JMS providers. The ultimate goal was to set up a highly available messaging system that can manage high throughput.

Me and my colleagues dug deep in our memories and on the internet to find all of the open source JMS providers. And there are plenty. In the end we looked at the following:
  • ActiveMQ
  • OpenMQ
  • RabbitMQ
  • OpenJMS
  • HornetQ
Besides the requirements of high availability and throughput there were some other requirements:
  • must be JMS 1.1 compliant
  • must be easy to set up and administer
  • vibrant community for support
Because of these reasons we quickly abandoned OpenJMS, which seems to have stopped evolving somewhere in 2006.
RabbitMQ is not JMS compliant, which we really need.
OpenMQ was dropped a little later, since in the first performance tests we found that it was noticably slower than ActiveMQ and HornetQ.

In a next phase we did quite extensive load tests on HornetQ and ActiveMQ, which I'll summarize below.

For our tests we used the following setup:
3 similar machines with 2 quad cores, 8GB of RAM, RHEL 5, Java Hotspot VM 64 bit (1.6.0_21-b06). 2 machines were used for hosting the JMS providers. One machine contains a master, the other a backup instance. The third machine is used for generating load.

The load generation is done using the Sonic test harness. This framework allowed us to generate load on the the JMS providers with different producers and consumers of the JMS queues.

These are some of the results we got (based on size of message, whether messages are persisted, if a transaction is used, number of concurrent producers and receivers):


As you can see both JMS providers are about equal when you look at non peristent messaging.BUT when looking at persistent messaging, HornetQ is just amazing. The throughput you get there is just mindblowing. This has without a doubt to do with the Asynchronous IO feature of HornetQ. This is only available on *NIX based systems but it is well worth it when you are looking for a performant open source JMS provider.

There are other sources of comparisons like these, but they were in our opinion either biased or did not give us enough insight for our situation:

Monday, February 7, 2011

RHEL: install Yum and add DVD as repository

Installing Yum using RPM. As you can see this is done by discovering the dependencies the hard way, exactly the reason why we need Yum.

[root@i8c-ODB11R2 Server]# rpm -i yum-3.2.22-26.el5.noarch.rpm
warning: yum-3.2.22-26.el5.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 1e 5e0159
error: Failed dependencies:
python-elementtree is needed by yum-3.2.22-26.el5.noarch
python-iniparse is needed by yum-3.2.22-26.el5.noarch
python-sqlite is needed by yum-3.2.22-26.el5.noarch
rpm-python is needed by yum-3.2.22-26.el5.noarch
urlgrabber >= 3.1.0 is needed by yum-3.2.22-26.el5.noarch
yum-metadata-parser >= 1.1.0 is needed by yum-3.2.22-26.el5.noarch
Installing the Yum dependencies:
[root@i8c-ODB11R2 Server]# rpm -i python-elementtree-1.2.6-5.x86_64.rpm python-iniparse-0.2.3-4.el5.noarch.rpm python-sqlite-1.1.7-1.2.1.x86_64.rpm rpm-python-4.4.2.3-18.el5.x86_64.rpm python-urlgrabber-3.1.0-5.el5.noarch.rpm m2crypto-0.16-6.el5.6.x86_64.rpm python-urlgrabber-3.1.0-5.el5.noarch.rpm yum-metadata-parser-1.1.2-3.el5.x86_64.rpm yum-3.2.22-26.el5.noarch.rpm libxml2-python-2.6.26-2.1.2.8.0.1.x86_64.rpm createrepo-0.4.11-3.el5.noarch.rpm

Once Yum is installed, you need a Yum repository to install packages from and allow Yum to search the metadata. The repository is created using the createrepo command. We'll create the repository under /root/yumrepo
[root@i8c-ODB11R2 Server]# mkdir /root/yumrepo
[root@i8c-ODB11R2 Server]# cd /root/yumrepo
[root@i8c-ODB11R2 Server]# createrepo -vpo /root/yumrepo /media/
[root@i8c-ODB11R2 Server]# .... listing all packages ...
[root@i8c-ODB11R2 Server]#
3180/3187 - VT/qspice-0.3.0-54.el5.x86_64.rpm
3181/3187 - VT/qspice-libs-0.3.0-54.el5.x86_64.rpm
3182/3187 - VT/qspice-libs-devel-0.3.0-54.el5.x86_64.rpm
3183/3187 - VT/virt-manager-0.6.1-12.el5.x86_64.rpm
3184/3187 - VT/virt-viewer-0.0.2-3.el5.x86_64.rpm
3185/3187 - VT/xen-3.0.3-105.el5.x86_64.rpm
3186/3187 - VT/xen-devel-3.0.3-105.el5.i386.rpm
3187/3187 - VT/xen-devel-3.0.3-105.el5.x86_64.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Now under /root/yumrepo add symbolic links to the folders of the DVD. Of course when you want to use the repository the DVD must be mounted (to /media in this case).
[root@i8c-ODB11R2 yumrepo]# ln -s /media/Server/ Server
[root@i8c-ODB11R2 yumrepo]# ln -s /media/VT VT
[root@i8c-ODB11R2 yumrepo]# ln -s /media/images/ images
[root@i8c-ODB11R2 yumrepo]# ln -s /media/isolinux/ isolinux
[root@i8c-ODB11R2 yumrepo]# ln -s /media/Cluster Cluster
[root@i8c-ODB11R2 yumrepo]# ln -s /media/ClusterStorage/ ClusterStorage

Next thing is to make the repository known to Yum. This is done by creating a file /etc/yum.repos.d/DVD.repo with the following content:
[dvd]
name=RHEL5 DVD
baseurl=file:///root/yumrepo
enabled=1
gpgcheck=0
Now this is done you can use the repo by issuing commands like "yum install" or "yum update", as shown below:
[root@i8c-ODB11R2 yumrepo]# yum update
dvd | 951 B 00:00
dvd/primary | 830 kB 00:00
dvd 3187/3187
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.18-194.0.0.0.3.el5 set to be installed
---> Package kernel-headers.x86_64 0:2.6.18-194.0.0.0.3.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================
Package Arch Version Repository Size
=================================================================================================================================
Installing:
kernel x86_64 2.6.18-194.0.0.0.3.el5 dvd 20 M
Updating:
kernel-headers x86_64 2.6.18-194.0.0.0.3.el5 dvd 1.1 M

Transaction Summary
=================================================================================================================================
Install 1 Package(s)
Upgrade 1 Package(s)

Total download size: 21 M
Is this ok [y/N]: y
Downloading Packages:
---------------------------------------------------------------------------------------------------------------------------------
Total 12 GB/s | 21 MB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : kernel 1/3
Updating : kernel-headers 2/3
Cleanup : kernel-headers 3/3

Installed:
kernel.x86_64 0:2.6.18-194.0.0.0.3.el5

Updated:
kernel-headers.x86_64 0:2.6.18-194.0.0.0.3.el5

Complete!

Friday, December 10, 2010

Command Line JMX + SSL

The tool I use for command line JMX and which I think really works well is http://wiki.cyclopsgroup.org/jmxterm

You can download and start the single jar as shown below. In this example I'm connecting to a HornetQ instance.
[jeroen@homesrv~]# java -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://192.168.1.3:10995/jmxrmi
Welcome to JMX terminal. Type "help" for available commands.
$>
Next you can fetch a list of all the available beans (not showing all the beans):
$>beans #domain = JMImplementation: JMImplementation:type=MBeanServerDelegate #domain = com.sun.management: com.sun.management:type=HotSpotDiagnostic #domain = java.lang: java.lang:name=Code Cache,type=MemoryPool ... #domain = java.util.logging: java.util.logging:type=Logging #domain = org.hornetq: org.hornetq:address="hornetq.notifications",module=Core,name="notif.fb6d27d5-02ac-11e0-ae9b-1cc1de6fb76e",type=Queue ... org.hornetq:module=JMS,name="TEST.IN",type=Queue org.hornetq:module=JMS,type=Server
To see the available attributes and operations on a specific MBean you can use the info command:
$>info -b org.hornetq:module=JMS,name="TEST.IN",type=Queue #mbean = org.hornetq:module=JMS,name="TEST.IN",type=Queue #class name = org.hornetq.jms.management.impl.JMSQueueControlImpl # attributes %0 - Address (java.lang.String, r) %1 - ConsumerCount (int, r) %2 - DeadLetterAddress (java.lang.String, rw) %3 - DeliveringCount (int, r) %4 - ExpiryAddress (java.lang.String, rw) %5 - JNDIBindings ([Ljava.lang.String;, r) %6 - MessageCount (int, r) ... %12 - Temporary (boolean, r) # operations %0 - void addJNDI(java.lang.String jndiBinding) %1 - boolean changeMessagePriority(java.lang.String messageID,int newPriority) %2 - int changeMessagesPriority(java.lang.String filter,int newPriority) %3 - int countMessages(java.lang.String filter) ... %20 - int sendMessagesToDeadLetterAddress(java.lang.String filter) #there's no notifications
To get the actual value of an MBean attribute, use the get command
$>get -b org.hornetq:module=JMS,name="TEST.IN",type=Queue MessageCount
#mbean = org.hornetq:module=JMS,name="TEST.IN",type=Queue:
MessageCount = 0;
To invoke an operation use the run command:
$>run -b org.hornetq:module=JMS,name="TEST.IN",type=Queue resetMessageCounter
#calling operation resetMessageCounter of mbean org.hornetq:module=JMS,name="TEST.IN",type=Queue
#operation returns:
null

$>get -b org.hornetq:module=JMS,name="TEST.IN",type=Queue MessageCount
#mbean = org.hornetq:module=JMS,name="TEST.IN",type=Queue:
MessageCount = 0;
As you can see the tool is very easy to use. BUT, since we are talking about Java Management Extensions which let you control, in this case, a HornetQ server, I think some security is in order.
The JMX agent on the JVM is by default started with TLS/SSL and username + password authentication. More information on how to set this up is on: http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html#auth. In the example above, security was switched off, which is easy to do by setting the following properties:
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.ssl=false

Now to enable TLS/SSL and username + password on the JMX agent set the following properties, first 2 were set before as well:
com.sun.management.jmxremote com.sun.management.jmxremote.port=10999 com.sun.management.jmxremote.password.file=$HOME/jmxremote.password javax.net.ssl.keyStore=hornetq-srv1-keystore.jks javax.net.ssl.keyStorePassword=changeit
The jmxremote.password is following the template under java.home/jre/lib/management/jmxremote.password.template

On the client side, in this case jmxterm, you need to set the following properties:
javax.net.ssl.trustStore=jmxclient-truststore.jks
javax.net.ssl.trustStorePassword=changeit
This is all following the java specs and should work, but when we start with the following command
[jeroen@homesrv~]# java -Djavax.net.ssl.trustStore=jmxclient-truststore.jks -Djavax.net.ssl.trustStorePassword=changeit -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://192.168.1.3:10999/jmxrmi
The following error pops up:
constituent[0]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/jmxterm.jar
constituent[1]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-beanutils.jar
constituent[2]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-cli.jar
constituent[3]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-collections.jar
constituent[4]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-lang.jar
constituent[5]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-logging.jar
constituent[6]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-io.jar
constituent[7]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/jcli.jar
constituent[8]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/jline.jar
---------------------------------------------------
java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLKeyException: RSA premaster secret error
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at org.cyclopsgroup.jmxterm.cc.SessionImpl.doConnect(SessionImpl.java:85)
at org.cyclopsgroup.jmxterm.cc.SessionImpl.connect(SessionImpl.java:49)
at org.cyclopsgroup.jmxterm.cc.CommandCenter.connect(CommandCenter.java:110)
at org.cyclopsgroup.jmxterm.boot.CliMain.execute(CliMain.java:139)
at org.cyclopsgroup.jmxterm.boot.CliMain.main(CliMain.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.Launcher.launchStandard(Launcher.java:353)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:264)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.uberjar.boot.Bootstrapper.bootstrap(Bootstrapper.java:209)
at org.codehaus.classworlds.uberjar.boot.Bootstrapper.main(Bootstrapper.java:116)
Caused by: javax.net.ssl.SSLKeyException: RSA premaster secret error
at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.(RSAClientKeyExchange.java:97)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:673)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:230)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:465)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
... 25 more
Caused by: java.security.NoSuchAlgorithmException: SunTlsRsaPremasterSecret KeyGenerator not available
at javax.crypto.KeyGenerator.(DashoA13*..)
at javax.crypto.KeyGenerator.getInstance(DashoA13*..)
at com.sun.net.ssl.internal.ssl.JsseJce.getKeyGenerator(JsseJce.java:223)
at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.(RSAClientKeyExchange.java:89)
... 37 more

It looks like the uberjar is doing something fishy with the classloaders which make that the standard sun security provider is no longer available. Bummer ...

This is the only explanation I can think of. Because when connecting with jconsole, like below, I only need to enter the username/pwd. No other issues.
[jeroen@homesrv~]# jconsole -J-Djavax.net.ssl.trustStore=jmxclient-truststore.jks -J-Djavax.net.ssl.trustStorePassword=changeit service:jmx:rmi:///jndi/rmi://192.168.1.3:10995/jmxrmi
I guess for command line jmx with SSL I'll have to look elsewhere ...
If someone has ideas please post :-)