Start script for stand-alone Java process

Directory structure

  • lib
    • some.jar
    • another.jar
  • config
    • dev
    • uat
  • start.sh
  • .setenv.sh

.setenv.sh

#!/bin/bash 

# environment (typically 'dev', 'uat' or 'prod') 
env=

# absolute path to java home ( >= 1.7 ) 
# such that $java_home/bin/java exists 
java_home=${JAVA_HOME} 

# add any additional environment setup here:

start.sh

#!/bin/bash 

# script should exit if any command fails 
set -e 

function envfail() { 
  echo $1; 
  echo "Please adjust $setenv_script and retry ..." 
  exit 1 
} 
function check_not_empty() { 
  if [ -z "$1" ]; then 
    envfail "Variable '$2' is unset or empty." 
  fi 
} 

script=$(readlink -f "$0") 
if [[ "$script" =~ .*[[:space:]].* ]]; then 
  echo "Script path contains spaces: $script" 
  echo "Please fix and retry ..." 
  exit 1 
fi 

# ok, now we know $script contains no spaces so 
# we don't need quoting acrobatics from here on 

dir=$(dirname $script) 
parent=$(dirname $dir) 
setenv_script=$parent/setenv.sh 

if [ ! -e $setenv_script ]; then 
  echo "$setenv_script not found" 
  cp $dir/.setenv.sh $setenv_script 
  envfail "Created a default $setenv_script for you." 
fi 
setenv_script_file=$(readlink -f $setenv_script) 
if [ -r $setenv_script_file ]; then 
  echo "Sourcing $setenv_script" 
  source $setenv_script 
else 
  envfail "Cannot read $setenv_script" 
fi 

check_not_empty "$env" 'env' 
config_env=$(readlink -f "$dir/config/$env") 
if [[ ! -r "$config_env" || ! -d "$config_env" ]]; then 
  envfail "Invalid env=$env : Cannot read directory $config_env" 
fi 

check_not_empty "$java_home" 'java_home' 
java="$java_home/bin/java" 
if [ ! -e "$java" ]; then 
  envfail "$java is not an executable file" 
fi 
java_version=$("$java" -version 2>&1 \
               | head -n 1 | cut -d'"' -f2 | cut -d'.' -f2) 
if [ $java_version -lt "7" ]; then 
  envfail "Java 1.7 or higher is required." 
fi 

# turn on bash debug output for the following lines 
set -x 

cd $dir 
mkdir -p "../log" 

nohup \ 
"$java" -cp "$config_env:lib/*" \
        net.doepner.example.Main \ 
        > "../log/example_stdout.log" 2>&1 & 

Jenkins Maven builds on OpenShift

Short version: If you want proper Maven builds with Jenkins on OpenShift, please vote for change request JENKINS-19844.

Full story:

Today I installed Jenkins on my OpenShift account to use it as Maven release build server for some of my Java based github projects. I ran into various obstacles and partially misleading information.

Installing the Jenkins “cartridge” on the OpenShift web console was the easiest part.

Then I logged into my new Jenkins using the auto-generated “admin” login. I created a “New Item” to “Build a maven2/3 project”, i.e. a new Maven build job, and configured it: Selected “Git” SCM and pasted the github URL of the project I want to build.

At first all “Build Now” attempts failed silently, until I realized I had to go into “Manage Jenkins” – “Configure System” page to change the “# of executors” from 0 to 1.

Next thing was that the Maven installation was not found. I set up ssh access to my OpenShift Jenkins (paste contents of ~/.ssh/id_rsa.pub from my Linux laptop into web console, then find the ssh hostname to connect) and ran a “find -name mvn /usr” on the host which located a Maven installation at /usr/share/java/apache-maven-3.0.4. I entered this in the “Maven installation” section on the Jenkins “Configure System” page.

Now I got at least some “Console output” when I clicked “Build Now” and navigated to the page of that build. The next error, however, has so far been a blocker for me. It is described here and seems to be a limitation of the Maven agent binding address in Jenkins.

I found several blogs recommending the “free-style” Jenkins job type as a workaround, instead of “maven2/3 project”. But that has many limitations and is not an acceptable solution for me.

Finally I noticed that the issue has already been reported in 2013 as JENKINS-19844 “Maven agent socket bind too inflexible (allow Jenkins in virtualized environment)”, but was closed by mistake due to a mix-up of JIRA issue numbers (19844 vs 19884).

I used my account at jenkins-ci.org and reopened the Jenkins issue. Now I can only hope that someone from Jenkins committers team will care enough about this and apply the suggested code changes. Then we have to wait until OpenShift provides a Jenkins version that contains the fix.

Additional Note: I also read about other issues with Maven on OpenShift, e.g. Jenkins having no write access to ~/.m2/repository. I could not verify those problems but they seem to be fixable in ~/.m2/settings.xml, using $OPENSHIFT_DATA_DIR. Via ssh, I was able to create and edit ~/.m2/settings.xml.

Looking for a Senior Java developer position?

At my workplace we are currently looking for a Senior Java developer with potential to grow into a Team Lead role.

The work is server-side Java development of event-driven reference systems for a large client in the investment banking domain. Our Java team enhances existing systems, adds major new feature sets and builds system integration components like trade event feeds. Production support is currently not part of our responsibilities.

The team is in Halifax, collocated with Project Management, a QA team of dedicated application testers and a UI side team of DotNet developers. Our methodology is a lightweight, flexible waterfall process, with us participating in analysis, providing design, construction and testing.

As a trusted partner in the delivery of larger pieces of work, we work closely with our customer’s Business Analysts and Technical Leads on a daily basis. We follow a quality oriented approach with mandatory code reviews, continuous integration and extensive QA testing cycles.

The technologies we use are core Java 7, XML / JSON messaging via Tibco RV, JMS and web services, Apache Camel, persistence in Oracle via JDBC or JPA, Maven builds on Jenkins build servers, SVN for version control (maybe Git in the future) and Intellij as our Java IDE of choice. Other standard libs include JUnit, Mockito and Slf4J/Logback.

We do not use any JEE or Java web containers for most of our processes. The Spring framework is used in existing applications, but usually not for new light-weight processes. Due to this Java centric, container-less nature, our developers have to be well-versed in all levels of Java SE, remote debugging, JDBC, Oracle SQL and working on the command-line.

The deployment environments are Linux boxes, so Unix and bash scripting skills are a plus. We also use Cygwin on our Windows VMs.

If you are interested, please email [oliver].[doepner] at [cgi].[com] – remove the ‘[‘ ‘]’ brackets and turn into valid email format.

Simple XML processing in Java using XPath

Often I just want to get or set a few values from/on a given XML document. XPath is the standard for specifying locations in an XML document, with a Java XPath API since Java 5.

But the API is a little clunky, resulting in code like this:

DocumentBuilder builder = 
    DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document document = builder.parse(new File("/widgets.xml"));

XPath xpath = XPathFactory.newInstance().newXPath();
String expression = "/widgets/widget";
Node widgetNode = (Node) xpath.evaluate(expression, 
    document, XPathConstants.NODE); 

Wouldn’t it be nice if it was as simple as this:

Xml xml = new Xml(Paths.get("/widgets.xml"));
Xml node = xml.node("/widgets/widget");

This Xml class and some supporting code is in my x-xml github repo. It implements a Tree interface like this:

Interface

package net.doepner.xml;

/**
 * Conveniently extract and change text values
 * and navigate nodes on an XML document
 */
public interface Tree {

    /**
     * @param xpath An XPath expression
     * @return The nodes matching the XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    Iterable<? extends Tree> nodes(String xpath);

    /**
     * @param xpath An XPath expression
     * @return The single node matching the XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    Tree node(String xpath);

    /**
     * @param xpath An XPath expression
     * @return The text value from the unique XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    String get(String xpath);

    /**
     * @param xpath An XPath expression
     * @param value The text value to be set on the unique XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    void set(String xpath, String value);
}

Implementation

Here is the Xml class that implements the Tree interface:

package net.doepner.xml;

import com.doepner.libs.messaging.XmlException;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;
import java.nio.file.Path;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;

/**
 * Allows node navigation and getting and setting of xpath values
 */
public class Xml implements Tree {

    private final TransformerFactory tf = TransformerFactory.newInstance();
    private final XPath xp = XPathFactory.newInstance().newXPath();
    private final QNames qNames = new QNames();

    private final Node root;

    public Xml(Path path) {
        this(new InputSource(path.toFile().toURI().toASCIIString()));
    }

    public Xml(String payload) {
        this(new InputSource(new StringReader(payload)));
    }

    public Xml(InputSource inputSource) {
        this(parse(inputSource));
    }

    private Xml(Node root) {
        this.root = root;
    }

    private static Document parse(InputSource inputSource) {
        try {
            return DocumentBuilderFactory.newInstance()
                    .newDocumentBuilder().parse(inputSource);
        } catch (ParserConfigurationException 
                | SAXException | IOException e) {
            throw new XmlException(e);
        }
    }

    @Override
    public Iterable<Xml> nodes(String xpath) {
        final Collection<Xml> result = new LinkedList<>();
        for (Node node : new IterableNodes(eval(xpath, NodeList.class))) {
            result.add(new Xml(node));
        }
        return result;
    }

    @Override
    public Xml node(String xpath) {
        return new Xml(eval(xpath, Node.class));
    }

    @Override
    public String get(String xpath) {
        return eval(xpath, String.class);
    }

    @Override
    public void set(String xpath, String value) {
        eval(xpath, Node.class).setTextContent(value);
    }

    private final Map<String, XPathExpression> cache = new HashMap<>();

    private <T> T eval(String xpath, Class<T> resultType) {
        return resultType.cast(evaluate(resultType, getCompiled(xpath)));
    }

    private XPathExpression getCompiled(String xpath) {
        final XPathExpression cachedExpression = cache.get(xpath);
        if (cachedExpression != null) {
            return cachedExpression;
        } else {
            final XPathExpression expr = compile(xpath);
            cache.put(xpath, expr);
            return expr;
        }
    }

    private Object evaluate(Class<?> resultType, XPathExpression expr) {
        try {
            return expr.evaluate(root, qNames.get(resultType));
        } catch (XPathExpressionException e) {
            throw new XmlException(e);
        }
    }

    private XPathExpression compile(String xpath) {
        try {
            return xp.compile(xpath);
        } catch (XPathExpressionException e) {
            throw new XmlException(e);
        }
    }

    @Override
    public String toString() {
        try (final StringWriter writer = new StringWriter()) {
            tf.newTransformer().transform(new DOMSource(root),
                    new StreamResult(writer));
            return writer.getBuffer().toString();
        } catch (TransformerException | IOException e) {
            throw new XmlException(e);
        }
    }
}

Supporting classes

Custom exception wrapper class
package net.doepner.xml;
 
/**
 * An XML exception
 */
public class XmlException extends RuntimeException {
 
    public XmlException(Throwable cause) {
        super(cause);
    }
}
The mapping from Java types to QNames
package net.doepner.xml;

import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import javax.xml.namespace.QName;
import java.util.HashMap;
import java.util.Map;

import static javax.xml.xpath.XPathConstants.NODE;
import static javax.xml.xpath.XPathConstants.NODESET;
import static javax.xml.xpath.XPathConstants.STRING;

/**
 * Mapping from Java type to QName
 */
public class QNames {

    private final Map<Class<?>, QName> qNames = new HashMap<>();

    {
        qNames.put(String.class, STRING);
        qNames.put(Node.class, NODE);
        qNames.put(NodeList.class, NODESET);
    }

    public QName get(Class<?> resultType) {
        return qNames.get(resultType);
    }
}
Convenient iteration of XML node lists
package net.doepner.xml;

import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import java.util.Iterator;
import java.util.NoSuchElementException;

/**
 * Convenient iteration of XML node lists
 */
public class IterableNodes implements Iterable<Node> {

    private final NodeList nodeList;

    public IterableNodes(NodeList nodeList) {
        this.nodeList = nodeList;
    }

    @Override
    public Iterator<Node> iterator() {

        return new Iterator<Node>() {

            int index = 0;

            @Override
            public boolean hasNext() {
                return index < nodeList.getLength();
            }

            @Override
            public Node next() {
                if (hasNext()) {
                    return nodeList.item(index++);
                } else {
                    throw new NoSuchElementException();
                }
            }

            @Override
            public void remove() {
                throw new UnsupportedOperationException();
            }
        };
    }
}