Grouping Java skills by categories

I would like to group skill items from typical Java developer resumes – aka CVs – in a somewhat standardized way. See this jsfiddle for some work-in-progress.

I came up with the primary and secondary categories listed below, each with some sample items.

Your feedback is highly appreciated. Please comment on this blog post.

  • Java features and APIs:
    • Configurable JVM feature (Garbage collection, Remote debugging, JMX, etc)
    • Java language feature (Generics, Threads, Annotations, Enums, Lambdas, …)
    • Java SE API (Collections, Date/Time, NIO, java.util.concurrent, …)
    • Java EE API (EJB, JMS, JPA, JSF, JSP, …)
    • Java SE UI (Swing, JavaFX, WebStart, Applets, …)
    • 3rd party Open Source web frameworks (Struts, Wicket, Spring MVC, …)
    • Other 3rd party Open Source framework (Spring, Hibernate, …)
    • Closed-source vendor technology (Java based but no JSR)
  • Java Tools/Servers:
    • Application Servers (Websphere, Weblogic, JBoss, Glassfish, Tomcat, Jetty, …),
    • Developer Tools (IntelliJ, Eclipse, Netbeans, …)
    • Build automation tools (Maven, Ant, Jenkins, Teamcity, Bamboo, …)
    • Profiling and test tools (JVisualVM, SoapUI, JMeter, …)
  • Methodologies (software engineering approaches and practices):
    • Project level: Agile, Waterfall, RUP, …
    • Engineering level: Dependency Injection (DI), Object-Oriented Development (OOP), Continuous Integration (CI), …
  • Relevant non-Java technologies:
    • Web technologies (HTML, CSS, JQuery, JavaScript, SVG, …)
    • Scripting languages (bash, Perl, ksh, Groovy, ruby, etc)
    • Operating systems (Linux, MacOS, Solaris, …)
    • Database servers (Oracle, MS SQL server, Sybase, …)
    • Team tools (Confluence, JIRA, Crucible, …)
    • Other programming languages (C, C++, Cobol, Fortran, Scala, …)

Java Value Types and Generics specialization

It is exciting to see that Brian Goetz and others are working towards Generics over Primitive Types (List<int> et al) and Value Types (structured immutable data types with value semantics and without object overhead) for Java 10.

The motivation for these initiatives comes from the somewhat clunky divide between objects and primitives in Java.

Objects are the only choice in Java when it comes to collections content and generic type instantiation in general. There is no such thing as List<int> in Java right now.

Objects are also the only option for structured data in Java, including simple aggregation. There are no structs, no tuples, etc. And because the primitive types in Java cannot be “primitively” extended, combined or range restricted, even things like complex numbers, unsigned ints, big decimals and the like all require classes and objects to represent the values.

Arrays are the only way to aggregate a bunch of primitives (of the same type) in Java. But Arrays themselves are objects, with reference identity not value identity, and cannot be used as a substitutes for tuples.

Object purists might argue that Java should just get rid of primitives altogether and strictly follow the “everything is an object” paradigm.

But objects bear management overheads in terms of memory and processing time, bring complexity around identity versus equality, require reference checking for garbage collection purposes, cannot live on the stack, lead to array data in non-contiguous memory, etc.

As the Java architects point out, there currently is an unsolvable conflict between object identity and value semantics in Java: Even if objects a1 and a2 of immutable type A have completely identical state, the JVM cannot treat them as identical “values”, because synchronization relies on monitor object identity, and A1 and A2 would represent separate independent locks. Conflating them into one “value object” would break that distinction.

All this suggests that Java needs to be extended to support structured types with value semantics, conceptually somewhere between primitives and objects. Hopefully Java 10 will come with such types and hopefully smart engineers like Brian Goetz will make sure the resulting type system won’t feel like an incoherent Frankenstein monster.

JavaOne 2014 keynote ends awkwardly

Today we watched the JavaOne Technical Keynote at my workplace and the content was fine but a little stale, mostly rehashing last year’s messages, some might even say JavaOne 2014 was “off to a timid start“.

The technical keynote was the final part of a day of keynotes on Sep 28th. Mark Reinhold (Java Platform Architect) looked back at the history of Java, talked about Java 8 and briefly about Project Jigsaw (modules for Java). Then Brian Goetz (Java Language Architect) was finally starting to talk about exciting new initiatives for Java 10 and beyond.

At that point the keynote ended almost rudely, as Reinhold first told Goetz to speed things up, just to then cut him off a few minutes later, taking the slide clickers from him, quickly skipping through remaining slides in awkward silence and ending it with a curt “Have a good week” to the audience.

How unprofessional! Reinhold could have at least smoothed it over by saying that Goetz was trying to explain cool stuff like Value types and refer to a JavaOne session that will go into the details.

Or allow for 10 extra minutes, but maybe Oracle management was too inflexible for that. What a shame and how ironic that the longer term future of Java technology was not important enough. What was the motto of the conference again? “Create the future”? But don’t talk about it for too long, I guess.

Start script for stand-alone Java process

Directory structure

  • lib
    • some.jar
    • another.jar
  • config
    • dev
    • uat
  • start.sh
  • .setenv.sh

.setenv.sh

#!/bin/bash 

# environment (typically 'dev', 'uat' or 'prod') 
env=

# absolute path to java home ( >= 1.7 ) 
# such that $java_home/bin/java exists 
java_home=${JAVA_HOME} 

# add any additional environment setup here:

start.sh

#!/bin/bash 

# script should exit if any command fails 
set -e 

function envfail() { 
  echo $1; 
  echo "Please adjust $setenv_script and retry ..." 
  exit 1 
} 
function check_not_empty() { 
  if [ -z "$1" ]; then 
    envfail "Variable '$2' is unset or empty." 
  fi 
} 

script=$(readlink -f "$0") 
if [[ "$script" =~ .*[[:space:]].* ]]; then 
  echo "Script path contains spaces: $script" 
  echo "Please fix and retry ..." 
  exit 1 
fi 

# ok, now we know $script contains no spaces so 
# we don't need quoting acrobatics from here on 

dir=$(dirname $script) 
parent=$(dirname $dir) 
setenv_script=$parent/setenv.sh 

if [ ! -e $setenv_script ]; then 
  echo "$setenv_script not found" 
  cp $dir/.setenv.sh $setenv_script 
  envfail "Created a default $setenv_script for you." 
fi 
setenv_script_file=$(readlink -f $setenv_script) 
if [ -r $setenv_script_file ]; then 
  echo "Sourcing $setenv_script" 
  source $setenv_script 
else 
  envfail "Cannot read $setenv_script" 
fi 

check_not_empty "$env" 'env' 
config_env=$(readlink -f "$dir/config/$env") 
if [[ ! -r "$config_env" || ! -d "$config_env" ]]; then 
  envfail "Invalid env=$env : Cannot read directory $config_env" 
fi 

check_not_empty "$java_home" 'java_home' 
java="$java_home/bin/java" 
if [ ! -e "$java" ]; then 
  envfail "$java is not an executable file" 
fi 
java_version=$("$java" -version 2>&1 \
               | head -n 1 | cut -d'"' -f2 | cut -d'.' -f2) 
if [ $java_version -lt "7" ]; then 
  envfail "Java 1.7 or higher is required." 
fi 

# turn on bash debug output for the following lines 
set -x 

cd $dir 
mkdir -p "../log" 

nohup \ 
"$java" -cp "$config_env:lib/*" \
        net.doepner.example.Main \ 
        > "../log/example_stdout.log" 2>&1 & 

Jenkins Maven builds on OpenShift

Short version: If you want proper Maven builds with Jenkins on OpenShift, please vote for change request JENKINS-19844.

Full story:

Today I installed Jenkins on my OpenShift account to use it as Maven release build server for some of my Java based github projects. I ran into various obstacles and partially misleading information.

Installing the Jenkins “cartridge” on the OpenShift web console was the easiest part.

Then I logged into my new Jenkins using the auto-generated “admin” login. I created a “New Item” to “Build a maven2/3 project”, i.e. a new Maven build job, and configured it: Selected “Git” SCM and pasted the github URL of the project I want to build.

At first all “Build Now” attempts failed silently, until I realized I had to go into “Manage Jenkins” – “Configure System” page to change the “# of executors” from 0 to 1.

Next thing was that the Maven installation was not found. I set up ssh access to my OpenShift Jenkins (paste contents of ~/.ssh/id_rsa.pub from my Linux laptop into web console, then find the ssh hostname to connect) and ran a “find -name mvn /usr” on the host which located a Maven installation at /usr/share/java/apache-maven-3.0.4. I entered this in the “Maven installation” section on the Jenkins “Configure System” page.

Now I got at least some “Console output” when I clicked “Build Now” and navigated to the page of that build. The next error, however, has so far been a blocker for me. It is described here and seems to be a limitation of the Maven agent binding address in Jenkins.

I found several blogs recommending the “free-style” Jenkins job type as a workaround, instead of “maven2/3 project”. But that has many limitations and is not an acceptable solution for me.

Finally I noticed that the issue has already been reported in 2013 as JENKINS-19844 “Maven agent socket bind too inflexible (allow Jenkins in virtualized environment)”, but was closed by mistake due to a mix-up of JIRA issue numbers (19844 vs 19884).

I used my account at jenkins-ci.org and reopened the Jenkins issue. Now I can only hope that someone from Jenkins committers team will care enough about this and apply the suggested code changes. Then we have to wait until OpenShift provides a Jenkins version that contains the fix.

Additional Note: I also read about other issues with Maven on OpenShift, e.g. Jenkins having no write access to ~/.m2/repository. I could not verify those problems but they seem to be fixable in ~/.m2/settings.xml, using $OPENSHIFT_DATA_DIR. Via ssh, I was able to create and edit ~/.m2/settings.xml.

Looking for a Senior Java developer position?

At my workplace we are currently looking for a Senior Java developer with potential to grow into a Team Lead role.

The work is server-side Java development of event-driven reference systems for a large client in the investment banking domain. Our Java team enhances existing systems, adds major new feature sets and builds system integration components like trade event feeds. Production support is currently not part of our responsibilities.

The team is in Halifax, collocated with Project Management, a QA team of dedicated application testers and a UI side team of DotNet developers. Our methodology is a lightweight, flexible waterfall process, with us participating in analysis, providing design, construction and testing.

As a trusted partner in the delivery of larger pieces of work, we work closely with our customer’s Business Analysts and Technical Leads on a daily basis. We follow a quality oriented approach with mandatory code reviews, continuous integration and extensive QA testing cycles.

The technologies we use are core Java 7, XML / JSON messaging via Tibco RV, JMS and web services, Apache Camel, persistence in Oracle via JDBC or JPA, Maven builds on Jenkins build servers, SVN for version control (maybe Git in the future) and Intellij as our Java IDE of choice. Other standard libs include JUnit, Mockito and Slf4J/Logback.

We do not use any JEE or Java web containers for most of our processes. The Spring framework is used in existing applications, but usually not for new light-weight processes. Due to this Java centric, container-less nature, our developers have to be well-versed in all levels of Java SE, remote debugging, JDBC, Oracle SQL and working on the command-line.

The deployment environments are Linux boxes, so Unix and bash scripting skills are a plus. We also use Cygwin on our Windows VMs.

If you are interested, please email [oliver].[doepner] at [cgi].[com] – remove the ‘[‘ ‘]’ brackets and turn into valid email format.

Simple XML processing in Java using XPath

Often I just want to get or set a few values from/on a given XML document. XPath is the standard for specifying locations in an XML document, with a Java XPath API since Java 5.

But the API is a little clunky, resulting in code like this:

DocumentBuilder builder = 
    DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document document = builder.parse(new File("/widgets.xml"));

XPath xpath = XPathFactory.newInstance().newXPath();
String expression = "/widgets/widget";
Node widgetNode = (Node) xpath.evaluate(expression, 
    document, XPathConstants.NODE); 

Wouldn’t it be nice if it was as simple as this:

Xml xml = new Xml(Paths.get("/widgets.xml"));
Xml node = xml.node("/widgets/widget");

This Xml class and some supporting code is in my x-xml github repo. It implements a Tree interface like this:

Interface

package net.doepner.xml;

/**
 * Conveniently extract and change text values
 * and navigate nodes on an XML document
 */
public interface Tree {

    /**
     * @param xpath An XPath expression
     * @return The nodes matching the XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    Iterable<? extends Tree> nodes(String xpath);

    /**
     * @param xpath An XPath expression
     * @return The single node matching the XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    Tree node(String xpath);

    /**
     * @param xpath An XPath expression
     * @return The text value from the unique XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    String get(String xpath);

    /**
     * @param xpath An XPath expression
     * @param value The text value to be set on the unique XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    void set(String xpath, String value);
}

Implementation

Here is the Xml class that implements the Tree interface:

package net.doepner.xml;

import com.doepner.libs.messaging.XmlException;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;
import java.nio.file.Path;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;

/**
 * Allows node navigation and getting and setting of xpath values
 */
public class Xml implements Tree {

    private final TransformerFactory tf = TransformerFactory.newInstance();
    private final XPath xp = XPathFactory.newInstance().newXPath();
    private final QNames qNames = new QNames();

    private final Node root;

    public Xml(Path path) {
        this(new InputSource(path.toFile().toURI().toASCIIString()));
    }

    public Xml(String payload) {
        this(new InputSource(new StringReader(payload)));
    }

    public Xml(InputSource inputSource) {
        this(parse(inputSource));
    }

    private Xml(Node root) {
        this.root = root;
    }

    private static Document parse(InputSource inputSource) {
        try {
            return DocumentBuilderFactory.newInstance()
                    .newDocumentBuilder().parse(inputSource);
        } catch (ParserConfigurationException 
                | SAXException | IOException e) {
            throw new XmlException(e);
        }
    }

    @Override
    public Iterable<Xml> nodes(String xpath) {
        final Collection<Xml> result = new LinkedList<>();
        for (Node node : new IterableNodes(eval(xpath, NodeList.class))) {
            result.add(new Xml(node));
        }
        return result;
    }

    @Override
    public Xml node(String xpath) {
        return new Xml(eval(xpath, Node.class));
    }

    @Override
    public String get(String xpath) {
        return eval(xpath, String.class);
    }

    @Override
    public void set(String xpath, String value) {
        eval(xpath, Node.class).setTextContent(value);
    }

    private final Map<String, XPathExpression> cache = new HashMap<>();

    private <T> T eval(String xpath, Class<T> resultType) {
        return resultType.cast(evaluate(resultType, getCompiled(xpath)));
    }

    private XPathExpression getCompiled(String xpath) {
        final XPathExpression cachedExpression = cache.get(xpath);
        if (cachedExpression != null) {
            return cachedExpression;
        } else {
            final XPathExpression expr = compile(xpath);
            cache.put(xpath, expr);
            return expr;
        }
    }

    private Object evaluate(Class<?> resultType, XPathExpression expr) {
        try {
            return expr.evaluate(root, qNames.get(resultType));
        } catch (XPathExpressionException e) {
            throw new XmlException(e);
        }
    }

    private XPathExpression compile(String xpath) {
        try {
            return xp.compile(xpath);
        } catch (XPathExpressionException e) {
            throw new XmlException(e);
        }
    }

    @Override
    public String toString() {
        try (final StringWriter writer = new StringWriter()) {
            tf.newTransformer().transform(new DOMSource(root),
                    new StreamResult(writer));
            return writer.getBuffer().toString();
        } catch (TransformerException | IOException e) {
            throw new XmlException(e);
        }
    }
}

Supporting classes

Custom exception wrapper class
package net.doepner.xml;
 
/**
 * An XML exception
 */
public class XmlException extends RuntimeException {
 
    public XmlException(Throwable cause) {
        super(cause);
    }
}
The mapping from Java types to QNames
package net.doepner.xml;

import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import javax.xml.namespace.QName;
import java.util.HashMap;
import java.util.Map;

import static javax.xml.xpath.XPathConstants.NODE;
import static javax.xml.xpath.XPathConstants.NODESET;
import static javax.xml.xpath.XPathConstants.STRING;

/**
 * Mapping from Java type to QName
 */
public class QNames {

    private final Map<Class<?>, QName> qNames = new HashMap<>();

    {
        qNames.put(String.class, STRING);
        qNames.put(Node.class, NODE);
        qNames.put(NodeList.class, NODESET);
    }

    public QName get(Class<?> resultType) {
        return qNames.get(resultType);
    }
}
Convenient iteration of XML node lists
package net.doepner.xml;

import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import java.util.Iterator;
import java.util.NoSuchElementException;

/**
 * Convenient iteration of XML node lists
 */
public class IterableNodes implements Iterable<Node> {

    private final NodeList nodeList;

    public IterableNodes(NodeList nodeList) {
        this.nodeList = nodeList;
    }

    @Override
    public Iterator<Node> iterator() {

        return new Iterator<Node>() {

            int index = 0;

            @Override
            public boolean hasNext() {
                return index < nodeList.getLength();
            }

            @Override
            public Node next() {
                if (hasNext()) {
                    return nodeList.item(index++);
                } else {
                    throw new NoSuchElementException();
                }
            }

            @Override
            public void remove() {
                throw new UnsupportedOperationException();
            }
        };
    }
}

Get java version string via shell commands

Determine the pure java version string from any Unix/Linux shell (including Cygwin):

java -version 2>&1 | head -n 1 | cut -d'"' -f2

This requires only the very commonly available and lightweight “head” and “cut” commands.

I originally found the one-liner on stackoverflow. Thanks to the friendly folks who shared it.

To get only the major version part (e.g. 8 for Java 1.8.x, 11 for 11.x), use this:

java -version 2>&1 \
  | head -1 \
  | cut -d'"' -f2 \
  | sed 's/^1\.//' \
  | cut -d'.' -f1

Note: The sed step is required for versions up to Java 8 that start with the “1.” prefix.

Example: Ensure Java 11 or higher:

#!/bin/bash

version=$(java -version 2>&1 \
  | head -1 \
  | cut -d'"' -f2 \
  | sed 's/^1\.//' \
  | cut -d'.' -f1 
)

if [ $version -lt "11" ]; then
  echo "Java 11 or higher is required."
  exit 1
fi

Reconfigure log4j 1.x without restart

Disclaimer: This blog post is about log4j 1.x. which is by now a dormant project and not recommended for new projects. Logback and log4j 2.x support dynamic configuration changes at runtime via a simple attribute in their respective config files (see related log4j 2.x and logback docs). Please consider upgrading your logging backend to one of those libraries and use them in combination with the slf4j logging facade API.

Support for dynamic logging config changes at runtime is useful when you want to debug an application without restarting it: You can change settings in your logging config file, like log levels, appenders, log output formats, etc. and the logging framework automatically notices the change and reconfigures itself after a certain delay.

Good old log4j 1.x supports this, but you have to programatically set it up, typically somewhere in the start-up code of your Java application, using

  • DOMConfigurator.configureAndWatch(..) if you use log4j.xml
  • PropertyConfigurator.configureAndWatch(..) if you use log4j.properties

Please note that the log4j 1.x FAQ does not recommend doing this in a JEE container:

“Because the configureAndWatch launches a separate watchdog thread, and because there is no way to stop this thread in log4j 1.2, the configureAndWatch method is unsafe for use in J2EE environments where applications are recycled.”

The following example code works both for log4j.xml or log4j.properties files, as long as they are in the classpath as file resources, i.e. not packaged inside a jar:

    public static enum ConfigFileType {
         xml, properties
    }

    public void configureLogging(ConfigFileType configFileType) {
        final URL configFile = getClass().getResource("/log4j." + configFileType);

        if (configFile != null && "file".equals(configFile.getProtocol())) {
            final int scanDelayMinutes = getSystemProperty("log4j.config.rescan.delay.minutes", 2);
            final long scanDelayMillis = scanDelayMinutes * 60 * 1000;

            switch (configFileType) {
                case xml:
                    DOMConfigurator.configureAndWatch(configFile.getPath(), scanDelayMillis);
                    break;
                case properties:
                    PropertyConfigurator.configureAndWatch(configFile.getPath(), scanDelayMillis);
                    break;
            }
        }
    }

    private static int getSystemProperty(String name, int defaultValue) {
        try {
            return Integer.parseInt(System.getProperty(name));
        } catch (NumberFormatException e) {
            return defaultValue;
        }
    }

You can override the default rescan delay as a system property option in your java command, for example:

java -Dlog4j.config.rescan.delay.minutes=5 [..]

The period is in minutes to prevent too much rescanning. Log4j uses a “FileWatchdog” class that repeatedly checks File#lastModified(). On some filesystems – like ReiserFS on Linux – scan delays of less than a minute can cause problems (thanks to Rod Oliveira for pointing this out).

Java software engineering – reference resources

Official Java and JEE

Java Technology Reference

Java Standard Edition (JSE)

Java Enterprise Edition (JEE)

The official Java tutorials

The official JEE 7 tutorial

JEE 7 Technologies index

Java language spec and JVM spec

Java community

Oracle Java community

OpenJDK

Java Community Process (JCP)

Apache Commons

Apache.org Java projects

JBoss.org

Spring

Google Guava

Trending Java projects on github

JEE and Java web servers

Apache Tomcat

JBoss Wildfly

Glassfish

Build and test automation

Sonatype Maven books

Jenkins documentation (wiki)

JUnit reference documentation

Source and version control

The SVN reference book

Git reference documentation

Java IDEs

Intellij IDEA documentation

Eclipse documentation

Netbeans knowledge base

Vim configuration for Java coding