Java Value Types and Generics specialization

It is exciting to see that Brian Goetz and others are working towards Generics over Primitive Types (List<int> et al) and Value Types (structured immutable data types with value semantics and without object overhead) for Java 10.

The motivation for these initiatives comes from the somewhat clunky divide between objects and primitives in Java.

Objects are the only choice in Java when it comes to collections content and generic type instantiation in general. There is no such thing as List<int> in Java right now.

Objects are also the only option for structured data in Java, including simple aggregation. There are no structs, no tuples, etc. And because the primitive types in Java cannot be “primitively” extended, combined or range restricted, even things like complex numbers, unsigned ints, big decimals and the like all require classes and objects to represent the values.

Arrays are the only way to aggregate a bunch of primitives (of the same type) in Java. But Arrays themselves are objects, with reference identity not value identity, and cannot be used as a substitutes for tuples.

Object purists might argue that Java should just get rid of primitives altogether and strictly follow the “everything is an object” paradigm.

But objects bear management overheads in terms of memory and processing time, bring complexity around identity versus equality, require reference checking for garbage collection purposes, cannot live on the stack, lead to array data in non-contiguous memory, etc.

As the Java architects point out, there currently is an unsolvable conflict between object identity and value semantics in Java: Even if objects a1 and a2 of immutable type A have completely identical state, the JVM cannot treat them as identical “values”, because synchronization relies on monitor object identity, and A1 and A2 would represent separate independent locks. Conflating them into one “value object” would break that distinction.

All this suggests that Java needs to be extended to support structured types with value semantics, conceptually somewhere between primitives and objects. Hopefully Java 10 will come with such types and hopefully smart engineers like Brian Goetz will make sure the resulting type system won’t feel like an incoherent Frankenstein monster.

JavaOne 2014 keynote ends awkwardly

Today we watched the JavaOne Technical Keynote at my workplace and the content was fine but a little stale, mostly rehashing last year’s messages, some might even say JavaOne 2014 was “off to a timid start“.

The technical keynote was the final part of a day of keynotes on Sep 28th. Mark Reinhold (Java Platform Architect) looked back at the history of Java, talked about Java 8 and briefly about Project Jigsaw (modules for Java). Then Brian Goetz (Java Language Architect) was finally starting to talk about exciting new initiatives for Java 10 and beyond.

At that point the keynote ended almost rudely, as Reinhold first told Goetz to speed things up, just to then cut him off a few minutes later, taking the slide clickers from him, quickly skipping through remaining slides in awkward silence and ending it with a curt “Have a good week” to the audience.

How unprofessional! Reinhold could have at least smoothed it over by saying that Goetz was trying to explain cool stuff like Value types and refer to a JavaOne session that will go into the details.

Or allow for 10 extra minutes, but maybe Oracle management was too inflexible for that. What a shame and how ironic that the longer term future of Java technology was not important enough. What was the motto of the conference again? “Create the future”? But don’t talk about it for too long, I guess.

Simple XML processing in Java using XPath

Often I just want to get or set a few values from/on a given XML document. XPath is the standard for specifying locations in an XML document, with a Java XPath API since Java 5.

But the API is a little clunky, resulting in code like this:

DocumentBuilder builder = 
    DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document document = builder.parse(new File("/widgets.xml"));

XPath xpath = XPathFactory.newInstance().newXPath();
String expression = "/widgets/widget";
Node widgetNode = (Node) xpath.evaluate(expression, 
    document, XPathConstants.NODE); 

Wouldn’t it be nice if it was as simple as this:

Xml xml = new Xml(Paths.get("/widgets.xml"));
Xml node = xml.node("/widgets/widget");

This Xml class and some supporting code is in my x-xml github repo. It implements a Tree interface like this:

Interface

package net.doepner.xml;

/**
 * Conveniently extract and change text values
 * and navigate nodes on an XML document
 */
public interface Tree {

    /**
     * @param xpath An XPath expression
     * @return The nodes matching the XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    Iterable<? extends Tree> nodes(String xpath);

    /**
     * @param xpath An XPath expression
     * @return The single node matching the XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    Tree node(String xpath);

    /**
     * @param xpath An XPath expression
     * @return The text value from the unique XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    String get(String xpath);

    /**
     * @param xpath An XPath expression
     * @param value The text value to be set on the unique XPath location
     * @throws XmlException Wrapping any underlying XML API exception
     */
    void set(String xpath, String value);
}

Implementation

Here is the Xml class that implements the Tree interface:

package net.doepner.xml;

import com.doepner.libs.messaging.XmlException;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;
import java.nio.file.Path;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;

/**
 * Allows node navigation and getting and setting of xpath values
 */
public class Xml implements Tree {

    private final TransformerFactory tf = TransformerFactory.newInstance();
    private final XPath xp = XPathFactory.newInstance().newXPath();
    private final QNames qNames = new QNames();

    private final Node root;

    public Xml(Path path) {
        this(new InputSource(path.toFile().toURI().toASCIIString()));
    }

    public Xml(String payload) {
        this(new InputSource(new StringReader(payload)));
    }

    public Xml(InputSource inputSource) {
        this(parse(inputSource));
    }

    private Xml(Node root) {
        this.root = root;
    }

    private static Document parse(InputSource inputSource) {
        try {
            return DocumentBuilderFactory.newInstance()
                    .newDocumentBuilder().parse(inputSource);
        } catch (ParserConfigurationException 
                | SAXException | IOException e) {
            throw new XmlException(e);
        }
    }

    @Override
    public Iterable<Xml> nodes(String xpath) {
        final Collection<Xml> result = new LinkedList<>();
        for (Node node : new IterableNodes(eval(xpath, NodeList.class))) {
            result.add(new Xml(node));
        }
        return result;
    }

    @Override
    public Xml node(String xpath) {
        return new Xml(eval(xpath, Node.class));
    }

    @Override
    public String get(String xpath) {
        return eval(xpath, String.class);
    }

    @Override
    public void set(String xpath, String value) {
        eval(xpath, Node.class).setTextContent(value);
    }

    private final Map<String, XPathExpression> cache = new HashMap<>();

    private <T> T eval(String xpath, Class<T> resultType) {
        return resultType.cast(evaluate(resultType, getCompiled(xpath)));
    }

    private XPathExpression getCompiled(String xpath) {
        final XPathExpression cachedExpression = cache.get(xpath);
        if (cachedExpression != null) {
            return cachedExpression;
        } else {
            final XPathExpression expr = compile(xpath);
            cache.put(xpath, expr);
            return expr;
        }
    }

    private Object evaluate(Class<?> resultType, XPathExpression expr) {
        try {
            return expr.evaluate(root, qNames.get(resultType));
        } catch (XPathExpressionException e) {
            throw new XmlException(e);
        }
    }

    private XPathExpression compile(String xpath) {
        try {
            return xp.compile(xpath);
        } catch (XPathExpressionException e) {
            throw new XmlException(e);
        }
    }

    @Override
    public String toString() {
        try (final StringWriter writer = new StringWriter()) {
            tf.newTransformer().transform(new DOMSource(root),
                    new StreamResult(writer));
            return writer.getBuffer().toString();
        } catch (TransformerException | IOException e) {
            throw new XmlException(e);
        }
    }
}

Supporting classes

Custom exception wrapper class
package net.doepner.xml;
 
/**
 * An XML exception
 */
public class XmlException extends RuntimeException {
 
    public XmlException(Throwable cause) {
        super(cause);
    }
}
The mapping from Java types to QNames
package net.doepner.xml;

import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import javax.xml.namespace.QName;
import java.util.HashMap;
import java.util.Map;

import static javax.xml.xpath.XPathConstants.NODE;
import static javax.xml.xpath.XPathConstants.NODESET;
import static javax.xml.xpath.XPathConstants.STRING;

/**
 * Mapping from Java type to QName
 */
public class QNames {

    private final Map<Class<?>, QName> qNames = new HashMap<>();

    {
        qNames.put(String.class, STRING);
        qNames.put(Node.class, NODE);
        qNames.put(NodeList.class, NODESET);
    }

    public QName get(Class<?> resultType) {
        return qNames.get(resultType);
    }
}
Convenient iteration of XML node lists
package net.doepner.xml;

import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import java.util.Iterator;
import java.util.NoSuchElementException;

/**
 * Convenient iteration of XML node lists
 */
public class IterableNodes implements Iterable<Node> {

    private final NodeList nodeList;

    public IterableNodes(NodeList nodeList) {
        this.nodeList = nodeList;
    }

    @Override
    public Iterator<Node> iterator() {

        return new Iterator<Node>() {

            int index = 0;

            @Override
            public boolean hasNext() {
                return index < nodeList.getLength();
            }

            @Override
            public Node next() {
                if (hasNext()) {
                    return nodeList.item(index++);
                } else {
                    throw new NoSuchElementException();
                }
            }

            @Override
            public void remove() {
                throw new UnsupportedOperationException();
            }
        };
    }
}

Get java version string via shell commands

Determine the pure java version string from any Unix/Linux shell (including Cygwin):

java -version 2>&1 | head -n 1 | cut -d'"' -f2

This requires only the very commonly available and lightweight “head” and “cut” commands.

I originally found the one-liner on stackoverflow. Thanks to the friendly folks who shared it.

To get only the major version part (e.g. 8 for Java 1.8.x, 11 for 11.x), use this:

java -version 2>&1 \
  | head -1 \
  | cut -d'"' -f2 \
  | sed 's/^1\.//' \
  | cut -d'.' -f1

Note: The sed step is required for versions up to Java 8 that start with the “1.” prefix.

Example: Ensure Java 11 or higher:

#!/bin/bash

version=$(java -version 2>&1 \
  | head -1 \
  | cut -d'"' -f2 \
  | sed 's/^1\.//' \
  | cut -d'.' -f1 
)

if [ $version -lt "11" ]; then
  echo "Java 11 or higher is required."
  exit 1
fi

Reconfigure log4j 1.x without restart

Disclaimer: This blog post is about log4j 1.x. which is by now a dormant project and not recommended for new projects. Logback and log4j 2.x support dynamic configuration changes at runtime via a simple attribute in their respective config files (see related log4j 2.x and logback docs). Please consider upgrading your logging backend to one of those libraries and use them in combination with the slf4j logging facade API.

Support for dynamic logging config changes at runtime is useful when you want to debug an application without restarting it: You can change settings in your logging config file, like log levels, appenders, log output formats, etc. and the logging framework automatically notices the change and reconfigures itself after a certain delay.

Good old log4j 1.x supports this, but you have to programatically set it up, typically somewhere in the start-up code of your Java application, using

  • DOMConfigurator.configureAndWatch(..) if you use log4j.xml
  • PropertyConfigurator.configureAndWatch(..) if you use log4j.properties

Please note that the log4j 1.x FAQ does not recommend doing this in a JEE container:

“Because the configureAndWatch launches a separate watchdog thread, and because there is no way to stop this thread in log4j 1.2, the configureAndWatch method is unsafe for use in J2EE environments where applications are recycled.”

The following example code works both for log4j.xml or log4j.properties files, as long as they are in the classpath as file resources, i.e. not packaged inside a jar:

    public static enum ConfigFileType {
         xml, properties
    }

    public void configureLogging(ConfigFileType configFileType) {
        final URL configFile = getClass().getResource("/log4j." + configFileType);

        if (configFile != null && "file".equals(configFile.getProtocol())) {
            final int scanDelayMinutes = getSystemProperty("log4j.config.rescan.delay.minutes", 2);
            final long scanDelayMillis = scanDelayMinutes * 60 * 1000;

            switch (configFileType) {
                case xml:
                    DOMConfigurator.configureAndWatch(configFile.getPath(), scanDelayMillis);
                    break;
                case properties:
                    PropertyConfigurator.configureAndWatch(configFile.getPath(), scanDelayMillis);
                    break;
            }
        }
    }

    private static int getSystemProperty(String name, int defaultValue) {
        try {
            return Integer.parseInt(System.getProperty(name));
        } catch (NumberFormatException e) {
            return defaultValue;
        }
    }

You can override the default rescan delay as a system property option in your java command, for example:

java -Dlog4j.config.rescan.delay.minutes=5 [..]

The period is in minutes to prevent too much rescanning. Log4j uses a “FileWatchdog” class that repeatedly checks File#lastModified(). On some filesystems – like ReiserFS on Linux – scan delays of less than a minute can cause problems (thanks to Rod Oliveira for pointing this out).

Java software engineering – reference resources

Official Java and JEE

Java Technology Reference

Java Standard Edition (JSE)

Java Enterprise Edition (JEE)

The official Java tutorials

The official JEE 7 tutorial

JEE 7 Technologies index

Java language spec and JVM spec

Java community

Oracle Java community

OpenJDK

Java Community Process (JCP)

Apache Commons

Apache.org Java projects

JBoss.org

Spring

Google Guava

Trending Java projects on github

JEE and Java web servers

Apache Tomcat

JBoss Wildfly

Glassfish

Build and test automation

Sonatype Maven books

Jenkins documentation (wiki)

JUnit reference documentation

Source and version control

The SVN reference book

Git reference documentation

Java IDEs

Intellij IDEA documentation

Eclipse documentation

Netbeans knowledge base

Vim configuration for Java coding

Java remote debugging JVM options

Java 5, 6, 7 and newer

The Java™ Platform Debugger Architecture (JPDA) supports certain JVM invocation options.

Usually this boils down to:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=y

This will cause your JVM process to execute but wait (suspend=y) and listen for a socket connection from a remote debugging client on a free port (which it will write to stdout. The suspend=y option is especially useful if you need to debug code that runs during the start-up phase of your application.

If you don’t want the process to do the initial waiting, then use suspend=n.

Java 1.4

If you are for some terrible reason stuck on Java 1.4, then you must use the older approach like this:

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n

Tomcat

To enabled remote debugging for a Apache Tomcat on Windows, create bin\setenv.bat in your Tomcat installation, with this content:

set "CATALINA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n"

Network port

If you use the settings above, i.e. server=y and no “address” option, JPDA will pick a free port and write a message about this to stdout. Look for something like this:

Listening for transport dt_socket at address: [PORT]

Intellij configuration

Add a debug configuration via Run menu:

Open Source games using libgdx

There are these demos.

And I found a list at the bottom of the main libgdx documentation page and used it as a startting point for the list below. I haven’t verified yet whether their respective licensing is covered by the OSI Open source definition.

I am actively maintaining this list, so feel free to send me additions or corrections:

I have also asked for a more comprehensive list on the libgdx forum and on libgdx lead developer Mario Zechner’s blog.

Create single copyright holder, GPL licensed source code repository

I sometimes want to create a source code repository and build process for a GPL licensed software project, where the sole copyright holder (“owner”) is a single legal entity, like a company or a foundation.

This is how I do it:

  • Link to the license from README.md (e.g. the Free Software Foundation’s GPL, version 2)
  • At top of each source file: Copyright notice, license statement, warranty disclaimer
  • Include a plain-text copy of the full license in the root of your distributable package(s)
  • In the README.md, indicate that the project mandates copyright assignment
  • Define the legal agreement process between code contributors and the project owner

The GPL was chosen for its strong copyleft nature and combined with the sole copyright holder approach to enable dual licensing business models.

On sites like github, the code contribution itself is usually done by pull request. The legal agreement process between code contributor and project owner should be complete before the pull request is accepted.

I am not yet sure how to best automate updating the year range in the copyright notice, probably at commit time.

Using libgdx for cross-platform app development

I am looking for a framework that allows me to develop modern apps (mobile, web, desktop) all from one Java codebase. I prefer Java because I know it very well, it is already cross-platform and a statically typed language that allows IntelliJ, Eclipse and Netbeans to be better than any dynamically typed scripting language editor could ever be.

Currently my favorite is libgdx. I am planning to use it with IntelliJ Community Edition and with Maven.

By using RoboVM, libgdx even supports iOS.

For user input (forms) libgdx provides the scene2d.ui widgets. I hope that will be sufficient for most of my UIs. Now I just have to get OpenGL to work on my Linux box …