Easily generate CSV in Java (e.g. from Servlet)

To generate CSV from Java consider this simple interface:

package net.doepner;

import java.io.IOException;

/**
 * Generates CSV (comma separated values) for rows of Java objects
 */
public interface ICsvWriter {

    /**
     * Adds a row of objects to the CSV document
     *
     * @param values The objects in the row (count must match the number of the
     *               headers)
     */
    void row(Object... values);

    /**
     * Writes CSV based on the the String representations of the objects
     *
     * @param appendable The writer to append to
     * @throws IOException If underlying IO fails
     */
    void appendTo(Appendable appendable) throws IOException;
}

I implemented the interface with this CsvWriter class:

package net.doepner;

import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;
import java.util.LinkedList;
import java.util.regex.Pattern;

/**
 * Convenient generation of CSV
 */
public class CsvWriter implements ICsvWriter {

    private static final CharSequence CSV_ROW_END = "\r\n";
    private static final char LINE_BREAK_WITHIN_CELL = '\n';

    private static final Pattern QUOTES = Pattern.compile("\"");
    private static final String ESCAPED_QUOTE = "\"\"";

    private final Object[] headers;
    private final Collection<Object[]> rows = new LinkedList<Object[]>();

    /**
     * @param headers The objects representing the column headers
     */
    public CsvWriter(Object... headers) {
        this.headers = Arrays.copyOf(headers, headers.length);
        if (cols() == 0) {
            throw new IllegalArgumentException("No columns");
        }
    }

    @Override
    public final void row(Object... values) {
        if (values.length != cols()) {
            throw new IllegalArgumentException("Specify " + cols() + "values ");
        }
        rows.add(Arrays.copyOf(values, values.length));
    }

    @Override
    public final void appendTo(Appendable appendable) throws IOException {
        appendRow(appendable, headers);

        for (Object[] row : rows) {
            appendRow(appendable, row);
        }
    }

    private static void appendRow(Appendable appendable, Object[] row)
            throws IOException {
        boolean first = true;
        for (Object value : row) {
            if (first) {
                first = false;
            } else {
                appendable.append(",");
            }
            appendable.append('"');
            appendable.append(toCsvString(value));
            appendable.append('"');
        }
        appendable.append(CSV_ROW_END);
    }

    private static CharSequence toCsvString(Object object) {
        if (object instanceof Iterable) {
            final StringBuilder sb = new StringBuilder();
            boolean first = true;
            for (Object o : (Iterable<?>) object) {
                if (first) {
                    first = false;
                } else {
                    sb.append(LINE_BREAK_WITHIN_CELL);
                }
                sb.append(toCsvString(o));
            }
            return sb.toString();
        } else {
            if (object == null) {
                return "";
            } else {
                final String s = object.toString();
                return QUOTES.matcher(s).replaceAll(ESCAPED_QUOTE);
            }
        }
    }

    private int cols() {
        return headers.length;
    }
}

Example for how it can be used in a Servlet, here with full support for Unicode characters using UTF-8, in a way that even Excel understands:

    private static final char BYTE_ORDER_MARK = (char) 0xfeff;
 
    private static void generateCsv(ServletResponse resp, 
                                    Iterable<IMessage> messages)
            throws ServletException {

        resp.setContentType("text/csv");
        resp.setCharacterEncoding("UTF-8");

        final ICsvWriter csv = new CsvWriter(
                "Date", "Sender", "Recipients", "Subject");

        for (IMessage msg : messages) {
            csv.row(msg.getDateTime(), msg.getSender(), 
                    msg.getRecipients, msg.getSubject());
        }

        final PrintWriter writer = getResponseWriter(resp);
        try {
            // The BOM is required so that Excel will recognize UTF-8
            // characters properly, i.e. all non-ASCII letters, etc.
            writer.print(BYTE_ORDER_MARK);
            writer.flush();

            csv.appendTo(writer);
            writer.flush();

        } catch (IOException e) {
            throw new ServletException(e);
        }
    }

    private static PrintWriter getResponseWriter(ServletResponse resp)
            throws ServletException {
        try {
            return resp.getWriter();
        } catch (IOException e) {
            throw new ServletException(e);
        }
    }

Tomcat: Require authentication but no roles

Sometimes you might want to protect some parts of your Java web application from anonymous access, but not impose any authorization constraints: Every authenticated user should be automatically authorized, no matter what roles they have or don’t have.

Tomcat supports this by setting allRolesMode=”authOnly” on the Realm definition, usually in META-INF/context.xml, in combination with <security-constraint> entries in WEB-INF/web.xml that declare an <auth-constraint> with <role-name>*</role-name>.

Simple accordion using CSS3 :target selector (no JavaScript)

I made a simple “accordion” style UI that consists of several collapsed sections with only the “current” one expanded, using the CSS3 :target selector. I wanted the basic functionality of the JQuery UI accordion, but in a minimalistic way, without any JavaScript.

The :target selector is supported in all modern browsers, including IE9 and IE10, but not in IE8 and earlier.

In a real application with a variable number of accordion sections you would probably generate the HTML for the sections with some sort of iteration. You need to make sure that each section id is a valid unique id on the page and that the href of the link in the section header points to the section id, prefixed by ‘#’.

The minimal sample code is in jsfiddle 1, a version with some more visual style is in jsfiddle 2. For illustration, I am also posting the sample code here (see below).

I used a <div> for the whole accordion, a <fieldset> for each section, a <legend> for the section header and nested <div>…</div> for the section content.

Please note that you can use other HTML tags, as long as you use the CSS classes used in the stylesheet and the same kind of nesting.

HTML code:

<div class="accordion">
  <fieldset class="section" id="id1">
    <legend class="section-header">
      <a href="#id1">Section 1</a>
    </legend>
    <div class="section-content">
      Section 1 content
    </div>
  </fieldset>
  <fieldset class="section" id="id2">
    <legend class="section-header">
      <a href="#id2">Section 2</a>
    </legend>
    <div class="section-content">
      Section 2 content
    </div>
  </fieldset>
</div>

CSS stylesheet code:

.accordion .section {
    margin: 1em;
}
.accordion .section-header a {
    text-decoration: none;
    color: gray;
}
.accordion .section:hover .section-header a {
    color: red;
}
.accordion .section .section-content {
    display: none;
}
.accordion .section:target .section-content {
    display: block;
}
.accordion .section:target {
    border: 2px solid navy;
}
.accordion .section:target .section-header a {
    color: navy;
    cursor: default;
}
.accordion .section-header a:before {
    content:"\25BA\0000a0"
    /* unicode triangle pointing right */
}
.accordion .section:target .section-header a:before {
    content:"\25BC\0000a0"
    /* unicode triangle pointing down */
}

Rails: Generic drop-down value lists with management UI

Recently, I had to implement the following feature in a Rails 3.2 application:

  • Certain model attributes have a finite list of possible values.
  • On the web interface users edit the attributes via drop-down lists.
  • Each value list is fully manageable (add, edit, delete) via an admin web interface.
  • Each value list is either alphabetically ordered by name or custom ordered.
  • Values can be “deactivated” by admins so that users can still see but not select them.
  • Some value lists (and corresponding drop-down lists) are “grouped” by category.
  • The groupings are also manageable via admin web interface.
  • The categories themselves are also implemented as manageable value lists.

Example for a grouped value list: Cities where a company is present, grouped by global corporate regions.

The implementation also had to satisfy these design goals:

  • Each value type is represented by a separate Active Record model and database table.
  • Minimize code duplication on the controller, model, view and helper levels.
  • All drop-down lists are generated by one generic form helper method.
  • The implementation must work well with the client_side_validation gem.
  • Use Oracle database with foreign key and uniqueness constraints.

Models

To reduce code duplication, I first tried Active Record bases class inheritance, with the common code in abstract base classes. But this did not work well with the uniqueness checks generated by the client_side_validations gem and so I used modules instead.

I keep all my modules in app/modules. To have them auto-loaded, I add this to config/application.rb:

config.autoload_paths += %W(#{config.root}/app/modules)

app/modules/

combo_common.rb

module ComboCommon
  extend ActiveSupport::Concern

  included do
    attr_accessible :active, :description, :name, :order

    default_scope order('"ORDER", name')

    validates :name, :presence => true, :uniqueness => true

    validates :order, :presence => true,
              :numericality => {:only_integer => true, :greater_than => 0}
  end

  def is_active
    active ? "Yes" : "No"
  end

  module ClassMethods
    def title
      name.demodulize.titleize
    end
  end
end

combo_with_category.rb

module ComboWithCategory
  extend ActiveSupport::Concern

  included do
    attr_accessible :category_id, :category
    validates :category_id, :presence => true
  end

end

app/models/

Example : cities grouped by region

combo/city.rb

class Combo::City < ActiveRecord::Base
  include ComboCommon, ComboWithCategory

  def self.category_type
    Combo::Region
  end

  belongs_to :category, :class_name => category_type.name

end

combo/region.rb

class Combo::Region < ActiveRecord::Base
  include ComboCommon

  has_many :category_members, class_name: Combo::Location.name,
           foreign_key: "category_id", readonly: true
end

app/controllers/

combo/controller_base.rb

class Combo::ControllerBase < ApplicationController

  before_filter :ensure_admin

  def init_type(type)
    @type = type
  end

  def index
    @items = @type.all
    render "combo/index"
  end

  def new
    @item = @type.new(active: true, order: 100)
    render :template => "combo/new", :layout => "popup"
  end

  def edit
    @item = @type.find(params[:id])
    render :template => "combo/edit", :layout => "popup"
  end

  def create
    @item = @type.new(params[type_symbol])
    if @item.save
      redirect_to_index "was successfully created."
    else
      render "combo/new"
    end
  end

  def update
    @item = @type.find(params[:id])
    if @item.update_attributes(params[type_symbol])
      redirect_to_index "was successfully updated."
    else
      redirect_to_index "was NOT successfully updated."
    end
  end

  def destroy
    item = @type.find(params[:id])
    begin
      item.destroy if item
      redirect_to_index("was successfully deleted.")
    rescue ActiveRecord::StatementInvalid => e
      msg = e.message
      # See http://docs.oracle.com/cd/E11882_01/server.112/e17766/e2100.htm
      # for reference documentation of Oracle error codes
      if msg.include?("ORA-02292") and msg.include?("child record found")
        redirect_to_index("could not be deleted because it is still in use! ")
      else
        raise
      end
    end
  end

  private

  def type_symbol
    @type.name.delete("::").underscore.to_sym
  end

  def redirect_to_index(message)
    redirect_to polymorphic_path(@type), :notice => @type.title + " " + message
  end

end

Example : cities grouped by region

combo/cities_controller.rb

class Combo::CitiesController < Combo::ControllerBase

  before_filter { init_type(Combo::City) }

end

combo/regions_controller.rb

class Combo::RegionsController < Combo::ControllerBase

  before_filter { init_type(Combo::Region) }

end

app/views

combo/index.html.erb

<div class="heading clearfix">
    <h2><%= @type.title.pluralize %></h2>

    <div class="actions">
        <%= link_to 'New Entry', new_polymorphic_path(@type),
                    :class => "button colorbox" %>
    </div>
</div>

<table id="index" class="display">
    <thead>
    <tr>
        <th>Name</th>
        <th>Description</th>
        <% if @type.respond_to? :category_type %>
            <th>Category</th>
        <% end %>
        <th>Order</th>
        <th>Active</th>
        <th>Created</th>
        <th>Modified</th>
        <th class="no-search no-sort"></th>
        <th class="no-search no-sort"></th>
    </tr>
    </thead>

    <tbody>
    <% @items.each do |item| %>
        <tr>
            <td><%= item.name %></td>
            <td><%= item.description %></td>
            <% if item.respond_to? :category %>
                <td><%= item.category.name %>
            <% end %>
            <td><%= item.order %></td>
            <td><%= item.is_active %></td>
            <td><%= item.created_at.to_s(:date) %></td>
            <td><%= item.updated_at.to_s(:date) %></td>
            <td><%= link_to 'Edit', edit_polymorphic_path(item),
                            :class => "colorbox" %></td>
            <td><%= link_to 'Delete', item,
                            :confirm => 'Are you sure?',
                            :method => :delete %>
            </td>
        </tr>
    <% end %>
    </tbody>

</table>

combo/edit.html.erb

<h3>Editing <%= @type.title %></h3>

<%= render "combo/form" %>

combo/new.html.erb

<h3>New <%= @type.title %></h3>

<%= render "combo/form" %>

combo/_form.html.erb

<%= form_for @item, :validate => true do |f| %>
    <% if @item.errors.any? %>
        <div id="error_explanation">
            <h2><%= pluralize(@item.errors.count, "error") %>
                prohibited this value list item from being saved:</h2>
            <ul>
                <% @item.errors.full_messages.each do |msg| %>
                    <li><%= msg %></li>
                <% end %>
            </ul>
        </div>
    <% end %>

    <table class="no-borders">
        <tr>
            <td><%= label_for f, :name %></td>
            <td><%= f.text_field :name, :class => "required" %></td>
        </tr>
        <tr>
            <td><%= label_for f, :description %></td>
            <td><%= f.text_field :description %></td>
        </tr>
        <% if @type.respond_to? :category_type %>
            <tr>
                <td><%= label_for f, :category_id, "Category" %></td>
                <td><%= combo(f, :category_id, @type.category_type) %>
                </td>
            </tr>
        <% end %>
        <tr>
            <td><%= label_for f, :order %></td>
            <td><%= f.number_field :order %></td>
        </tr>
        <tr>
            <td><%= label_for f, :active %></td>
            <td><%= f.check_box :active %></td>
        </tr>
    </table>

    <br/>

    <div class="actions">
        <%= f.submit %>
        <%= link_to 'Cancel', polymorphic_path(@type), :class => "button" %>
    </div>
<% end %>

helper methods

app/helper/application_helper.rb

module ApplicationHelper
  include ComboUtil
end

app/modules/combo_util.rb

module ComboUtil

  def combo_link(type)
    link_to simple_plural(type).titleize, polymorphic_path(type)
  end

  def combo(form, attr, type, options = {}, html_options = {})
    form.select(attr, combo_options(type, form.object.send(attr)), options,
                html_options)
  end

  def combo_filter(type)
    render 'combo_filter', type: type
  end

  private

  def simple_plural(type)
    type.name.demodulize.pluralize
  end

  def combo_options(type, id)
    if type.respond_to? :category_type
      grouped_options_for_select(all_grouped_options(type),
                                 disabled: inactive_options(type),
                                 selected: id)
    else
      options_for_select(all_options(type),
                         disabled: inactive_options(type),
                         selected: id)
    end
  end

  def all_grouped_options(type)
    type.category_type.all.map do |cat|
      [cat.name, cat.category_members.map { |m| [m.name, m.id] }]
    end
  end

  def all_options(type)
    type.all.map { |m| [m.name, m.id] }
  end

  def inactive_options(type)
    type.where(active: false).map { |m| m.id }
  end

end

Install cygwin and cygwinports packages using apt-cyg

I recently started using apt-cyg and like it a lot. It allows easy installation of packages from Cygwin and Cygwin ports repositories.

It is inspired by Debian’s apt but it does not support anything like a sources.list and it can actually only work against one repo (mirror) at a time. Well, I guess that’s bearable since there are really only two sources (main cygwin and cygwin ports). But I might try apt-cyg-multi which claims to have added the missing cross-repository functionality.

For now, I found it convenient to configure these bash aliases (use a suitable Cygwin mirror and Cygwin Ports mirror for your location):

alias cyg='apt-cyg -m http://mirrors.kernel.org/sources.redhat.com/cygwin/'
alias cyp='apt-cyg -m http://mirrors.kernel.org/sources.redhat.com/cygwinports/'

Update, June 2013: There is a promising-looking apt-cyg fork on github. I just started using it.

Extremely simple one-level drop-down menu using CSS

The CSS below styles an unordered list with nested sub-lists as a horizontal menu with drop-down sub-menus (only one level). I tried to keep the code minimal with no effort for good looks. You can add your own styling as you like.

I have tested this successfully in IE8, IE9 and recent versions of Firefox, Chrome, Safari and Opera. I consider IE6 and IE7 broken beyond repair and basically ignore them.

Please test it in your browser at http://jsfiddle.net/Yk6MS/4/, and let me know about any problems you might find.

Style sheet:

.menu,
.menu ul {
  background-color: tan;
}
.menu > li {
  display: inline-block;
  white-space: nowrap;
}
.menu ul {
  position: absolute;
  display: none;  
}
.menu > li:hover {
  position: relative;
}
.menu > li:hover ul {
  display: block;
}

Example HTML:

<ul class="menu">
  <li>Abra
     <ul>
      <li>Amfdf rt</li>
      <li>Brr zz</li>
     </ul>
  </li>
  <li>Kadabra
     <ul>
      <li>Kala mala</li>
      <li>Hauki dauki</li>
     </ul>
  </li>
  <li>Simsalabim
    <ul>
      <li>Dudel didel</li>
      <li>Dudel döh</li>
      <li>Labedi bap</li>
    </ul>
  </li>
</ul>

JRuby on Rails 3.2

I currently have to work on an application using JRuby on Rails 3.2. My background is 12+ years of Java centric web development using Eclipse or IntelliJ.

There are a few things that I love about working with Java in Eclipse or IntelliJ that I miss when working with Rails:

  • Java’s backwards compatibility
  • Java’s cross-platform consistency
  • That Java has a static type system and is a compiled language
  • The relatively rigid syntax of Java that prevents idiosyncratic DSLs
  • Being able to control mutability, extensibility and visibility at compile-time (private, final)
  • Java’s distinction of interfaces vs implementation classes
  • Powerful auto-completion and Refactoring in the IDE
  • Powerful code inspections and quickfixes in the IDE
  • IDE and compiler feedback about mistakes and design flaws in my code
  • The powerful and rich ORM features of JPA and Hibernate

Dynamic language fans might cringe at some of the points listed above. Feel free to provide constructive feedback or well-reasoned criticism.

There are however a couple of things that I like about Ruby on Rails 3.2:

  • Dependency management using gems and bundler
  • A very standardized project layout (directory structure and conventions)
  • Code generation (as a convenient way to quickly create sample code)
  • Out-of the box integration and support for JQuery, CoffeeScript, SASS
  • Cool gems like colorbox-rails for turning a regular view into a modal dialog
  • The asset pipeline

I am still in the early phase of my Rails experience and I will try to post more about it as things evolve.

Save video stream as file using rtmpdump (even on Windows)

The rtmpdump tool can help you save video streams as local files.

It comes with most Linux distributions, e.g. on Ubuntu or Debian (with sudo):
sudo apt-get install rtmpdump

On Windows, the most convenient way is probably via Cygwin and Cygwin Ports:

  1. Install Cygwin
  2. Install Cygwin Ports
  3. On the package selection screen, select “rtmpdump”

Read the manual page and study the options:
man rtmpdump