Continuous Integration

The notorious complexity of software used in scientific research presents serious challenges to reproducibility, a central tenet of the scientific method. If an experimental result obtained in one instance cannot be reproduced by others we have a problem. The proliferation of software components in science coupled with the mobility of experts in each component are a recipe for obsolescence and system decay. These are problems Continuous Integration was developed to help with. Continuous integration is well understood practice in the software engineering community. Here we’ll be looking at the practice but with particular focus on its applicability in increasing understanding and robustness of artifacts within the research software community. The image below is a link to RENCI’s continuous integration site.

Hudson is the build automation tool used to periodically build a wide assortment of software components. It’s a mature, stable application with a large user base and extensive feature set. Automated builds notify teams of failures quickly to help keep systems healthy.

Nexus is the artifact repository. An artifact repository is a searchable, web based endpoint for accessing binaries of various kinds. It’s a stable, one stop shop for versioned binary artifacts. Knowing what version of an executable was used and how to reliably retrieve it in the future is key to reproducible computing results.

Opengrok is the source code search tool. It provides full text search access to multiple code bases. It’s aware of a wide variety of programming languages and is able to cross reference symbols. It can also show history information and do diffs using various popular SCM tools.

Aside | Posted on by | Leave a comment

Architecture and Configuration


Let’s just start with a picture.

Figure 1. CI Server environment architecture.

The CI Server

The three systems – Hudson, Nexus and Opengrok are installed in a single Tomcat instance. The Tomcat instance, bundled with all three applications is a single binary download, itself available from the Nexus repository. An automated build creates the package. As the CI system matures, new versions of the system will be created with updated components and an incremented overall version number.

Server Installation

At a high level, installing involves downloading and unpacking the archive, sourcing the environment script and configuring the applications.

  • Get:
    • Download the ci-server archive.
    • Unpack the archive:
      • tar xvzf ci-server-0.1-SNAPSHOT.tar.gz
  • Install build tools. At RENCI, these include:
    • The Subversion client
    • The Git client
    • Java JDK 1.6.0_21
    • Apache Maven 2.2.1
    • Fortran 11.1 – and any other compilers needed.
  • Initialize:
    • Source the environment file
    • source ci-server-0.1-SNAPSHOT/bin/
  • Apache:
    • Install apache
    • Install mod_ssl
    • Ensure mod_proxy_ajp is installed and enabled.
      • /etc/httpd/conf.d/proxy_ajp.conf should contain entries like:
        • ProxyPass /hudson ajp://localhost:8009/hudson
        • ProxyPass /nexus ajp://localhost:8009/nexus
        • ProxyPass /opengrok ajp://localhost:8009/opengrok
    • Configure an SSL certificate for mod_ssl.
  • LDAP:
    • Import the RENCI LDAP key into the Java keystore
    • Configure Nexus LDAP parameters
    • Configure Hudson LDAP
  • Plugins / Jobs:
    • Use Hudsons administrative interface to add desired plugins
    • At RENCI, these include:
      • Maven (enabled by default)
      • Subversion (enabled by default)
      • Git
      • GForge
      • Trac
      • Violations
      • Greenballs
    • Setup a job to execute the ci sys-admin tasks:
      • rencici/continuous-integration/server/maint-pom.xml
  • Network:
    • Open port 443 for HTTP over SSL
    • Open port 80 for HTTP plain
    • Open port 22 for SSH
    • Ensure other inbound TCP ports are closed
  • Services:
    • Use chkconfig or similar to register services (httpd, tomcat)
    • For Tomcat, a sample startup script is provided called ci-server.
      • The following two lines must be modified for your environment:
        • the_user=build
        • source /ci/app/ci-server-0.1-SNAPSHOT/bin/

In general, the HTTPS port (443) accepts connections while the HTTP port (80) does not. However, port 80 is open at the firewall. An Apache VirtualHost is configured to allow non-SSL connections for serving binary artifacts from the repository. This is done by denying connections to Location elements other than the repository’s content area. A primary motivation for this configuration is that Nexus and Hudson accept LDAP authentications so we want to be careful not to allow passwords to be transmitted unencrypted.

Apart from SSH, these are the only services. The Tomcat instance and HTTPD are registered as services with the system’s init.d daemon. There is a dedicated user account for the above tools and Tomcat runs as this user.

Both Hudson and Sonatype Nexus are configured to support LDAP authentication for administrative access. A number of plugins have also been added to Hudson for better integration with the local development environment.

Administrative Policy

There will be two admin users to provide support redundancy. Others may be granted admin access on a temporary basis for one or two days to solve specific problems.

Posted in Uncategorized | Leave a comment

Designing the Stack

Continuous Integration (CI)


Continuous Integration is more a way of working than it is a specific set of tools. In a CI environment the software shared with the team always works. Automated tests are included with builds to verify that each check-in passes the tests. Bad builds are fixed quickly. The idea is to bake quality in from the start and keep it there with every change.

Tools are the background to CI culture. Continuous Integration environments often start with an automated build, add automated unit tests and move on to artifact management, automated functional tests and source searchability among other components. With each step, transparency, quality and repeatability increase, broken builds, bugs and other bad surprises decrease.

OSG Cyberinfrastructure at RENCI uses a set of Continuous Integration tools as the technology background to the CI culture. Currently, these are:

It currently manages 16 diverse modules in support of researchers using the Open Science Grid. These range from Fortran and C components built with make to Java applications built with Maven.


This configuration was used at a financial services company for three years where it formed the tech backbone of an agile development process. It served a team ranging in size between five and 100 developers geographically distributed between four sites in the US, Europe and India.

The stack was used to build a brokerage equity research and trading platform that’s comprised of software ranging in age from 0-15 years. This included:

  • Compiling and / or packaging code in
    • Java / J2EE (versions ranging from 1 – 6)
    • Adobe Flex
    • JavaScript
    • C/C++
    • Visual C++
    • Microsoft .NET (C#)
    • PL/SQL
  • Automated deployment to development servers
  • Static code vulnerability analysis with Fortify
  • Automated data integrity checks for data loads
  • Executing unit tests using JUnit and EasyMock
  • Executing functional tests with
    • HP Mercury QuickTest Pro for Adobe Flex
    • Eviware SOAP UI

A build farm of just under a dozen machines hosted instances on Unix and Windows building multiple release branches of the platform.

In the process of choosing that configuration we evaluated many tools. We frequently re-evaluated the stack and, at one point abandoned Artifactory for Nexus. Still, I thought it would be good to give them another look to see what’s changed in the meantime. The nutshell is that – so far – the reasons for the choice still appear to hold true of the current environment. The automated build landscape is the most complex and definitely bears watching and further investigation. As noted below, I’ll be getting together with various folks to round out the evaluation which is in progress.


We’ll only be reviewing tools that rise above the rest. So given that the ones we’ll be looking at are all good and all do the basics, how do we make a decision? Well, we need to look at the environment’s requirements. Some of these tools may be a perfect choice for someone else or for a subset of our environment but not such a great choice for the big picture. Here’s an overview of the target environment for context:

  • Code Heterogeneity: We’ll be building C,C++,Java,Fortran,C# code primarily as well as packaging things in any number of scripting languages. There is a corresponding variety of build tools (Ant, Maven, make, cmake, you name it). We support researchers. New researchers can show up with just about any kind of code imaginable.
  • Multi-Platform: There’ll be several flavors of Linux, Unix and Windows. This dimension will also span i386 and x86_64 versions, etc. So ease of installation, version controlled configuration and portability are key.
  • Multiple Teams: There are multiple teams who’ll be using the CI environment. It may occasionally be useful to have separate instances to support different teams appropriately.
  • Manageability: The need to manage multiple instances practically is important. Because of the multi-platform and multiple-team aspects of our situation we can expect to have several instances. Support for a central, consolidated interface to manage multiple instances is not that common. Also, the ease with which configuration can be stored, versioned and deployed is a factor here.
  • Integration: Developers will need to know about build status. The following methods of notification should be supported:
    • Must integrate with GForge
    • A simple web interface with status at a glance.
    • A dashboard visual overview of lots of projects.
    • RSS feed support.
    • Eclipse IDE integration.

Evaluation details follow. The evaluation activity is in progress.

Automated Build Systems

Evaluation Set


At a high level, these systems all provide automated builds for multiple projects. They are generally able to

  • fetch updates from an SCM repository
  • invoke a tool like Maven, Ant or Make to execute the build
  • Send email to a configurable list of users with results
  • Format those emails to be very clear and explanatory
  • Provide multiple views of build status
  • Shell out to build any kind of code via a shell script

All of the systems in the evaluation set do the basics. All of them also have Eclipse plugins.

Here’s a nutshell overview of noteworthy items so far…

CruiseControl (2.8.3)

  • Excellent configuration documentation.
  • Versionable, centralized text configuration file.
    • Easy to store all configuration in SCM
    • One consolidated config file says it all
    • Excellent configuration documentation
  • Intuitive default View
    • All projects lined up, sortable, status, etc.
  • Information filled dashboard view
  • Tiled view scales really well
  • Extensive drill-down capability for status detail
  • Excellent integration: Eclipse, RSS, Firefox, Tray App, etc.
  • Scalable remote management tools. Needed for VM/QA.

Continuum (1.3.6)

  • Interesting: multiple build queues
  • Interesting: Hudson clusters.
  • Extensive management interface (but)
  • It’s too Maven specific – needs to work well with others.
  • Eclipse Plugin – version number 0.0.1, looks very light.
  • Un-intuitive UI (where’s the build queue / dashboard ?)
  • No simple way to point at a project on the filesystem (?)
  • Does not appear to support Git SCM.
  • Does not integrate with GForge. [dealbreaker]
  • Doesn’t support RSS feeds.

Hudson (1.381)

Note: this is very much in progress as I look for someone more familiar with Hudson to discuss this in detail.

  • Assorted niceties CruiseControl doesn’t do:
    • Create Views for grouping builds.
    • Built in file browsing of project files
    • Incremental build status display
    • Maven-aware configuration and display
      • Maven specific configuration process
      • Individual mojos
      • Surefire unit test results
  • Has Eclipse Plugin, Git support, etc.
  • Seems like the projects have to be under the Hudson directory. Other tools need to be able to share the source. Don’t really want Hudson to own it all. Configurable?
  • Managing configuration’s complex compared to CC
    • There are many config files
    • The number changes over time
  • The UI’s pretty busy compared to other build tool homepages. Like, there’s a really big picture of a butler. What’s that about? YMMV.

Here’s an example of versioning configuration under Hudson:

  1. Add any new jobs, users, plugin configurations, et cetera:
  2. svn add -q --parents *.xml jobs/*/config.xml users/*/config.xml userContent/*
  3. Remove anything from SVN that no longer exists (such as a deleted job):
    svn status | grep '\!' | awk '{print $2;}' | xargs -r svn rm
  4. Check it in!
    svn ci --non-interactive --username=mrhudson -m "automated commit of Hudson configuration"

I plan to get together with a couple of local Hudson users to review their experiences as they have probably addressed several of the items I cite above.

Artifact Repositories

Evaluation Set


Artifact repositories make systematic publishing, discovery and management of software artifacts simple, robust and secure. Complex systems increasingly compose themselves by downloading multiple components from elsewhere. This may happen at build time or, in more dynamic environments, at execution time. Artifacts may be executables, Java binaries, scripts of various kinds (shell, Perl, Python, etc). They all have in common that we’d like to be able to

  • Get artifacts from a well known URL over HTTP
  • Know that the file has not been corrupted
  • Have clear, consistent artifact versions
  • Take advantage of transitive dependencies
  • Search for artifacts

General Attributes

Mature artifact repositories are able to receive artifacts published via Apache Maven. Maven is a highly flexible mechanism able to build and deploy any kind of binary file with associated version and meta data. A hash is also created to allow verification of the integrity of the download.

Repositories can be created to map local as well as remote resources.

Search allows artifacts to be found by keyword. This generally includes full text search as well as attribute oriented search.


  • Attractive user interface
  • Lots of features (RPC, CLI, …).
  • Binary repository
    • When corrupted you’re stuck and lose data
    • Integrates well with nothing
    • Unlikely to perform at scale


  • Trivial install and upgrade process
  • Excellent documentation
  • Works with local file system
  • Remote repositories are also searchable
  • Has LDAP plugin for SSO integration

From real world experience, the binary repository behind Artifactory is a non-starter for the most basic reason – data integrity. It also inhibits intgration. As one Nexus user understated it: “having my repository data available directly on the file system makes me feel better”.

Source Search

Evaluation Set


When dealing with a lot of source it’s really important to be able to do full text search of the source and link between symbols’ uses and their definitions. It’s also really helpful to be able to trace changes in source control back to the developers who made the changes. An unspoken requirement for all of these tools is that they need to be web  based for maximum reach. So this tool’s goal is absolute transparency of the source.

Note: Atlassian Fisheye is probably the best thing available in this space from a raw feature perspective. If we are interested in spending about $8K for the 100+ users license, we should definitely look into it.

In the meantime, OpenGrok is robust, highly functional, has great history and diff features, integrates with several back ends.

So OpenGrok wins. It’s relatively easy to set up (WAR file in Tomcat and a Cron job for the indexer). For a large installation we’d want to do some homework to figure out disk capacity requirements in further detail.

Posted in artifactory, cruisecontrol, hudson, sonatype nexus, Uncategorized | Leave a comment