Child pages
  • Taming uPortal -- Multi-Project, Multi-Environment Builds

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

For all its excellent capabilities, as a software system uPortal can be a difficult beast to work with. In its role as a Portal framework targeting the JSR-168 Portlet Specification (soon to be JSR-286, Portlet 2.0), it's very nature is to combine with and aggregate independent software projects. It is also an integration platform: it commonly requires passwords, URLs, driver classes, and diverse configuration settings to communicate properly with external systems.

These responsibilities add complexity to the portal as a software project. It's challenging to manage the relationships between the uPortal software and it's dependent projects, especially Java Portlet applications, but also non-portlet components that form a deployment unit with the portal like Apache or Tomcat. The portal, moreover, commonly expresses itself in multiple environments: production, test, development, local, etc. These environments usually require different data, configuration, integration settings.

This page describes some coordinated practices designed to tame these complexities. These solutions are not the only solutions, and may not be suitable for everyone. All options have advantages & drawbacks.

Jasig Source Code

The first order of business is obtaining source code – uPortal, community portlets, perhaps others – from Jasig. Jasig projects are free open source software (FOSS) and can be easily downloaded by anyone. In fact there are many ways to get Jasig source code, and several things you can do with it once you have it. Not surprisingly, some approaches will put you in a better position for the future than others.

In choosing your methods, one of the leading considerations is how easy will it be to upgrade to future releases of uPortal and/or Jasig community portlets. The answer varies between institutions and from project to project, but always it gets dramatically harder as you apply more local customizations to Jasig source code. This trend applies no matter which practice(s) for managing Jasig source you adopt. But some local changes are unavoidable, and there are tactics that can help you manage these differences when the time comes to upgrade.

Vendor Branching

Vendor branching is probably the leading practice in software for addressing this class of problem. In essence, it's a strategy for harnessing & applying the differences between version releases of 3rd-party libraries in svn merge operations. Vendor branching must be set up from the beginning, before you make any customizations.

The canonical setup process is as follows (Subversion):

  1. Create a space in SVN for vendor project foo at /vendor/foo/
  2. Import foo source code under /vendor/foo/current/
  3. Using svn copy tag the foo source using the current version name, i.e. /vendor/foo/1.0.0-GA/
  4. Again using svn copy, fold the foo source into your project, i.e. myProject/trunk/foo/

Now you can make local changes to foo source code where necessary.

Later, you merge changes to foo from future releases using the following process:

  1. Checkout /vendor/foo/current/ and apply changes from the new release of foo; svn commit
  2. Create a new tag for the foo source based on the new version name, i.e. /vendor/foo/1.1.0-GA/
  3. Inside a checkout of myProject/trunk/, svn merge the differences to foo between the first tag and the second
  4. Resolve conflicts (if any); svn commit

The vendor branch allows you to apply the difference between foo 1.0.0-GA and 1.1.0-GA using svn merge. The advantage of svn merge is it's ability to reconcile these differences with local customizations automatically, provided there are no conflicts. Without the vendor branch, this process must be done by hand. This advantage is the upside of vendor branching; the downsides are in complexity & effort.

Vendor branching makes most sense when:

  • You're likely to benefit greatly from the automated merge process (e.g. your customizations extend beyond configuration settings & added files)
  • You don't expect to replace the vendor source (vendor drop) very often
  • You have a team member who is comfortable with advanced SCM concepts

Subversion Externals Definitions

The svn:externals feature is another good alternative for managing updates to Jasig source code. In a nutshell, this feature allows you to federate Subversion repositories: yours, and an external repository hosting 3rd-party source code (Jasig). This approach makes it very easy to change the version – even the revision – of the 3rd-party project that your local version is based off.

A good way to set up svn:externals is as follows:

  1. Create an ext/ subdirectory within your project in your subversion, e.g. myProject/trunk/ext/
  2. svn add a file called .externals to this directory; specify the intended value of the svn:externals property within it
  3. From inside ext/, use svn propset svn:externals -F .externals . to set the svn:externals property based on the file
  4. Use svn commit -N . .externals to commit changes to the file and the svn:externals property of the ext/ directory atomically
Code Block
titleExample .externals File
uPortal -r 21424
email-preview -r 21396
NewsReaderPortlet -r 20765
FeedbackPortlet -r 21397
AnnouncementsPortlet -r 21338
CalendarPortlet -r 20743
JasigWidgetPortlets -r 20743

Now if you perform an svn update from a checkout of myProject/trunk/ or the ext/ directory within it, you will also get a copy of the 3rd-party source code pegged on the revision you specified.

Later, you can pick a different revision using the following process:

  1. Edit the .externals file
  2. From inside ext/, use svn propset svn:externals -F .externals .
  3. Use svn commit -N . .externals to commit the change
  4. Use svn update from myProject/trunk/ or the ext/ directory within it

This practice allows you to upgrade 3rd-party source code very quickly. In open source, this advantage is a major benefit since it makes it very easy to contribute bug fixes and uncontroversial enhancements to the project directly, then pull them into your project with a revision bump. This process supports a very tight feedback loop between the open source project and the organizations using it. The result is increased activity and a faster rate of progress on the project.

On the downside, this approach does not provide the automated merge capabilities that vendor branching does. In fact it does no merging at all; you have to bring additional tools & practices to bear for merging local customizations with 3rd-party source code. (See Project Overlays below.) Lastly, although putting your bug fixes & enhancements into the parent project first is elegant and greatly streamlined by this practice, it requires access to a project committer.

svn:externals makes most sense when:

  • You expect to make few changes to 3rd-party source beyond configuration settings & added files
  • You want the freedom to pick up bug fixes & enhancements rapidly as they happen in the community
  • You are able to work with a project committer

Overlay Projects

You can use an overlay approach when you want to work with a pristine (unmodified) copy of 3rd-party source code, but you nevertheless need to apply some local differences such as configuration settings, additional files, skinning, etc. There's more than one way to do overlays, but remember that some overlay approach will likely be needed when you choose the svn:externals strategy described above.

Maven Overlays

The Maven WAR Plugin provides baked-in overlay support for WAR projects. It's powerful, and relatively simple to implement. This approach allows you to manage only the file(s) you want to change, pulling in the bulk of project files directly from 3rd-party source.

These are the basic steps for creating a Maven overlay project:

  1. Create a pom.xml file based on the project you'll be overlaying (you need to build the same way)
  2. Add the origin project as a dependency (must be <packaging>war</packaging>)
  3. Add to your project any file(s) in the origin project you wish to replace (must be same name & location)
  4. Add any additional file(s) your project needs beyond what the origin project has
  5. Use mvn install or mvn package (as normal) to build your project (the origin artifact must be in a visible repo)

You will see that, although your src/ files remain as sparse as you made them, your artifact and your target/ directory include the entire origin artifact, exactly as if those files were in your project directly. This is the default behavior for Maven overlays; many configuration options are available for fine-tuning this behavior to suit your needs.

This overlay solution is good when:

  • Your project & the origin project specify the war packaging type
  • You don't expect pom.xml file of the origin project to change much (you have to keep yours mostly in sync)

"Brute Force" Overlays


Maven Filters


Groovy "Puppet Master" Build Script

Sample Script


This section is unfinished. There needs to be some explanation to go with the sample script, and the script itself is missing part of the example (portlets). But if you're familiar with uPortal and Groovy, you may be able to put together the purpose and function of this script just by looking at it.

Code Block
// Prefix for shell output from this script
def PFX = '[build.groovy]';

// Location of the Apache Ant executable that will be used to build uPortal, 
// because the process execution features in Groovy don't seem to want to use 
// the PATH variable
def ANT_HOME = System.getenv('ANT_HOME');
def ANT_FILENAME = System.getProperty('') =~ /Windows/ ? 'ant.bat' : 'ant';

//Location of the Apache Maven executable that will be used to build portlets, 
//because the process execution features in Groovy don't seem to want to use 
//the PATH variable
def M2_HOME = System.getenv('M2_HOME');
def M2_FILENAME = System.getProperty('') =~ /Windows/ ? 'mvn.bat' : 'mvn';
def M2_EXEC = "${M2_HOME}/bin/${M2_FILENAME}";

// Flags to skip some stages of the complete build;  useful for frequent, local builds
def skipClean = Boolean.valueOf(System.getProperty('build.clean.skip'));
def skipPortal = Boolean.valueOf(System.getProperty('build.portal.skip'));

// Optional Ant target to run;  the default is 'deploy-ear' but 'deploy-war' 
// will be faster if you don't need to process webapps other than uPortal
def antTarget = System.getProperty('') ?: 'deploy-ear';

def ant = new AntBuilder(); 

 * Figure out the environment for which we're building. We'll take
 * any name they feed us to allow for a variety of build variants
 * but test, qa, and prod are the meat-and-potatoes builds.
 * The default is just the development environment.

def env = args.length > 0 ? args[0] : 'local';

 * Verify that we have a known environment and report...

def propsFile = new File("overlay/uPortal/build.${env}.properties"); 
if (!propsFile.exists())
    // Not a known environment...
    println "${PFX} Target environment '${env}' not recognized;  try...";
    println "${PFX}";
    println "${PFX}   > groovy build.groovy [local|dev|test|qa|prod]";
    println "${PFX}";
    println "${PFX} Aborting the build process.";
    return 100;

// Environmentt & Maven settings
def envSettings = "${env}";
if (System.getProperty('maven.test.skip')) {
    envSettings += " -Dmaven.test.skip=${System.getProperty('maven.test.skip')}";
if (System.getProperty('maven.offline')) {
    envSettings += " -Dmaven.test.skip=${System.getProperty('maven.offline')}";

 *   Clean & reset uPortal source code from Jasig...

if (!skipClean) {
     * Throw away the old blend tree, if there is one. This will keep
     * us from getting artifacts from previous builds...
    println "${PFX}";
    println "${PFX}";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX} Clearing out the previous build";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX}";
    println "${PFX}";
    ant.delete( dir:'work/up-blend' );
     * Copy the source tree into a fresh blended tree...
    println "${PFX}";
    println "${PFX}";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX} Bringing in the uPortal source for environment: ${env}";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX}";
    println "${PFX}";
    ant.copy (todir:'work/up-blend')

 *   Apply Oakland's uPortal overlay & build uPortal...

if (!skipPortal) {
     * Apply our overlays to the blended tree...
    println "${PFX}";
    println "${PFX}";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX} Overlaying local files for environment: ${env}";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX}";
    println "${PFX}";
    ant.copy (todir:'work/up-blend', overwrite:true)
     * Execute the uPortal build system on the blended tree...
    println "${PFX}";
    println "${PFX}";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX} Calling on uPortal's build process to make JARs and WARs.";
    println "${PFX} * * * * * * * * * * * * * * * * * * * * * * * * * * *";
    println "${PFX}";
    println "${PFX}";

    // Assemble the ant command line...
    def cmd = "${ANT_EXEC} ${envSettings} ${antTarget}";
    // Execute the command...
    def process = cmd.execute(null, new File('work/up-blend'));
    // Who knows... something about the output. Heh.
    process.consumeProcessOutput(System.out, System.err);