Developing Plug-Ins - Continous Integration with Jenkins

This page can contain obsolete information, iDempiere moved to maven instead of buckminster

This tutorial is brought to you by Jan Thielemann from evenos GmbH (www.evenos-consulting.de). If you have questions, criticism or improvement suggestions, feel free to visit me (Jan.thielemann) or write me an email

Goal of this tutorial

When you produce source code, at some point there is always the need to automatically build and publish your code. The Jenkins continuous integration server offers a great possibility for this. So in this tutorial we will cover:

• How to setup Buckminster for Jenkins
• How to build iDempiere using Jenkins
• How to build your own Plug-Ins using Jenkins
• How to publish the build results to a web server
• How to use the Director to install or update bundles from the web

Prerequisites

• A running Jenkins instance
• A running Eclipse instance with buckminster installed
• Custom plugins and features which you want to build

This tutorial will not explain how to setup Jenkins. If you don't have any Plug-Ins, take a look at the Plug-In Development section in this wiki. In this tutorial we use two custom plugins and one feature containing these plugins to show how you can build them using Jenkins.

Buckminster and Jenkins

The first thing we have to do is make Buckminster available in Jenkins. To do so, go to your Jenkins application menu and select the "Manage Jenkins" item. Then go to "Manage Plugins". Search for Buckminster plugin and install it.

The next step is to install Buckminster headless on the same machine on which Jenkins is installed. Open up a terminal and connect to where your Jenkins is installed. In our case, Jenkins is installed on a remote server in /var/lib/jenkins. We will install Buckminster in /opt/buckminster-headless-3.7. So go to /opt and create a directory:

mkdir /opt/buckminster-headless-3.7

Now we need the p2 director which you can find here. Extract the zip into the buckminster headless directory. After the zip is extracted make sure that the director is executable:

chmod +x director

Next, install buckminster headless:

./director -r http://download.eclipse.org/tools/buckminster/headless-3.7/ -d /opt/buckminster-headless-3.7/ -p Buckminster -i org.eclipse.buckminster.cmdline.product 
./director -r http://download.eclipse.org/tools/buckminster/headless-3.7/ -d /opt/buckminster-headless-3.7/ -p Buckminster -i org.eclipse.buckminster.core.headless.feature.feature.group 
./director -r http://download.eclipse.org/tools/buckminster/headless-3.7/ -d /opt/buckminster-headless-3.7/ -p Buckminster -i org.eclipse.buckminster.pde.headless.feature.feature.group

If you get any problems with Jenkins related to maven or a "maven provider" in the adempiere.rmap, try the following. Instead of using the above to install Buckminster headless, run this:

./director -r http://download.eclipse.org/tools/buckminster/headless-3.7/ -d /opt/buckminster-headless-3.7/ -p Buckminster -i org.eclipse.buckminster.cmdline.product 

Now switch to the Buckminster installation directory and run these commands:

./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.7/ org.eclipse.buckminster.maven.headless.feature
./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.7/ org.eclipse.buckminster.core.headless.feature
./buckminster install http://download.eclipse.org/tools/buckminster/headless-3.7/ org.eclipse.buckminster.pde.headless.feature

Please notice the following: We made bad experiences when using Buckminster headless from the 4.2 repository so go and try 3.7. You also can have different buckminster installations at the same time.

Now that we have Buckminster installed, we have to configure it in Jenkins. To do so, go again to your "Manage Jenkins" menu. Then go to "Configure System". Find the Buckminster section and click on "Buckminster installations. Add a new buckminster installation and uncheck the "Install automatically" button. Enter "Buckminster Headless 3.7" as the name, /opt/buckminster-headless-3.7/ as the installation directory and "-Xmx512m" as the additional startup parameters. Make sure that the provided number (512) fits the RAM of your server.

That's it! You are done with the first part of this tutorial.

Build iDempiere

iDempiere project in Jenkins

If you want to use Jenkins to build your iDempiere, this is how you do it. First go to your Jenkins and create a new job. Provide a name like "iDempiere" and select free-style as the project type. As the SCM you may want to chose Mercurial. If you do, you must provide a repository. You can chose your own or use https://bitbucket.org/idempiere/idempiere. The default branch for iDempiere is "development". Since we want to build iDempiere on a regulary base, chose periodically in the build triggers section. Here you can provide a schedule like "H 5 * * *". This will build the project every day at 05:00. Now to the more interesting part, the build itself. First add a new shell exectuion step:

rm -rf ${WORKSPACE}/buckminster.output/ ${WORKSPACE}/buckminster.temp/ ${WORKSPACE}/targetPlatform/

This will remove the targetPlatform so we will always start with a clear workspace. The next build step is a run buckminster:

importtargetdefinition -A '${WORKSPACE}/org.adempiere.sdk-feature/build-target-platform.target'
import '${WORKSPACE}/org.adempiere.sdk-feature/adempiere.cquery'
build -t
perform -D qualifier.replacement.*=generator:buildTimestamp -D generator.buildTimestamp.format=\'v\'yyyyMMdd-HHmm -D target.os=*   -D target.ws=*   -D target.arch=* -D product.features=org.idempiere.eclipse.platform.feature.group -D product.profile=DefaultProfile -D product.id=org.adempiere.server.product   'org.adempiere.server:eclipse.feature#site.p2'

Please notice: The perfom command is a single line, ending with "...#site.p2'".

The above buckminster commands will build a p2 repository for the iDempiere server. Instead of "...#site.p2" you can also chose other action types like "...#create.product.zip" but for the case of this tutorial, we will use the site.p2 action.

That's it! Try to build the project. If you want to take a look at the output, go to $JENKINS_HOME/workspace/iDempiere/buckminster.output/

p2 repository

Ok, now that we can build iDempiere, we want to publish the result to a web-server. For this task, every time we succeed in building iDempiere, we will copy the created p2 repository to a place from which we can publish it. In our case, we decided to copy the p2 repository to /opt/idempiere-builds/idempiere.p2/. First, create this directory:

mkdir /opt/idempiere-builds
mkdir /opt/idempiere-builds/idempiere.p2

Since Jenkins is responsible for building and copying the files, we must switch the owner of idempiere.p2 to Jenkins. Also we want a web-server to be able to read the directory so we change the ownership to the following:

chown jenkins:www-data /opt/idempiere-builds/idempiere.p2

Go back to the iDempiere job on your Jenkins and click on configure. Below the run Buckminster build step, add another shell execution:

rm -rf /opt/idempiere-builds/idempiere.p2/*
cp -fR ${WORKSPACE}/buckminster.output/org.adempiere.server_1.0.0-eclipse.feature/site.p2/* /opt/idempiere-builds/idempiere.p2

This will first delete all old resources and then copy the new created. Build your project. If everything worked as expected, you should now find a p2 repository in /opt/idempiere-builds/idempiere.p2.

Web Server

Building the p2 repository should work by now. The next step is to publish it. You can use your preferred webserver. In our case, we will use apache2. First, we create a new symlink in /var/www/ so apache can find the directory:

cd /var/www
ln -s /opt/idempiere-builds/idempiere.p2

Next open the apache config with your favourite text editor and add a new alias so we can find the repository over the internet:

nano /etc/apache2/sites-available/default

Somewhere in your VirtualHost, add the following:

        <Directory /var/www/idempiere.p2>
                AllowOverride AuthConfig
        </Directory>

        Alias /idempiere/p2 /var/www/idempiere.p2

Please notice: A webserver can be configured in many ways. If the above does not work for you, try to get a better understanding of your webserver. It's not that difficult.

Fine, thats it. By now, you should be able to access your p2 repository over the internet.

Build your own Plug-Ins

Preparing your Plug-Ins

If you followed our best practice on how to develop a plugin (Plug-In Development Guide), all your plugins should be located in separate mercurial repositories. But before we can start grabbing code from the repositories and build our own plugins we have to prepare them first.

To do so, open up eclipse and switch to a new, empty workspace. Go to Window->Preference->Plug-in Development->Target Platform. Click on Add and select "Nothing: Start with an empty target definition". Provide a name and add a directory. Select the Variables button and add workspace_loc. Then add targetPlatform to the path:

${workspace_loc}/targetPlatform

Make sure that the targetPlatform directory exists in your workspace. If not, create it before clicking finish. Back in the Preferences, make sure your new created target platform is the active one. Now you can start importing your plugins and features into the workspace by selecting File->Import>Existing Projects into Workspace. Normaly there will be a lot of red but don't worry, we will fix this in the next step.

So, we got a lot of build problems. This is because eclipse cannot find all the dependencies for our plugins. To fix it, the first thing we have to do is to create a new cquery. Click on File->New->Other and select Component Query file from the buckminster section. We call ours buckminster.cquery. Open it and enter your features identifier as the component name. Select eclipse.feature as the type, check the Use Resource Map and enter "buckminster.rmap" as the url. On the properties tab, add target.arch, target.os and target.ws each with "*" as the value.

<?xml version="1.0" encoding="UTF-8"?><cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="buckminster.rmap">
    <cq:rootRequest name="org.evenos.idempiere.uifeature" componentType="eclipse.feature"/>
    <cq:property key="target.arch" value="*"/>
    <cq:property key="target.os" value="*"/>
    <cq:property key="target.ws" value="*"/>
</cq:componentQuery>

You may have noticed that we don't have a buckminster.rmap yet, so again, create a new file but this time chose Resource Map from the buckminster section. Buckminster will use this file to locate the source code of all the dependencies. In our case, the rmap will be really simple. We only need two search paths and two locators. One for iDempiere and one for our plugins. In both cases, the search paths are for eclipse.feature as well as for osgi.bundle.

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0" xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
  <rm:locator pattern="^org\.evenos(\..+)?" searchPathRef="local"/>
  <rm:locator searchPathRef="idempiere-evebuild01"/>
  <rm:searchPath name="local">
    <rm:provider componentTypes="eclipse.feature,osgi.bundle" readerType="local">
      <rm:uri format="{0}/{1}">
        <bc:propertyRef key="workspace.root"/>
        <bc:propertyRef key="buckminster.component"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
  <rm:searchPath name="idempiere-evebuild01">
    <rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="p2" mutable="false">
      <rm:property key="buckminster.mutable" value="false"/>
      <rm:uri format="file:////opt/idempiere-builds/idempiere.p2/">
        <bc:propertyRef key="workspace.root"/>
        <bc:propertyRef key="buckminster.component"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
</rm:rmap>

As you can see, we tell buckminster to look in the local workspace for everything that starts with "org.evenos". Everything else should be in the p2 repository which we build in part two of this tutorial. If you want to test the materialization on your local computer, you can provide the full url to your p2 repository to download dependencies but since we will build on Jenkins, it is better to provide a file path rather than a http url. Howeverm if you want to test your config in eclipse, replace "file////opt..." with e. g. "http://www.evenos-consulting.de/idempiere/p2". Then open the buckminster.cquery and click on Resolve and Materialize. Buckminster should download all the dependencies from the p2 repository and all the red crosses should be gone.

The final step is to create a new target. Simply add a new target definition from the Plug-in Development section and call it for example "build-target-platform.target". Open the file and add a new location. Select directory and like before, provide the following path:

${workspace_loc}/targetPlatform

If you haven't noticed yet, all the files we just created are placed in our feature project. Don't forget to commit the changes in your feature project.

Preparing the workspace in Jenkins

To make building work in Jenkins, we first need to prepare the workspace. Do this by creating a new free-style job and provice a name, please don't use spaces in the name because this will only cause problems. After saving the new job, Jenkins should create a new workspace directory for this job. In our case, it is in /var/lib/jenkins/workspace/NameOfTheJob.

Switch to this directory and clone all the plugins and features you want to build. Make sure you clone with the Jenkins user. Otherwise, Jenkins won't be able to pull an update! Once you are done, you can start configuring the new job.

Build your Feature in Jenkins

If you have a separate mercurial repository for each of your plugins and fragments, follow theses steps to build them. If all of your plugins and fragments are in the same mercurial repository, just add another scm like before.

Go to Jenkins and click on your new job, then click configure. This time we don't select a SCM. We will rather use the command line to keep all the plugins up to date. In Build Triggers, select the iDempiere job for "Build after other projects are built".

In the Build section, the first step is a shell execute:

rm -rf ${WORKSPACE}/buckminster.output/ ${WORKSPACE}/buckminster.temp/ ${WORKSPACE}/targetPlatform/
cd ${WORKSPACE}/org.evenos.idempiere.uifeature
hg pull
hg up -C
cd ${WORKSPACE}/org.evenos.idempiere.base
hg pull
hg up -C
cd ${WORKSPACE}/org.evenos.idempiere.ui
hg pull
hg up -C

In this step, we first delete the target platform and the buckminster directories so we can start with a clear workspace. The next build step is a "Run Buckminster":

importtargetdefinition -A '${WORKSPACE}/org.evenos.idempiere.uifeature/build-target-platform.target'
resolve '${WORKSPACE}/org.evenos.idempiere.uifeature/buckminster.cquery'
clean
build -t
perform -D target.os=* -D target.ws=* -D target.arch=* org.evenos.idempiere.uifeature#site.p2

Here we first import the target definition we created in the last step. Buckminster will create a targetPlatform directory in the workspace for us. Then it will collect all the dependencies from our p2 repository and the workspace. After it has finished, it will build the code and finally create a p2 repository for our feature.

p2 repository

Follow the p2 repository section from the second part of this tutorial to create another directory to which you want to deploy the p2 repository containing your plugins. Then add another shell execution build step after the run buckminster:

rm -rf /opt/zerlina-builds/org.evenos.idempiere.uifeature.p2/*
cp -fR ${WORKSPACE}/buckminster.output/org.evenos.idempiere.uifeature_1.0.0-eclipse.feature/site.p2/* /opt/zerlina-builds/org.evenos.idempiere.uifeature.p2

Webserver

Follow the webserver section in the second part of this tutorial to configure an additional alias for your new p2 repository.

Good to know

Building Features which include Fragments

I recently found out that jenkins wasn't able to build fragments which are part of a feature project. To solve this issue, you can do the following:

• Create a feature project as the master feature
• Create a feature project containing all your plugins
• Create a feature project containing your fragments
• Make the master feature contain the other features
• Build the master feature

Known Problems

If your buckminster.out does contain your plugins and fragments but not your feature, you need to create another feature project. Add your previously created feature to the newly created one and build it with jenkins. Now your buckminster.out should contain your feature as well as the plugins and fragments.

Build Jenkins Job on mercurial commit

If you want to automatically build your jobs when you commit something in mercurial, you can use Jenkins remote build trigger. Go to your job and select "Trigger build remotely" in the Build Triggers section. Here you have to provide a authentication token which can be any string. After you entered a token and saved the project, you can start a build by calling JENKINS_URL/job/iDempiere/build?token=TOKEN_NAME where you have to replace your jenkins url with the actual url and the token_name with the actual token you just created.

The next step is to go to your mercurial repository. Open the hgrc file which can be found in .hg/hgrc. If it is not there, create it. Now go to the hooks section or create it:

[hooks]
changegroup = curl www.evenos-consulting.de/jenkins/job/iDempiere/build?token=MYTOKEN

p2 Director

Once your Jenkins is running, you will probably search an easy way to install all the great software you just build. The p2 director is a great tool for installing and updating software from p2 repositories. In this section, i'll show you how you can use the director.

Install the director

First we have to install the director which can be found here or here. Go to the machine on which you want to install idempiere. I usually create a directory at /opt/idempiere-server. Chose a directory and unzip the director into this folder.

Install from p2 repository

Switch to the "director" directory. Now run the following command to install the server from your p2 repository:

./director -r http://www.evenos-consulting.de/idempiere/p2 -i org.adempiere.server.product -destination /opt/idempiere-server

The director will now install the server into the given destination.

After the server was installed, try installing the feature group which we also built:

./director -r http://www.evenos-consulting.de/idempiere/features/p2/ -destination /opt/idempiere-server -i org.evenos.idempiere.uifeature.feature.group

Notice that you will have to put ".feature.group" behind your features id.

When you are done installing the server and all of your features, run the console-setup to create the necessary properties. Finally start the server with the idempiere-server.sh:

nohup ./idempiere-server.sh &

Now use telnet to connect to the osgi console:

telnet localhost 12612

Type in "ss" to get a list of all the osgi bundles in the system. Search your own plugins and make sure they are active or at least resolved. If you have any org.adempiere.ui.zk fragments, you will probably need to refresh the zk bundle or the even the org.zkoss.library.

Update from p2 repository

Updating from a p2 repository can be a little bit tricky. What you basically want to do is to uninstall the bundles which you want to update and then simply install the latest version:

./director -r http://www.evenos-consulting.de/idempiere/p2 -u org.adempiere.server.product -destination /opt/idempiere-server
./director -r http://www.evenos-consulting.de/idempiere/p2 -i org.adempiere.server.product -destination /opt/idempiere-server

Before doing this, it might be a good idea to save your idempiere.ini because otherwise you may have to run the setup again.

mv /opt/idempiere-server/idempiere.ini /opt/
./director -r http://www.evenos-consulting.de/idempiere/p2 -u org.adempiere.server.product -destination /opt/idempiere-server
./director -r http://www.evenos-consulting.de/idempiere/p2 -i org.adempiere.server.product -destination /opt/idempiere-server
mv /opt/idempiere.ini /opt/idempiere-server/

Depending on your changes, it might be necessary to restart the server or at least some of its bundles. Use telnet to connect to the osgi console and the exit command to shutdown osgi:

telnet localhost 12612
osgi> exit

Now if you want to automate the update of the server you need a little workaround for starting/stopping the server. Basically you have two options. You can use the script $IDEMPIERE_HOME/utils/unix/idempiere_Debian.sh to start and stop the server like you would with every other unix programm. The second option is to use the command line tools "expect" and "tcl" and hack the following script to stop the server:

#!/usr/bin/expect -f
log_user 0
spawn telnet localhost 12612
send -- "exit\n"
expect "$ "
send -- "exit\r"

Put the code above in a script inside your idempiere server folder (e. g. "stopServer"). You can now start idempiere using

nohup ./idempiere-server &

and stop it using:

./stopServer

An all together update script could look like this:

cd $IDEMPIERE_HOME
./stopServer
mv /opt/idempiere-server/idempiere.ini /opt/
cd director/
./director -r http://www.evenos-consulting.de/idempiere/p2 -u org.adempiere.server.product -destination /opt/idempiere-server
./director -r http://www.evenos-consulting.de/idempiere/p2 -i org.adempiere.server.product -destination /opt/idempiere-server
mv /opt/idempiere.ini /opt/idempiere-server/
cd $IDEMPIERE_HOME
nohup ./idempiere-server.sh &

Sample implement

https://bitbucket.org/idplugin/za.spsi.idempiere.pluginbuild.sdk-feature