Decrease Idempiere Repository

THIS PAGE CONTAINS DOCUMENTATION ABOUT MOVE OF IDEMPIERE TO DECREASE THE SIZE OF REPOSITORY EXECUTED IN NOV-18/2015.

Overview

At the moment, repository of idempiere is very big about 1.5G in bitbucket,

and about 2.3G when update working folder to head. and 2,4G after materialize.

Just focus size in bitbucket. it's take many time for clone, with slow bandwidth, we must separate many time to get small parts.

Mercurial convert extension

• Accident I see a extension can help decrease a repository.

it can help total remove a file out repository (total mean remove in all history of repository)

and even delete old repository history.

• reference:

Document about convert extension

Guideline for re-upload to bitbucket

• What's cost.

1. Process is very fast, about 30 minute.
2. Every commit log is maintained.
3. Every repository clone from this repository must re-clone I think this's big issue when developer has many private patch.

you can overcome this limit as below.

do exact this process (decrease repository) on your repository.

re-upload it to bitbucket. now you can sync with new main idempiere repository

Practise with idempiere repository

I guess jar and zip file in idempiere repository is reason of big size, so I will try remove it.

1. I has pure repository clone from idempiere at [IDEMPIERE_ROOT] = /home/hieplq/dev/bitbucket/main-sync/idempiere-root
2. step 1: get list of all jar and zip file in repository in all commit.

open terminal at folder [IDEMPIERE_ROOT] and run below command

hg log -vMr "file('re:^.*\.((jar)|(zip))$')" --template "files:\n{files % '{file}\n'}" | grep -v 2Pack.zip | grep -E "^.*((jar$)|(zip$))" > map.txt

I achieve text file with each line is a relate path to jar or zip file (2Pack.zip is not in list)

uibase/org-netbeans-api-visual.jar
uibase/org-openide-util.jar
zkwebui/WEB-INF/lib/Filters.jar
zkwebui/WEB-INF/lib/asm.jar
zkwebui/WEB-INF/lib/bsh.jar
zkwebui/WEB-INF/lib/cglib.jar

1. step 2: copy map.txt to parent folder of [IDEMPIERE_HOME]
2. step 3: refine map.txt
   1. open map.txt and delete lines contain "org.adempiere.server-feature/data/seed/Adempiere"
   2. use find and replace of editor (i use eclipse) to add "exclude " to head of every line in map.txt (better use a linux command)

sed -r 's/.*/exclude &/' map.txt > newmap.txt

I has new file as below

exclude uibase/org-netbeans-api-visual.jar
exclude uibase/org-openide-util.jar
exclude zkwebui/WEB-INF/lib/Filters.jar
exclude zkwebui/WEB-INF/lib/asm.jar
exclude zkwebui/WEB-INF/lib/bsh.jar
exclude zkwebui/WEB-INF/lib/cglib.jar

1. step 4: run convert command

open terminal at parent folder of [IDEMPIERE_HOME] and run below command

hg convert --filemap map.txt /home/hieplq/dev/bitbucket/main-sync/idempiere-root /home/hieplq/dev/bitbucket/main-sync/idempiere-root-converted

waiting for 30 minute, i have new repository with all commit history but total remove unnecessary jar and zip file

1. step 4b:remove last commit of convert plug-in by run: hg rollback --verbose -f
2. step 5: setup to when materialize, buckminster auto collect jar file from other where (not apply but try success with a jasper library).
3. step 6: re-upload to bitbucket (not apply)
4. below is image compare size before and after remove.

• idempiere-root-converted is repository after remove file and update working folder to head
• idempiere-root-converted-clone is repository clone from idempiere-root-converted without update working folder
• idempiere-root is repository before remove file and update working folder to head
• idempiere-root-clone is repository clone from idempiere-root without update working folder

can confirm file is removed by rerun command at step 1 in folder idempiere-root-converted

Prepare the repository

Before we do that important transition we should clean up some things in the trunk and set some rules so that this issue will not happen again. I think we should avoid as much jar files as possible in the trunk. As of today we have files like jasperreports.jar or groovy-all.jar with more than 5MB of binary code. See IDEMPIERE-2683 for ideas how to improve that.

this process already apply to idempiere thru IDEMPIERE-2940