IDempiere workshop 2015/transcript

From iDempiere en

First day - Monday

Agenda

We looked into the agenda at IDempiere_workshop_2015 and talked about additional things:

  • Commercial
    • sales stoppers by areas
    • iDempiere distros
      • iDempiere light
    • Forums - functional - or open an iDempiere-technical forum
    • Collect information about implementation
      • plugin to collect anonymous information
  • Ux
    • Usability of webstore - embedding into iDempiere
      • JSF?
    • Usability of iDempiere - UI - User experience
  • Functional
    • Asset Accounting
    • surcharges and discounts conditions
      • pricing matrix
    • global vs. specific price list
    • BOM alternatives/optionals logic
    • External notifications SMS, mailers, etc
      • EMail-Marketing
    • Order document type hardcoded business logic
    • Workflows
    • Quick Info
    • LCO Global taxation
    • Alerts
    • System configurator organization
    • Vision about manufacturing
    • Average costing
    • Reactivating invoices, payments, bank
  • Dictionary
    • Adding fields to sales orders - issues about positioning
    • Every record must have a unique key
Yes, about 20 tables have no key; most important is product price. Others are access tables. We do not need it for storage, translations. We can create a JIRA ticket and contribute a patch
  • Technical
    • OSGi in general
    • Complex automated testing - jmeter, fitnesse, selenium
    • RESTful webservices - token
    • Dependencies of jasper libraries - jasper core cannot be replaced
    • Mobile UI - Synchronize
      • extending
    • Best practices for commnunity plugins
    • Events generic
    • Replication
  • Performance
    • AWS
    • Database scaling
    • Deployment Process - best practices
    • Memory consumption
      • opening multiple windows
    • Periodical restart


each record should have a primary key

Each Record should have a primary key: Primary keys are needed to audit changes in the database. That is needed for pricing. Chuck wants to create a JIRA ticket for that (IDEMPIERE-2811). Someone has to work on that. We dont think that we need primary keys for Storage (too much overhead) and not for translations. There are some more tables to think about.

Performance

speed performance

  • Chuck has the idea: Change the log4j output to be better parseable (creating a csv file).
  • Carlos idea: window and view collecting all audit tables (create a window to say "what happened in this time in the system") (JIRA).
  • We can add a log of SQL queries (while we are working in logging - thats not about performance but about debugging) (JIRA)
  • traffic splitting can be done with pfSense. That allows to route iDempiere traffice over another network connection than users using youtube.
  • Performance issues can be in network, apache/nginx proxy or in iDempiere. In Carlos experience in most cases problems in the user experience (complaints: "everything is running slow!") are outside of iDempiere. If iDempiere is the problme in most cases it belongs to bad code.
  • If your server runs 100% there are tools to create a java vm menory dump. It can be analysed in eclipse to see what is happening
  • You can run the server in debug mode but that hits performance (also when you are not using it). With debug mode Eclipse can connect to the running server and see which code the long running processes are executing
  • There is no way to stop a running process. That can be changed if we change code of the processes but that is not easy to do everywhere. Up to now we have no interface and no user interface for that. We can use a template about how to stop a running thread (JIRA)
  • Often the problem is the code (mostly self-written plugins): the best is to have short transactions
  • If you have a very long running process that uses a transaction you have two possibilities:
a) you can use a postgres command to increase the transaction timeout. That can make the whole system - especially for other concurrent users or processes - very slow
b) cut your transaction in smaller pieces. That is the advised solution: Keep your transactions as short as possible.
  • idea: schedule background job (JIRA)
  • idea: force background based on parameters (JIRA)
  • pgbadger is a tool to find out about postgres queries. It analyses the queries form the logs of postgres and helps to find out which queries take more or less time.
  • pgtune can help tuning the postgres configuration
  • You can solve some issues in complex postgres queries by adding indexes
  • If the cpu is not at 100% but the process is stucked you have to think about locks. In most cases locks can be solved by writing the code better.
    • examples are ad_sequence and m_storage. If you lock these in a transaction and use a long-running loop you create a bottleneck.
    • you can sometimes refactor your code to get document numbers at the end of the process
    • In pgAdmin you can see the postgres postmaster process id of a locked process. Carlos used a new column in AD_Changelog with a default value of "pg_backend_pid()". That hits performance but it helps to know which process belongs to which postgres postmasster process.
    • idea: document sequences admitting holes (IDEMPIERE-2817)
  • If a table has a cache it has a ".get(...)" method. That reads That will be faster in speed and use less memory.
  • Chuck to create (jira) ticket to make AD_Client_ID, CreatedBy, UpdatedBy to reference type of "Search" instead of "Table" or "Table Direct". The reason is to prevent the system from trying to deference in DB and store in memory values that will never be used in an read-only situation. Consider creating sql query to show all records where either (1) a table column does not have a corresponding field, (2) a table column only has fields that are flagged at read only or not displayed.

memory performance

  • a memory issue can arise if you try to use a table/direct field on a big table. This kind of field will read all records to create the pull down list. There is a hard coded limit (of about 200 or so). That creates a message in the log but the user will get not all entries but only a part of it.
    • using a search field instead of a table list increases performance a lot
    • if you use validations or dynamic validations only the resulting (shorter) list will take memory in the browser. Doing a validation might also helps solving memory issues.

bandwidth performance

We talked about that at the fourth day (see down).

Session related things

  • There is the idea to not allow the user open the very same record in a second window or to make the second window read-only (and write a message to the user like "you opened this record twice")
  • there is a plugin from Nicolas that a user can not log into a second time
  • idea: restrict number of open windows - in general (JIRA)

Periodical restart

ADempiere needed a lot of restarts to run stable. iDempiere and Java 7 improved that a lot. In principle you can restart the server once a day. An idea is to have a script that restarts the server only if there are no processes running (JIRA).

Better search index

We spoke about using Lucene (that is e.g. used by Solr) as a search index. Norbert's idea was to create a special type of search field that uses the Lucene index for search.

To use that search index you have to actualize the index after changes in the database. That does not have to be done synchronous but in another thread or such.

A search like "%key" can not be indexed in PostgreSQL nor Oracle. It creates a full-table search.

Another problem is that big tables need a lot of time to be shown if you forget to give a search key. This does not directly belong to the search index but to a better query. In MRole you can restrict the maximum numbers of records a query can get.

Database scaling

HAProxy is a proxy that can load balance the users to many iDempiere application servers. All servers use the same database server. (Chuck Boecking uses HAProxy for one and a half year. It is very well documented and for him it works very well.)

The database can be replicated by PostgreSQL to another server.

  • The mode of replication can be "immediate". That makes sure that all clients have the same data. That makes commits (according to Carlos' tests) like 3 times slower. A problem in Carlos test is that the replica server shows a transaction as committed when the data is in the walog but not yet in the table. That can break iDempiere processes.
  • In the mode "referred" you are not sure that the replicated server has the same data. That is not slower than without replication.

You can use "referred" for backups. You can loose two seconds of data without a performance penalty.

There is a program "pgpool". You can use it as a load balancer for PostgreSQL. Carlos tested it (with PostgreSQL 9.x). You create a postgres Master server and several read-only replicas. All queries go through the pgpool. All queries that change data or function calls that write have to be routed to the master server. Any select without a transaction can be solved in the replica (info windows, reports, no financial reports - they have a big performance hit).

A better idea than using PostgreSQL replication (based on the postgres logs) you can use pgpool replication. That replicates all the tests to all servers. That worked very good but the time was 5 times as long with two servers than with a single database.

A much better idea will be to separate the calls for read and write in the iDempiere code. Walking tree worked on that: http://blogs.walkingtree.in/2013/03/07/seperate-database-for-read-and-write-in-adempiere/ This is not yet in iDempiere (JIRA)

Chuck advises also pgbouncer. That allows to use much more application servers on a single database instance because you do not need memory for every connection. Without that every application server opens like 10 database connections and each of these start a postmaster thread in the postgres database that uses memory.

Deployment Process - best practices

Get updates for your installation from a p2 repository using the script update.sh included in the server installation.

Jenkins can be used by everyone - just ask for a user if need be.

Plugin installation in the server is better done using the console (putting the jar into the plugin directory) than using Felix because then you do not exactly know where the code is.

Staged deployment can be done on a "cascade" of servers, for example:

  • test
  • integration test
  • user acceptance test
  • production

Let the developers talk to each other to avoid e.g. the same column being created twice. Share changes using 2Pack.

You can use 2Pack to export almost any data and that way deploy e.g.

  • new price lists for a tenant
  • new windows to the system client

Be aware that you can not import a system 2Pack to a tenant.

2Pack has a feature to log UUID relations to record for example, that you imported data from a different tenant or installation and the data have new UUIDs in this installation or tenant. If you then import a second 2Pack containing updated data it would know the relation and would not create new entries but update the relevant entries.

2Pack can also be used to deliver "stuff" within a plugin which is then activated when the plugin is started. There are two different activators taking care of that. Refer to Developing Plug-Ins - 2Pack - Pack In/Out for more info.

Sometimes a 2Pack import fails during plugin start. You may then "force" the re-import of the 2Pack by editing the number in the packin window (e.g. from "1.0.0" to "0.1.0.0") to make the activator believe it is seeing a new 2Pack during plugin start and import it again. As the 2Pack is also made an attachment to the record you may also just try to start the import process from that entry.

Experience is that currently financial report setup is better exported using csv instead of 2Pack.

A 2Pack can be easily created from any page using the export button and selecting the "zip" format. Such an export of data can be very helpful to developers when users report a (supposed) bug.

AWS

Carlos thinks that Amazon services are quite expensive at the long run. He tried to use it and the servers were like 3 times slower than a dedicated server.

The application server and database server should go to different servers for any system that is not very small. That helps finding bottlenecks and allows better scaling. It is easy to use several application servers. Some users recommend to use like 20 users per instance of the iDempiere application server. This kind of server needs about 8GB of RAM.

Chuck Boecking and Norbert Bede have more experience with Amazon services and like how they scale.

stop processes

It is an idea to create a way to stop a process (JIRA). That means we need a UI for that (a stop button). We have to know what happens if the user closes the window and/or the session gets lost. We need a template how to write a process that can be stopped. You have to use the best practices of Java regarding to the isInterrupted method that allows to end a process that is inside a loop. We need also a way to stop long-running processes outside of iDempiere code. For JasperReport there is a way to use the MaxPagesGovernor interface to do that. And we need a way to stop a running PostgreSQL query (from another thread).


Second Day - Tuesday

At the beginning we discussed about the Agenda for today. We want to talk about OSGi in general and about the pricing system and new ways to use it to enhance the way that prices are calculated.

Configuring iDempiere during startup

You can search the code in Eclipse for "System.getProperties". That shows some interesting possibilities to configure iDempiere during startup.

PostgreSQL connection parameters

The following runtime parameter allows to set additional parameters to the postgres connections. 

-Dorg.idempiere.postgresql.URLParameters="defaultRowFetchSitze=1000"

debug SQL queries

There is a new parameter to see all SQL database queries that are used by iDempiere. You can not change that value in a running server. That might be an improvement (JIRA). 

-Dorg.idempiere.db.postgresql.debug=true

connection pool

The connection pool that iDempiere uses is provided by c3p0.

You can change c3p0 pool parameters like the number of connections in the file PostgresSQL/pool.properties


connection leaks

Yesterday while considering performance problems and locks we forgot to talk about connection leaks. Now while we talk about the number of connections we do that.

You always have to close a connection. You always have to close a ResultSet and a Connection object in a finally block. That can happen in a hard to find way if you reuse a variable e.g. for a statement (and not close the former object). If you don't use finally blocks and close the connection that the connection will stay open in the PostgreSQL server. That is a quite expensive resource to leave open. You can see that the postgres server has more and more connections open. It will close after the connection timeout of postgres.

Carlos said that there were many close commands without a finally block in the ADempiere code. Today all of them should be done in iDempiere but you can not be sure about plugins.

Best practices are to use the DB class "DB.get(...)" or the Query class.

best practice tags

Chuck Boeckings idea is to create "BEST PRACTICE" tags in the code to mark best practices in the code. (JIRA)

Diego Ruiz did a list of best practices in the wiki (Contributing_to_Trunk).


Pricing System

How does it work, what is part of it

These tables are used in the pricing system:

  • M_PriceList
  • M_PriceListVersion
  • M_ProductPrice
  • M_ProductPriceVendorBreak (it works not only for vendors since Deepak added that it works for customers too)

The price lists are set in BP and BPG. It is used (and can be changed) in Order and Invoice.

There is also a DiscountSchema (with SChemaLine and SchemaBreak).

The end date of a price list version is the date of the next version (there is no end date in the record).

The Discount Schema Break can be used to calculate a discount that changes the price list. You do not have to copy the price lists to use that. It is calculated when the price is used.

There is a contribution from Adaxa (Dirk applied it some time ago but it does not work any more) to have a uom field in the price list.

Another thing that can change prices are the promotions.

Norbert Bede has a contribution that allows to have a global price list and have a bp specific price list that links to a global price list for baseline prices. (JIRA)


change the pricing system

If you want to change the way that the prices works there is only one class to change: MProductPricing. But there are also three sql functions and some views that use the prices. These functions compose the price of a BOM if there is no price set. It may be a compromise to use the advanced functionality only if there is no bom or there is always a price for boms set.

  • bompricelimit
  • bompricelist
  • bompricestd

These functions have to be changed if you use a new price system. Or you can

  • the view m_product_substituderelated_v is shown on the InfoWindow

A better approach might be to use a callout to override how a price is calculated.

The database functions show the base price. If we want to add dynamic conditions we have to do that in java code. The dynamic price can not be used in views and reports or in the InfoWindow.

Shaun Melville has a use case where you need a matrix to find an entry in a "fee" table that is used to calculate the price. That is a use case where you need a lookup in a matrix based on some fixed sql code. It is not dynamic but can not be done with standard iDempiere pricing.

Anton Fildan showed us how SAP does it: http://help.sap.com/saphelp_46c/helpdata/en/dd/56168b545a11d1a7020000e829fd11/content.htm?frameset=/en/dd/56168b545a11d1a7020000e829fd11/frameset.htm&current_toc=/en/de/7a8534c960a134e10000009b38f83b/plain.htm&node_id=4&show_children=true#jump4

Thomas Bayen showed his concept of a single table with all conditions (prices, costs, calculations). It is similar to the sap approach. Thomas and Anton want to work on that at Prices and Conditions (JIRA)

Improvements can be done in the price list table to make it possible to use the uom (unit of measure) and ASI (attribute set instances) in the price list matrix. This gives us a matrix to create the base prices for InfoWindows, reports, etc.

The second step is the dynamic part of the pricing. It is done iwth the DiscountSchema based on Product Category, Product, Qty. You can think about extending that too.

If we want to change the MProductPricing class it needs to become an Interface that can be exchanged by OSGi. Some of the information that we want to use can only be accessed by the order line (or invoice lines). We can use Attribute Set Instances (asi) for that (and fill them with a callout). But that may lead to problems if you exchange the product or not save the order line. Or we can extend the class to get the context (and window and tab number) of the window. That looks like the best approach.

Summary what can be done with the pricing system

  • adding uom to product price (there is a patch from Adaxa/Dirk Niemeyer - look for Price UoM extension)
  • adding ASI (attribute set instance) to product price
  • adding uom/ASI on the discount schema break table
  • discount schema break table not only by percentage but also amount. That can be overriding or add amount and a field can say if it stops the calculation or continues.
  • If we can make M_ProductPricing an OSGi interface we can do everything in the most flexible way. It has to be extensible (e.g. by overloading) and needs some more constructor parameters like the Context.

Promotions

Promotions are also a way to change prices. You can look at this page for some documentation and that page to read how to enable it: http://www.adempiere.com/Enabling_Promotions. You activate it by adding a model validator class in the client window.

First you define a promotion group. With that you can group products like coca cola and coca cola zero for example.

Then you define a Promotion. You can set an campaign that belongs to accounting.

  • Pre Conditions tab: The promotion is applied in several conditions. You can set different dimensions like a bp, a bp group, a price list or such. You can set a limit that a promotion is given only 1000 times.
  • Promotion Line tab:
  • Quantity distribution: You can say sth like: If the qty is >= 5 add one more, or add one for every 5.
  • Reward: What do you get as an reward

Deepak has a patch on jira to improve. Until now if you buy 6 you pay 5, the patch makes it possible that if you buy 5 one is added automatically for free.

Some people know that Adaxa has a good documentation about that and how to configure it. You have to ask Steven Sackett for more information.

At the end only one promotion per order line is used.

Chuck asked if we can pull the promotions out of the core as a plugin. He thinks that is easier to work with for private extensions of the promotion model. (JIRA)

If you enter a sales order the promotion is applied when the order is prepared (or even completed) with the DocAction button.

You can use a promotion code. You write it in the Precondition table and you use it in the order line to apply the promotion.

deactivate hazelcast cache

You can deactivate the hazelcast cache by deactivating the OSGi plugin org.idempiere.hazelcast.service in the osgi console (use "ss hazel" to find the number of the plugin in your running installation). When you start an hazelcast instance from inside eclipse there is no good configured hazelcast.xml file. That makes it broadcast a lot and creates wrong cache values and much traffic in the network - especially if there are several instances running in the lan. If you start iDempiere from the 3.x installation package there should be an prepared hazelcast.xml file that uses a hazelcast group to avoid connecting with other servers. You can even stop the broadcasting feature by deactivating broadcasts in the hazelcast.xml file.


OSGi

In ADempiere we had the "classpath hell". The core, patches, extensions and customizations etc. were all in one single classpath namespace. That made it very hard to care about the versions of all the jarfiles that are used from different parts of the code. OSGi solves that problem by separating namespaces for classpaths into different plugins. In OSGi talk plugins are called "bundles".

The different plugins allow to separate and isolate different things. You know who is responsible for which part of the code.

OSGi interfaces work by using implementations that are done as an OSGi interface and use Factory classes to find them through the OSGi layer. There is an entry "service ranking" for every implementation.


Costing

Warning - :-) I did not really understand everything of that while doing the transcript. Beware of errors. And please(!) correct it if you have more knowledge about costing. Thanks!

standard costing

The most simple costing is "Standard costing". It is used without further configuration.

average costing

We talk about "average costing". It does work if you follow certain rules.

You have a "Purchase Order", a "Material Receipt", an "Invoice Vendor" (the last two are connected through a "Match Invoice"). This Match Invoice updates the Product Cost.

For average costs you have to know about some restrictions. Carlos does not really know about them (as a list or such), so you have to try it out for your own business case. For example it is better to not create a Material Receipt before the Invoice is there and you can immediately match them. You should not reverse documents that are used but reverse accrual them.


Deepak showed us an interesting case as an example. If you receive a Material Receipt of 100 and our cost are 10€ at the beginning this will be a stock of 100x10€=1000€. If you take away 40 (60 left) there is a cost for that of 400€. Now we get the vendor invoice and it gives another price of 12€ (1200€). iDempiere creates a new cost entry of 13.33€. That leads to a balanced accounting. (Without average costing the difference is posted to the price variant account).

We talked about three different cases:

  • If we have a stock and the po matches the invoice: No accounting on inventory and cogs
  • If po and invoice do not match and we have a stock: iDempiere posts on the Inventory account
  • if po and invoice do not match and we have no stock: iDempiere posts to the COGS account

thoughts about the future

Carlos idea is to have an OSGi interface for costing that allows to exchange it. The MCost class has to be refactored and it should be easier to do that in a plugin. (IDEMPIERE-2827) The average costing has some issues that depend on the exact business case and it can make more sense to create implementations for specific cases.

Norbert Bede did a lot of changes to the average costing implementation on his internal fork and Deepak too. They want to contribute that but it will be much work to do that.

Chuck advises to set the default costing method to be "Average PO" (IDEMPIERE-2828).

Carlos advises to create some asserts. For example you should not create a Material Receipt before you have an PO when you are using average costing.

summary about costing

  • Refactor MCost
  • Make a CostFactory for plugins to make cost calculation plug-able
  • Average PO becomes Average costing
  • Average Invoice is to be inactivated
  • periodic costing can be done as a plugin (Norbert implemented it as a flag in the accounting schema). That can be easier to use.
  • we need use cases and test cases (e.g. with fitnesse) to check the new implementation (landed cost, estimated landed cost, and many other cases have to be tested)
  • recalculate cost table (cut the data at a certain point in time and recalculate starting from that point)
  • re-posting has to be considered


Third day - Wednesday

overview of automatic testing in iDempiere

"Fit" (framework for integration tests) is a project that helps with automatic testing. An addition to that is "Fitnesse". That allows to create wikipages to define software tests that are applied using the Fit framework.

Selenium is a tool for user interface testing. That can be integrated with fitnesse but it is more complex to set it up and make it work with iDempiere.

Some JUnit tests are in the core but not very many.

JMeter can be used for load testing. Deepak has a tool to use that with iDempiere.

Chuck has also a process and a script for automated testing.

JMeter

JMeter is a tool for load testing. It allows to record HTTP traffic between the browser and the server. Then you can send this requests (changed e.g. by randomized data) with a lot of simultaneous requests to test the system behavior under heavy load.

Deepak Pansheriya has some experience with it and shows us how to use it.

JMeter simulates all the traffic between the browser and the server.

  • start JMeter GUI
  • create a "Test Plan"
    • add a thread group
    • add http request
    • add recording controller
  • at "Workbench" configure at workbench add a HTTP proxy server
    • choose a port (for example 8181)
    • include pattern like ".*/zkau.*" and ".*\zul"
    • exclude pattern like "./zkau/comet.*" and many more. That has to be documented better. Deepak wants to provide a sample JMeter project (IDEMPIERE-2829)
    • the proxy server has to be started with the "start" button. Now you configure your chrome or firefox browser to use the JMeter proxy.

Normally zk uses changing random ids for the components on the screen. For testing you use the Eclipse launch configuration "server.product.functionaltest". That includes additional plugins for testing that make sure that same ids are created for webpages. This is done in the class AdempiereIdGenerator(?). You can see that in the HTML Inspector of your browser.

The recorded html queries include a desktop id (dtid). zk uses a desktop id with every request. For things like load testing you need different desktop ids for every session. For that Deepak creates a random id on the call of index.zul (the first request). Deepak uses a csv file to read the dtid, tdtid (temporary desktop id), tenant, role using a "CSV Data Set Config" in the Thread Group configuration of JMeter. This csv file is read one line for every connection that is made simultaneously to the server. When looking into the request you exchange the dtid entries with a variable call like "${dtid}".

If you look into the zk request you see a field with id "uuid". That is the generated id (or a random id if you use standard idempiere and not the fitnesse plugins).

You have to hang all requests at the Recording Controller to use the dtid variable instead of the fixed dtid.

Now you can use HTTP Request Defaults

In the "Thread Group" configuration you can set how many thread are started in which time and how threads are used for that.

In the "Summary Report" of the "Recording Controller" you can see statistics after the tests run. You start the test by using the toolbar button with the green triangle. Then you can see that samples (a sample is a single request) are created.

JMeter slows down if it has a lot of threads. To get real data you should use several JMeter machines with every machine using like 50 threads.

As an example: Deepak worked with a configuration to test 450 concurrent users working with iDempiere at the same time.

Deepak wants to give us a link to a small sample JMeter configuration.

iDempiere installation hardware requirements

(While talking about JMewter load testing we thought about the hardware requirements that you need for different loads)

Some examples about hardware requirements:

  • Deepak worked on a system with Trekglobal that could handle 450 concurrent users creating records and the like. We dont know the setup of this system.
  • Carlos worked on a system with 256MB of memory. iDempiere worked on that with one user. He did not try financial reports or such things.
  • The globalqss demo runs on 512MB (including the database). It allows financial reports and everything. It crashes from time to time but it works as a demo.
  • If the question is: "How big is a virtual system to try out iDempiere for first steps of newbies?" our answer is: 1GB (or more) and one core.

Automated tests using Chucks Boeckings way

Chuck Boecking did a series "Automated and Regression Testing with Fitnesse and Selenium" in his ERP Academy. (You can ask him for access to that.) Today he does not like that. It was more harder and less flexible than he expected.

Chuck now uses iDempiere processes to create test data, do business logic and then uses asserts to check the results. He has a class with a collection of all best practice code snippets e.g. to create a business partner, to create a product, etc.

He uses this class in a process to do all the tests he wants. He can create records, start processes, etc.

  • There is a ValueObject class ChueBoePopulateVO with all test data.
  • There is a ChuBoePolulate class with all best practices like createBP(...), createPayment(...), etc.
  • The class CreateReplenishData consists of code that is done one piece. There are several tests done in sequence.


Fitnesse

start Fitnesse server

Fitnesse is a framework to check functionals. There is a interface in the iDempiere server (a plugin that you can start) that allows the fitnesse server to access iDempiere from outside. There are several defined functions that you can user

In the Eclipse project there is a fitnesse project Launch configuration to launch the Fitnesse server. In the directory FitnesseRoot there are wiki pages that contain the tests.

You have to set a "String Substitution" of ADEMPIERE_WEB_PORT to the port of iDempiere inside Eclipse. That allows the fitnesse server to connect to the iDempiere server.

After starting the fitnesse server using the fitnessetest launch configuration you have access to fitnesse during the web interface. You need to run idempiere by launching not server.product but server.product.functionaltest. That starts two additional plugins.

given examples in the iDempiere trunk

In the server you find a good number of tests that are already prepared. In TestLoginGardenAdmin you can see how a test for a login works.

The next test we looked at was TestCreateBusinessPartner. That shows how to include the login test to do both things in a sequence.

TestCashPosOrder is a good example for a more complex order. It reads business partners and products, creates a POS order, completes it and asserts the grand total, balances, shipment, shipment lines, invoice, invoice line, payment, etc.

There is no command line tool or a jenkins plugin at the moment. We would like to have jenkins start the tests with every cycle. (JIRA)

An idea of Carlos is to write something to capture

FitRecorder

There is a class FitRecorder. You can write Fitnesse tests by using iDempiere with the zk interface. You can use it by adding the plugin for fitrecorder to the iDempiere server. It is a model validator with name org.idempiere.fitrecorder.FitRecorder (you have to enter that into the modelvalidator table).

It is documented at Fitnesse Recorder.

At the moment the FitnesseRecorder does not work. (IDEMPIERE-2831) (It may be that we did not follow the right procedure as documented at Fitnesse Recorder)

zk testing with selenium

Heng Sin extended fitnesse to work with selenium to do user interface tests. You need to install the selenium plugin on your firefox or chrome (or phantomjs) to use that. If you start it opens the browser and forces it to edit the website enering characters, clicks, etc.

JUnit testing

There is a plugin called org.adempiere.extend containing a number of JUnit tests.

To run them you need to create a file called test.properties defining the environment and setup to be used. There is a file testTemplate.properties in the plugin you can start from.

You can then right click in that file and select "Run as -> JUnit Test". More details can be found at the related ADempiere JUnit Test wiki page

plugin levels

Every plugin has a level in the run configuration of eclipse. The level is to sort the plugins during startup of the osgi container.

Carlos advises to use level 5 for your own plugins. That means that the iDempiere plugins (going up to level 4) are loaded and started before your plugin. You can even use level 6 if you have several plugins with dependencies.

translations

The best tool that Carlos knows about to work with translations is Altova. If you start with a new language that software helps a lot but it is not free and quite expensive. You can use a 30 days trial version.

For the later maintenance of the translation files Carlos uses vi, grep and such tools.

An idea of Thomas Bayen is to somehow use the csv importers. That allows to edit the data in a grid-like table view inside OpenOffice (and not in xml format).

The "centrally maintained" flag says that a name will be taken from the elements table if you start the "synchronize terminology" process.

When you change an Element value it is written to all records that use it by a afterSave ModelValidator.

It may help to have a popup menu on every label text in the user interface with a button to open the translation window.

Norbert Bede showed us an translation service at http://www.oneskyapp.com. It allows to translate in a social way. We want to try if that tool can help our community to maintain translations and let people share things.

Official translations can be found at http://wiki.idempiere.org/en/Translations. You can easy add your name to this page if you want to care about a translation.

The translations in the core are quite bad and outdated. They should be removed. (JIRA)

Distros

Carlos approach would be to create an installer that creates everything in one.

Jenkins has a script that installs the database seed (copied from the installer) and applies the migration scripts (copied from syncApplied).

You can create a plugin without code but only a 2pack file. That allows to apply your configuration using 2pack files while installing the system.


how to change the application dictionary

The numbers of fields are in a sequence with multiples of 10. The reason for this is to allow the users to add custom fields in between if needed with numbers non multiple of 10.

The tab editor destroy that because it renumbers the whole window to multiples of 10. That can be considered a bug. It can be changed to not destroy sequences dependent of the entity type. (IDEMPIERE-2833)

Jenkins

Jenkins allows it to create and test an own distro, migrate the database and create an installer zip file.


how to create and maintain database changes

Data can be exchanged as:

  • 2pack
  • migration scripts
  • csv import files

DB Solo is a commercial tool that allows to compare databases.

In trunk is a directory db/ddlutils. There is a tool to create xml files with schema and data. There are ant files to use it for the iDempiere database. That can be used to bring the whole database into a text format for using diff and version control.


stop processes

We talked about that issue at the first day. Diego Ruiz tried to do some code for a proof of concept to stop a process. He added a stop button to the user interface for a running process. That worked with an internal test process but not inside a plugin.

We found out the way how interrupts can be used in iDempiere. It seems that the SQL layer throws an InterruptedException when a process that runs a long running sql query after stopping the query on the database server.

If you have a long running piece of java code without SQL you have to check the interrupt yourself. You can decide where to place this code snippet so that it is inside a loop on a safe piece of code to be interrupted. 

 if (Thread.interrupted())
    throw new InterruptedException();

We need a way to mark processes as interruptible or not. One reason to do that is that we want only interrupt code that we checked and tested (independent from the fact that every idempiere process should be interruptible at any time). Another reason can be that sometimes we just don't want it.

We want to keep playing on that issue. (JIRA)


Fourth Day - Thursday

bandwidth performance

Carlos has no idea about the bandwidth requirement per user. If you use a cloud provider it makes sense to have a dedicated internet connection or a QoS (Quality of Service) router. Marco has experience with a setup where a user has a 2G connection (up to 384KBit) for three users and it works. That may be hard for reports and if you use attachments.

Chrome allows to measure bandwidth usage chrome://net-internals/#bandwidth

Collecting more information on that might be helpful.

Wiki organization

What can be improved?

  • editing is too complex (barrier)
  • policies
  • quality
  • absolescence
  • We have a lot of installation guides that are partly wrong, insecure and outdated

We talked about how to deal with different languages.

A policy should be to create a page in the English wiki for every interesting page in other languages. The page might only consist of a sentence "this thing is described in the xxx language wiki. You can look there or please translate it to here".

Norbert uses Confluence. It helps to connect jira tickets, wiki pages and to work with it in a social way. It has not the multi-language capabilities of mediawiki.

Anton's opinion is that the language translation are not good to use but Thomas said there is no better tool than MediaWiki for that.

iDempiere blog

Norbert's idea is to create a blog for iDempiere. (JIRA) Community members (perhaps only approved ones) can write things there. He advises to use Wordpress for that. The audience appreciated that. Norbert and Deepak are the blog committee.

I sorry if it's wrong when comment at here.

I see some project has a docbook document project besides.

it's good by some below reason:

+ export to more format for read offline.

+ nice layout. has editor as eclipse plug-in

+ tracking version like source code

example: jetty document project

result in html fomat

id fields

Shaun brought the question why the database uses integer ids and not the uuid field as the primary id.

We will not need any more centralized ids and such things. It will make replication, sharing and distribution of the database much easier.

We don't need sequences for ids any more (still for document numbers). We have to change all "order by" clauses to use the createdby and uuid fields to get a fixed order of records.

One way might be to use the uuid field as an alternate key (as an indexed unique not-null column). That might allow to use both fields as a key dependend on the usage. That does not work at the moment as iDempiere internally still uses the primary key from the numerical id.

One (probably better) idea is to change the type of the id columns from numeric to text. Then we can extend how ids are created.

We agree that it is a good idea to do that but it is a huge time.


Replication

We talked about replication and how it can work. There is a list of issues to be worked with at http://www.adempiere.com/Replication_HowTo

Norbert and Deepak shared some knowledge about replication.

Norbert Bede's idea was to export replicated data in different format to use the replication facility for several kinds of exports.

export templates

Thomas Bayen adds the idea to have a export format template for csv (additional to the import csv format). It should be possible to use the very same format for import and export. (JIRA IDEMPIERE-2103)

  • We need a new button to allow export based on templates. At the moment
  • We need an Export process that allows to start export based on template
  • the export process needs a filter
  • we want to add parameters. For that the template contains a context aware filter. That filter can use context variables set as parameters
    • Norbert's idea is that you can use a saved search as a filter (that idea applies not only to the export but also to reports and the like)
    • we agreed that we need both the context aware sql filter and the saved search filter
  • we need a log what is exported and when
  • we can add a storage provider to tell where to put the file. (that can be used also in the import for import and log files). Norbert said that he already has some code on that and will provide it. (JIRA)

saved searches

Norbert talked about his idea to use saved searches. (JIRA)

  • A search that one user uses can be made global or can be shared with several other users or roles.
  • This search can be used not only as a search in the data window but also in processes, reports, the exporter, ...

another point is to add relative search (last x days). (JIRA)

another point is to add "sql searches" to the list of the searches the user can choose of. That means I can a) do my own searches using the "find window", b) use searches from a power user that shared them with me and c) use very specific and powerful searches that the implementor did in sql (JIRA)

JPiere

Hideaki showed some of JPiere concepts in his iDempiere localisation and distribution. That can be a good example how to create an own (for example country specific) distro of iDempiere.

It consists of some parts. The so-called JPiere plugins (JPPS) can be used also by non-Japanese people.

multi row grids

There is a plugin that creates multi-row grids. A single record can have several lines. It also adds several header lines to the grid view.

matrix window

It is in the wiki documented as "window X2".

The matrix window allows to have a denormalized view of data. It is even editable. That looks like a great contribution for the community.

currency display

In japan you don't need the fraction of the currency to be shown (there is nothing like a cent in the Japanese currency). A plugin changes that. That can be useful for other countries that use currencies without a fractional.

Tax Rounding

JPiere has a tax provider that allows to set the way of tax rounding per business partner. I am not sure if this applies only to the Japaneses tax system).

tax calculation in bank statement

multi line reports

In reports you can use several lines. JPiere changed how the header of the report is printed.

show me the code

Hideaki's JPiere code can be found at bitbucket https://bitbucket.org/jpiere


Norberts ideas

Norbert showed us some of his ideas.

views

A good idea is to make it more configurable which view is shown in a window. That can be not only the grid view and the form view but also multi-line, Kanban, report, etc. (JIRA)

themes

Norbert showed us an improvement to deal with themes that allows to reload the theme in a running system.

replenishment

Norbert presents updates to the replenishment feature. Main driver is that the current solution is not interactive enough.

Info window for purchase replenishment

  • shows in the main tab all order lines which can currently not be fulfilled
  • in a tab you can see the purchase proposal and after which time you may need to have another purchase
  • you can then generate an order and if there is already an not completed order for that vendor you can add lines to that order
  • you may also choose a different vendor instead and add missing prices to that vendors price list from the products purchase tab

Other changes / enhancements:

  • The info window feature was extended to have "value preference" in the selection fields
  • The tab "Business Partner -> Location" has a new subtab "Order schedule" where you can put in weekdays and related info.
  • for easier mass update a separate window for the replenishment info has been created.

A new process calculates minimum and maximum replenishment level by standard way. It can be based on history (count of movements within a certain time frame). If you need your own specific calculation rules you can install and link to a specific class implementing that rule for the set of parameters currently defined on the window.

Experience:

  • configuration of displayed columns in info window would be nice (IDEMPIERE-2839).
  • also quick info based on columns of selected rows to show for example the total price in relation to a selected vendor (IDEMPIERE-2807).

TOC Buffer Management

Now when we have been talking about replenishment Carlos added a talk about Buffer Management.

"Theory of Constraints" is a methodology for manufacturing (and many other) processes. Carlos and Diego Ruiz did a plugin Plugin: TOC Buffer Management to allow the management of buffers to manage the bottlenecks that can be identified by the TOC. You use BOMs for your manufacturing needs and all products inside the BOM have some information about constraints and storage management to create an inventory buffer at specified levels.

The process can help with the replenishment of buffers in your manufacturing queue. The size of the buffer will tend to stabilize at an optimal level after you use it for a while.

The plugin does not count for seasonal changes in your processes. That may be an improvement in future.

There is a well done documentation in the plugins wikipage about the plugin.

view plugins

The idea from Norbert, Chuck and Thomas is to allow plugins to extend the grid and form view with many more ways to view data. It is interesting that Hideaki's X1 window (multi-line-view) works as a plugin.

webservices

Issues with webservices:

  • too many logins (bad performance)
  • RESTful access

tokens for login into webservices

Deepak added a token to authorize to the webservices. You login once, get back a token and this token allows to keep the session open in iDempiere and not authorize with every request. We create a new table to keep the tokens. It would be good if we can keep the whole session of the webservices request for the following requests. Deepak did not yet finish his work. (IDEMPIERE-2840)

When a client logs in to the webservice it can use both ways of authentication: Give all credentials or use the token. You can even give both to automatically login again if the token expired.

Another idea is to create authentication tokens for customers access to webservices. They have to expire in a much longer (and configurable) time than the automatic tokens. (IDEMPIERE-2841)

additional ideas for login

Norbert and Thomas added the idea to make it possible to provide a direct link to specific reports. (IDEMPIERE-2842)

Thomas has the idea to allow a login with a token inside the url to login. (IDEMPIERE-2843)

login with less information

Norbert asks to have an easier login without the need to give an Organization and Warehouse (JIRA may already be created by Nicolas)

Login data of Organization, Warehouse, Date can be changed to be entered not in login but as preferences. (JIRA)

REST Interface for Swing

The REST interface allows to get a lot of information and possibly dangerous information. The /server/ URLs should not be active when you use an apache to filter access. This code is used by the Swing client and allows to retrieve some information, start processes etc. That can be used as an example to do things from outside like start a process to retrieve a report or such. You can define if that part of the server is started in org.adempiere.server/WEB-INF/web.xml.

That should be the blocked in trunk by changing web.xml or we should block some urls like services and command with a System Configurator like "PROVIDE_STATUS_FOR_SWING" or we extract it as a plugin (JIRA). It is not needed if you don't use the Swing client (and the swing client can even be configured manually without that).


Fifth Day - Friday

PostgreSQL bottlenecks

A question was how to find locks and the like that create bottlenecks in the database. These kind of locks can stop many users working for a remarkably amount of time.

Carlos showed us how to use the idempiere-Monitor (URL .../idempiereMonitor). There is an entry that shows the number of open connections and transactions. The transaction count comes from the Trx java class. If you want to know what the printed data is about you can look into Trx.getStatus() to know how it works and possibly to show more information. You can also try to setup debugging in Trx to find a problem.

In PgAdmin there is a entry in the context menu of a server connection an item "server status". That gives you clues about the open connections and transactions.

asset accounting

Anton asked if the fixed assets module can be used without big issues.

  • There is a document of Adaxa describing how it works.
  • There is also a document from Edwin Ang. That document is expanded by NTier.
  • Chuck did a lesson about the fixed assets and can confirm that it works well. But he sais that you can use only straight line depreciation (lineare Abschreibung). The other depreciation methods are not implemented.

Steven Sackett from Adaxa has additional code done for the fixed assets. Deepak wants to create a plugin of that if Steven agrees. (JIRA)

Order Document Type hardcoded business logic

Chuck want to do the changes of document status less hard-coded and more flexible. There are many things that better can be done in workflows instead of in hard-coded document types and sub-types.

Deepak explained the flexible way how jBPM works using for example Javascript code to make the workflow of documents very flexible. He uses jBMP Studio in Alfresco.

Chuck wants to overwrite the standard implementations of prepare and complete. Or he wants to configure a workflow in a system table that allows to change the way that workflow changes are done.

It is not so easy to create new document base types.

You can overwrite MOrder with a class like ChuckOrder. That class can overload MOrder and have a method like prepared. That method can be copied from the mother class and can be changed. A proposed core change can be to refactor the document action methods to build small blocks that you can use in your own prepare() method. That means that for your own changes you do not have to copy & paste the whole prepare()-method. That is bad style. (IDEMPIERE-2847)

As an example how configuring works Carlos created a new document type "POS purchase order" by setting the right subtypes and document types in the document type window.

It is also possible to use workflows to create your own states that can be used additional to the document action status.

We want a possibility to extract all business logic from model classes and move it to a plugin (IDEMPIERE-2848) The first step to do that is a refactoring of all document classes.

We want to replace all the calls to constructors of model classes by using MTable.get(...).getPO(...). (IDEMPIERE-2849) That allows to extend the model classes because getPO(...) uses OSGi plugins to decide which class will be created.

rules

Carlos showed the Rule window. There you can write script code without restarting the server. That code will be called as callout, model validator and the like.

Rules are a big security hole. Norbert wants to use that for power users. To allow that we can build our own language that runs in a sandbox.


limits for table size

Hideaki had the question how big a table can be. Carlos knows of systems with 10-30 million entries in the fact account table. There financial reports and such are still working well. You can use the Housekeeping feature of iDempiere to delete data in tables like the changelog from time to time.

Chuck advises for very big companies to copy the tables to a reporting environment. ON a separate server you can have your data on a dedicated database or you can create cubes for reporting etc.

The limit of the id column is the limit of the java integer type (2 billion records). No one has seen a database in iDempiere close to that. We can not say when the server starts to get performance issues.


budget automation

Deepak showed us how he enhanced the automatic budget. There is a plugin in our Marketplace. You can read his document at http://logilite.com/budget-journal-enhanced-budget-module


Alerts

An alert allows you to create a sql select to check things in the database. You can configure your sql code in the "Alert Rule" tab. 

In the iDempiere Monitor you see the processor for the alert. It can be started there manually.

In the Alert main tab you set a processor (that can be like "every 10 minutes"). and a recipient. The recipient is a user with a configured notification type (that can be notification and/or eMail).

You can use Alerts to

  • check problematic conditions in the database and get a notification if something unusual happens
  • send regular reports to users

The Alert is not started when the SELECT gives no rows.

If you get an EMail for the Alert the sql result set is attached to the mail. In a notice the sheet is attached to the notice record in iDempiere. That can be used to send regularly reports to users.


Witholding Taxes in LCO Global taxation

withholding taxes

The LCO plugin (maintained by Carlos Ruiz) is a good base for country-specific withholding taxes. Because taxation systems in all countries can be very different it may be appropriate to fork that plugin for your country. Carlos will not maintain plugins for all different countries.

detailed names

The LCO group of plugins also contains a plugin for detailed names. Thomas Bayen uses that and can confirm that you can use it in Germany to use detailed names. For that you have to configure some things. The main thing is to not use most functionality of the tax type of LCO.

magnetic media

The magnetic media plugin in LCO is to exchange data with the Colombian government. It is not useful in other countries.


background reports

We want scheduling of background reports in xls. (JIRA) That can be used for reference and audit reasons.

reactivating invoices

Thomas Bayen opened the Project field. After a change he has to re-post the document. That allows to change the accounting dimension after an invoice is closed.

If a document is printed and given to the customer you should not

In many real world installations you have a lot of reversal documents.

Nicolas Micoud did a "special editor" to change fields after complete. That is an plugin and available in Nicolas Bitbucket. It is not ready to use for all business cases but can be extended for your needs.

You should not change the price, quantity, discounts and other amount-relevant parts of an invoice. It affects things like the business partner balance, costs and such things.

Norbert advices that people who have experience with changing invoice values should document that into the wiki. Like "I changed business partners this way and it worked well".

context usage

Chucks idea is to document better how to use context. There are different ways to use context variables using different syntax formats. It would be nice if someone writes a wikipage to document how the context Evaluator works and where it is used. (JIRA for Chuck) The code is in the Evaluator class and in Env.parse*.

document management

Deepak likes to add a small document management to iDempiere. Other proposed to connect Alfresco or a similar tool. That can be used to give documents to customers. Many users want a simple solution and not install Alfresco. Dirk works on StorageProvicers for Alfresco. Norbert has a StorageProvider for Amazon S3.

Norbert showed a very interesting concept for a document management system inside iDempiere. He wants to store metadata as attributes. For bigger needs you can connect to storage providers like S3, Jackrabbit or CMIS.

Norbert and Deepak agreed to work further on that concept. (IDEMPIERE-2851)

Sergey shows us his integration with the OpenKM document management system.

iDempiere application example: AidODys / OpenKM integration

Sergey presents an iDempiere application case dedicated to support education for children with visual impairments.

Texts are converted from OpenOffice to HTML. In that a number of options can be selected to make the text more readable through adapting size, color of words, marking syllables and more.

iDempiere is used for its subscription capabilities and user profiles. Users can interact with a website using webservices in the background.

Document management capabilities are taken from OpenKM but the user interface library of OpenKM is replaced using zk.

The application consists of a number of additional plugins. Some architectural problems have been solved by using hibernate.

Sysconfig Keys

We have to check if all needed sysconfig keys are in the standard iDempiere system. All keys that can be changed on client level should be there. (JIRA)

We can do a form with right typed entry fields when we add field group, reference and reference key. That form can be easier filled by end users. After confirming the form the data is written into the system configurator.

changeable window views

Chuck comes back to the question if we can create different plug-gable views for windows, not only grid view and form view. There is already the sort tab as an example. Other ideas are reports, a webstore view, a multicolumn view, Hideaki's multiline window, etc.

There are views that share the grid data (grid view and form view do that) which performs well when changing views. But there are also views that do not use the grid data

We want to have a plug-gable view interface for windows. (JIRA)

Deepak told us to add more possible ideas to the list of needed plugin interfaces:

configuring of views in system configurator

The view m_inout_candidate_v is used much and changed quite often in own installations. These changes can be destroyed through migration scripts. It would be nice to allow to change this name's view with a System Configurator. (IDEMPIERE-2855)

Retrieved from "https://wiki.idempiere.org/w-en/index.php?title=IDempiere_workshop_2015/transcript&oldid=19818"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.
GNU-Lizenz für freie Dokumentation 1.3 oder höher
Powered by MediaWiki