Plugin: CSV Importer

From iDempiere en
Jump to: navigation, search

Description

This plugin improves the normal CSV Importer process in iDempiere.

You can see the test results in the improvement section

Note: This plugin has a time improvement only for postgresql database instances

Feedback

Please rate this plugin:

Current user rating: 100/100 (2 votes)

 You need to enable JavaScript to vote


Votes for <a href="/en/Plugin:_CSV_Importer" title="Plugin: CSV Importer">Plugin: CSV Importer</a> (showing up to 30 votes)
TimeRatingUser
January 30, 2017, 11:54 pm100%Red1
May 20, 2015, 11:00 am100%Diego.ruiz


If you want to provide additional comments, please use the Discussion page

Configuration

After installing the plugin login as System and go to the Report/Process window, go to the Import CSV Process record and change the classname for "de.bxservice.process.ImportCSVProcess" to run the new class.

IMportPorcesscalssname.png

After you do that, you can configure the Import CSV Process as usual, in the Import Template window you will see a new field (Asynchronous Commit)

NewField.png

In the cases you want to preserve the normal iDempiere way of doing it don't check this field, if you want the import to run faster check it and run the process.


Improvement

The plugin was tested with several files, the first one a file with 5000 entries, and the behavior of the asynchronous commits was 30% faster.

In the tables below you can see the result after applying a csv file with 103.222 records. The first table shows the behavior in the normal iDempiere way, the second one the behavior after applying the plugin.


IDEMPIERE STANDARD WAY IDEMPIERE STANDARD IMPORTER


Csvpluginresults.png BX SERVICE IMPORTER


As you can see in the results the second process took 16 minutes 48 seconds less time committing, that means a 22.16% faster than the standard way.

Warning about Asynchronous Commit

As explained in the PostgreSQL documentation

Asynchronous commit introduces the risk of data loss. There is a short time window between the report of transaction completion to the client and the time that the transaction is truly committed (that is, it is guaranteed not to be lost if the server crashes). Thus asynchronous commit should not be used if the client will take external actions relying on the assumption that the transaction will be remembered. As an example, a bank would certainly not use asynchronous commit for a transaction recording an ATM's dispensing of cash. But in many scenarios, such as event logging, there is no need for a strong guarantee of this kind.

JIRA Reference

IDEMPIERE-2630

See Also

  • Plugin:_RED1_NINJA that has automated conversion of attached CSV into a new model fully rendered in Menu tree with CSV content injected as data.