Plugin: CSV Importer

Maintainer: Diego Ruiz - Thomas Bayen - BXService GmbH, Krefeld
Status: Beta - Unsupported
License: GPLv2
Sources: GitHub

**This plugin is no longer maintained by the authors. Many things have improved and changed in the Core CSV process since its creation, use it at your own risk.**

Description

This plugin improves the normal CSV Importer process in iDempiere.

You can see the test results in the improvement section

Note: This plugin has a time improvement only for postgresql database instances

If you want to provide additional comments, please use the Discussion page

Configuration

After installing the plugin login as System and go to the Report/Process window, go to the Import CSV Process record and change the classname for "de.bxservice.process.ImportCSVProcess" to run the new class.

After you do that, you can configure the Import CSV Process as usual, in the Import Template window you will see a new field (Asynchronous Commit)

In the cases you want to preserve the normal iDempiere way of doing it don't check this field, if you want the import to run faster check it and run the process.

Improvement

The plugin was tested with several files, the first one a file with 5000 entries, and the behavior of the asynchronous commits was 30% faster.

In the tables below you can see the result after applying a csv file with 103.222 records. The first table shows the behavior in the normal iDempiere way, the second one the behavior after applying the plugin.

IDEMPIERE STANDARD IMPORTER

BX SERVICE IMPORTER

As you can see in the results the second process took 16 minutes 48 seconds less time committing, that means a 22.16% faster than the standard way.

Warning about Asynchronous Commit

As explained in the PostgreSQL documentation

Asynchronous commit introduces the risk of data loss. There is a short time window between the report of transaction completion to the client and the time that the transaction is truly committed (that is, it is guaranteed not to be lost if the server crashes). Thus asynchronous commit should not be used if the client will take external actions relying on the assumption that the transaction will be remembered. As an example, a bank would certainly not use asynchronous commit for a transaction recording an ATM's dispensing of cash. But in many scenarios, such as event logging, there is no need for a strong guarantee of this kind.

JIRA Reference

IDEMPIERE-2630