Difference between revisions of "Plugin: CSV Importer"

Latest revision as of 17:34, 22 April 2024

Maintainer: Diego Ruiz - Thomas Bayen - BXService GmbH, Krefeld
Status: Beta - Unsupported
License: GPLv2
Sources: GitHub

**This plugin is no longer maintained by the authors. Many things have improved and changed in the Core CSV process since its creation, use it at your own risk.**

Description

This plugin improves the normal CSV Importer process in iDempiere.

You can see the test results in the improvement section

Note: This plugin has a time improvement only for postgresql database instances

If you want to provide additional comments, please use the Discussion page

Configuration

After installing the plugin login as System and go to the Report/Process window, go to the Import CSV Process record and change the classname for "de.bxservice.process.ImportCSVProcess" to run the new class.

After you do that, you can configure the Import CSV Process as usual, in the Import Template window you will see a new field (Asynchronous Commit)

In the cases you want to preserve the normal iDempiere way of doing it don't check this field, if you want the import to run faster check it and run the process.

Improvement

The plugin was tested with several files, the first one a file with 5000 entries, and the behavior of the asynchronous commits was 30% faster.

In the tables below you can see the result after applying a csv file with 103.222 records. The first table shows the behavior in the normal iDempiere way, the second one the behavior after applying the plugin.

IDEMPIERE STANDARD IMPORTER

BX SERVICE IMPORTER

As you can see in the results the second process took 16 minutes 48 seconds less time committing, that means a 22.16% faster than the standard way.

Warning about Asynchronous Commit

As explained in the PostgreSQL documentation

Asynchronous commit introduces the risk of data loss. There is a short time window between the report of transaction completion to the client and the time that the transaction is truly committed (that is, it is guaranteed not to be lost if the server crashes). Thus asynchronous commit should not be used if the client will take external actions relying on the assumption that the transaction will be remembered. As an example, a bank would certainly not use asynchronous commit for a transaction recording an ATM's dispensing of cash. But in many scenarios, such as event logging, there is no need for a strong guarantee of this kind.

JIRA Reference

IDEMPIERE-2630

@@ Line 1: / Line 1: @@
 *'''Maintainer:''' [[User:Diego.ruiz|Diego Ruiz]] - [[User:TBayen|Thomas Bayen]] - BXService GmbH, Krefeld
-*'''Status:''' Beta
+*'''Status:''' Beta - Unsupported
 *'''License:''' [http://www.gnu.org/licenses/gpl-2.0.html GPLv2]
-*'''Sources:''' [https://bitbucket.org/tbayen_bxservice/de.bxservice.csvimporter/overview bitbucket]
+*'''Sources:''' [https://github.com/bxservice/de.bxservice.csvimporter GitHub]
+'''**This plugin is no longer maintained by the authors. Many things have improved and changed in the Core CSV process since its creation, use it at your own risk.**'''
 == Description ==
 This plugin improves the normal CSV Importer process in iDempiere.
-You can see the improvements in the [http://wiki.idempiere.org/w-en/index.php?title=Plugin:_CSV_Importer#Improvement improvement section]
+You can see the test results in the [http://wiki.idempiere.org/w-en/index.php?title=Plugin:_CSV_Importer#Improvement improvement section]
-'''Note: This plugin works only for postgresql database instances'''
-== Feedback ==
-Please rate this plugin:
-{{#w4grb_rate:Plugin:_CSV_Importer}}
-<w4grb_ratinglist pagevotes idpage="Plugin:_CSV_Importer" items="30"/>
+'''Note: This plugin has a time improvement only for postgresql database instances'''
 If you want to provide additional comments, please use the [[Talk:Plugin:_CSV_Importer|Discussion page]]
@@ Line 24: / Line 17: @@
 == Configuration ==
 After installing the plugin login as System and go to the Report/Process window, go to the Import CSV Process record and change the classname for "de.bxservice.process.ImportCSVProcess" to run the new class.
+[[Image:IMportPorcesscalssname.png|800px]]
 After you do that, you can configure the Import CSV Process as usual, in the Import Template window you will see a new field (Asynchronous Commit)
@@ Line 37: / Line 32: @@
 In the tables below you can see the result after applying a csv file with 103.222 records. The first table shows the behavior in the normal iDempiere way, the second one the behavior after applying the plugin.
-''IDEMPIERE STANDARD IMPORTER''
-[[Image:Idempeirecsvimporter.png]]
-''BX SERVICE IMPORTER''
+[[Image:Idempeirecsvimporter.png|IDEMPIERE STANDARD WAY]]    ''IDEMPIERE STANDARD IMPORTER''
-[[Image:Csvpluginresults.png]]
+[[Image:Csvpluginresults.png]]       ''BX SERVICE IMPORTER''
+As you can see in the results the second process took '''16 minutes 48 seconds''' less time committing, that means a '''22.16% faster''' than the standard way.
+== Warning about Asynchronous Commit ==
+As explained in the [https://www.postgresql.org/docs/9.6/wal-async-commit.html PostgreSQL documentation]
+<blockquote>
+Asynchronous commit introduces the risk of data loss. There is a short time window between the report of transaction completion to the client and the time that the transaction is truly committed (that is, it is guaranteed not to be lost if the server crashes). Thus asynchronous commit should not be used if the client will take external actions relying on the assumption that the transaction will be remembered. As an example, a bank would certainly not use asynchronous commit for a transaction recording an ATM's dispensing of cash. But in many scenarios, such as event logging, there is no need for a strong guarantee of this kind.
+</blockquote>
-As you can see in the results the second process took 16 minutes 48 seconds less time committing, that means a 22.16% faster than the standard way.
+== JIRA Reference ==
+[https://idempiere.atlassian.net/browse/IDEMPIERE-2630 IDEMPIERE-2630]
+==See Also==
+*[[Plugin:_RED1_NINJA]] that has automated conversion of attached CSV into a new model fully rendered in Menu tree with CSV content injected as data.
-[[Category:Available_Plugins]]
+[[Category:Deprecated_Plugins]]