Hi Jos, It was not fast ... it was light speed ;-) It works very well. Thanks for you help. I understand your concern regarding 64 bits, but it's not the case. I have loaded 7million rows. After the load the DB has 2GB in total but the major file has 62MB, which is not a big deal if we compare to the text file with the data, which has 500MB. Saying that, you have any explanation why using jdbc results in slower performance after about 4million of loaded rows? Till there it was like 200 rows/sec and after that it was like 1 row/sec. Note: I have used batch update. A final comment. I have to confess that I am very impressed with the performance of MonetDB. My first contact with “Column-Oriented DB” was with Sybase IQ (in a presentation), but I don’t believe at the first place and now I have the opportunity to “touch the thing” ;-) and it was amazing. Congratulations. Best regards, Paulo Jorge Dias Jos van Dongen wrote:
Dear Paulo,
The fastest way to get large amounts of data into MonetDB is through a flat file and the use of the 'copy' command. This is even much faster than e.g. Oracle Bulk Loader. Note however that your 32bit environment will cause problems with large files (that's what I experienced). It's better to switch to 64 bit.
Met vriendelijke groet/kind regards,
Jos van Dongen Tholis Consulting T: 06-51169606 F: 0842-231136 M: jvdongen@tholis.com www.tholis.com
-----Original Message----- From: monetdb-users-bounces@lists.sourceforge.net [mailto:monetdb-users-bounces@lists.sourceforge.net] On Behalf Of Paulo Jorge Dias Sent: zaterdag 1 december 2007 20:26 To: monetdb-users@lists.sourceforge.net Subject: [MonetDB-users] Slow Data Load
Hi,
I am working on a Windows XP SP2 32bit with MonetDB v5.2.2 kernel v1.20.0
I have built a java program to upload data from an Oracle Database Table with 6milion rows.
After about 4milion of rows loaded the performance decreases dramatically.
I have tried to use other way, which is generating all insert statements to a file and then using mclient to execute all the insert statements.
Slower that the java option because the autocommit=on and parsing of each statement, but the result is the same.
After about 4milion of rows loaded the performance decreases dramatically.
Can someone give an explanation for this behavior?
Thanks,
Paulo Jorge Dias
-- View this message in context: http://www.nabble.com/Slow-Data-Load-tf4929292.html#a14108678 Sent from the monetdb-users mailing list archive at Nabble.com.
------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.16.12/1162 - Release Date: 30-11-2007 21:26
No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.16.12/1163 - Release Date: 1-12-2007 12:05
------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- View this message in context: http://www.nabble.com/Slow-Data-Load-tf4929292.html#a14121441 Sent from the monetdb-users mailing list archive at Nabble.com.