Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
Hi, Below are the steps. Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system. Single Big File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. Cat all files present inside testdata directory in to on single file by using following command: cat * > ../merged_data Load this merged file in to database using script single_import.sql from mclient Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon. Multiple small File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. 3. Load files in to database using multiple_import.sql using mclient Regards, Ashish From: Anthony Damico <ajdamico@gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Date: Monday, 17 November 2014 3:51 pm To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release please provide a list of every command that you executed to reproduce the problem. you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!! On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Thanks for looking in to it, I have uploaded required test files to you at below link: https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu<https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu> Regards, Ashish From: Anthony Damico <ajdamico@gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Date: Saturday, 15 November 2014 2:40 pm To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducibl... On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Additional information here, If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch. Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone! Regards, Ashish From: Ashish Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> Date: Friday, 14 November 2014 7:41 pm To: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release Is any body else seeing this problem with October,2014 release? I wish developer team can provide some insights in this area. Regards, Ashish From: Ashish Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> Date: Thursday, 13 November 2014 11:22 pm To: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release All, With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments. On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup. On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection. Can you guys help me with pointers on this issue, I will be happy to share any debug info. Regards, Ashish Kumar Singh Mobile: +91-9972045095<tel:%2B91-9972045095> Skype: toashishk@gmail.com<mailto:toashishk@gmail.com> _______________________________________________ users-list mailing list users-list@monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list
Hi all, Are you guys able to reproduce this issue? Any help required from my side do let me know. Regards, Ashish Sent from my iPad On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Hi, Below are the steps. Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system. Single Big File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. Cat all files present inside testdata directory in to on single file by using following command: cat * > ../merged_data Load this merged file in to database using script single_import.sql from mclient Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon. Multiple small File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. 3. Load files in to database using multiple_import.sql using mclient Regards, Ashish From: Anthony Damico <ajdamico@gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Date: Monday, 17 November 2014 3:51 pm To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release please provide a list of every command that you executed to reproduce the problem. you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!! On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Thanks for looking in to it, I have uploaded required test files to you at below link: https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu<https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu> Regards, Ashish From: Anthony Damico <ajdamico@gmail.com<mailto:ajdamico@gmail.com>> Reply-To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Date: Saturday, 15 November 2014 2:40 pm To: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Cc: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducibl... On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> wrote: Additional information here, If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch. Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone! Regards, Ashish From: Ashish Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> Date: Friday, 14 November 2014 7:41 pm To: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release Is any body else seeing this problem with October,2014 release? I wish developer team can provide some insights in this area. Regards, Ashish From: Ashish Singh <ashishkumar.singh@altair.com<mailto:ashishkumar.singh@altair.com>> Date: Thursday, 13 November 2014 11:22 pm To: "developers-list@monetdb.org<mailto:developers-list@monetdb.org>" <developers-list@monetdb.org<mailto:developers-list@monetdb.org>> Cc: "users-list@monetdb.org<mailto:users-list@monetdb.org>" <users-list@monetdb.org<mailto:users-list@monetdb.org>> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release All, With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments. On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup. On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection. Can you guys help me with pointers on this issue, I will be happy to share any debug info. Regards, Ashish Kumar Singh Mobile: +91-9972045095<tel:%2B91-9972045095> Skype: toashishk@gmail.com<mailto:toashishk@gmail.com> _______________________________________________ users-list mailing list users-list@monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org<mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list
Hi Ashish, please bear with us! We highly appreciate your feedback and will have a closer look at the reported problem as soon as our time and other obligations allow so. Please be aware that the members of this mailing-list provide free-of-charge voluntary spare-time best-effort services. For commercial support, please also consider https://www.monetdbsolutions.com/ Thanks! Stefan ----- Original Message -----
Hi all,
Are you guys able to reproduce this issue? Any help required from my side do let me know.
Regards, Ashish
Sent from my iPad
On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh < ashishkumar.singh@altair.com
wrote:
Hi,
Below are the steps.
Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system.
Single Big File Import:
1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql.
Cat all files present inside testdata directory in to on single file by using following command:
cat * > ../merged_data
Load this merged file in to database using script single_import.sql from mclient
Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.
Multiple small File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. 3. Load files in to database using multiple_import.sql using mclient
Regards,
Ashish
From: Anthony Damico < ajdamico@gmail.com > Reply-To: " users-list@monetdb.org " < users-list@monetdb.org > Date: Monday, 17 November 2014 3:51 pm To: " users-list@monetdb.org " < users-list@monetdb.org > Cc: " developers-list@monetdb.org " < developers-list@monetdb.org > Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
please provide a list of every command that you executed to reproduce the problem. you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!!
On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh < ashishkumar.singh@altair.com > wrote:
Thanks for looking in to it,
I have uploaded required test files to you at below link:
https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu
Regards, Ashish
From: Anthony Damico < ajdamico@gmail.com > Reply-To: " users-list@monetdb.org " < users-list@monetdb.org > Date: Saturday, 15 November 2014 2:40 pm To: " users-list@monetdb.org " < users-list@monetdb.org > Cc: " developers-list@monetdb.org " < developers-list@monetdb.org >
Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks
http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducibl...
On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh < ashishkumar.singh@altair.com > wrote:
Additional information here,
If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch.
Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone!
Regards, Ashish
From: Ashish Singh < ashishkumar.singh@altair.com > Date: Friday, 14 November 2014 7:41 pm To: " developers-list@monetdb.org " < developers-list@monetdb.org > Cc: " users-list@monetdb.org " < users-list@monetdb.org > Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
Is any body else seeing this problem with October,2014 release?
I wish developer team can provide some insights in this area.
Regards, Ashish
From: Ashish Singh < ashishkumar.singh@altair.com > Date: Thursday, 13 November 2014 11:22 pm To: " developers-list@monetdb.org " < developers-list@monetdb.org > Cc: " users-list@monetdb.org " < users-list@monetdb.org > Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
All,
With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments.
On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup.
On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection.
Can you guys help me with pointers on this issue, I will be happy to share any debug info.
Regards, Ashish Kumar Singh Mobile: +91-9972045095 Skype: toashishk@gmail.com
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Sure, take your time! We have older release of monetdb. Whenever you guys plan to have a look do let me know i can help in terms providing a test environment, machine etc.. For couple days if that helps. Sent from my iPad
On 18-Nov-2014, at 10:27 pm, Stefan Manegold <Stefan.Manegold@cwi.nl> wrote:
Hi Ashish,
please bear with us!
We highly appreciate your feedback and will have a closer look at the reported problem as soon as our time and other obligations allow so.
Please be aware that the members of this mailing-list provide free-of-charge voluntary spare-time best-effort services.
For commercial support, please also consider https://www.monetdbsolutions.com/
Thanks! Stefan
----- Original Message -----
Hi all,
Are you guys able to reproduce this issue? Any help required from my side do let me know.
Regards, Ashish
Sent from my iPad
On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh < ashishkumar.singh@altair.com
wrote:
Hi,
Below are the steps.
Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system.
Single Big File Import:
1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql.
Cat all files present inside testdata directory in to on single file by using following command:
cat * > ../merged_data
Load this merged file in to database using script single_import.sql from mclient
Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.
Multiple small File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. 3. Load files in to database using multiple_import.sql using mclient
Regards,
Ashish
From: Anthony Damico < ajdamico@gmail.com > Reply-To: " users-list@monetdb.org " < users-list@monetdb.org > Date: Monday, 17 November 2014 3:51 pm To: " users-list@monetdb.org " < users-list@monetdb.org > Cc: " developers-list@monetdb.org " < developers-list@monetdb.org > Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
please provide a list of every command that you executed to reproduce the problem. you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!!
On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh < ashishkumar.singh@altair.com > wrote:
Thanks for looking in to it,
I have uploaded required test files to you at below link:
https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu
Regards, Ashish
From: Anthony Damico < ajdamico@gmail.com > Reply-To: " users-list@monetdb.org " < users-list@monetdb.org > Date: Saturday, 15 November 2014 2:40 pm To: " users-list@monetdb.org " < users-list@monetdb.org > Cc: " developers-list@monetdb.org " < developers-list@monetdb.org >
Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks
http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducibl...
On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh < ashishkumar.singh@altair.com > wrote:
Additional information here,
If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch.
Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone!
Regards, Ashish
From: Ashish Singh < ashishkumar.singh@altair.com > Date: Friday, 14 November 2014 7:41 pm To: " developers-list@monetdb.org " < developers-list@monetdb.org > Cc: " users-list@monetdb.org " < users-list@monetdb.org > Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
Is any body else seeing this problem with October,2014 release?
I wish developer team can provide some insights in this area.
Regards, Ashish
From: Ashish Singh < ashishkumar.singh@altair.com > Date: Thursday, 13 November 2014 11:22 pm To: " developers-list@monetdb.org " < developers-list@monetdb.org > Cc: " users-list@monetdb.org " < users-list@monetdb.org > Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
All,
With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments.
On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup.
On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection.
Can you guys help me with pointers on this issue, I will be happy to share any debug info.
Regards, Ashish Kumar Singh Mobile: +91-9972045095 Skype: toashishk@gmail.com
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
Hey guys, Just double checking with you guys if there are any plans to see if reported behaviour is an issue, and how can i help? Regards, Ashish Sent from my iPad
On 18-Nov-2014, at 10:42 pm, Ashish Kumar Singh <ashishkumar.singh@altair.com> wrote:
Sure, take your time! We have older release of monetdb.
Whenever you guys plan to have a look do let me know i can help in terms providing a test environment, machine etc.. For couple days if that helps.
Sent from my iPad
On 18-Nov-2014, at 10:27 pm, Stefan Manegold <Stefan.Manegold@cwi.nl> wrote:
Hi Ashish,
please bear with us!
We highly appreciate your feedback and will have a closer look at the reported problem as soon as our time and other obligations allow so.
Please be aware that the members of this mailing-list provide free-of-charge voluntary spare-time best-effort services.
For commercial support, please also consider https://www.monetdbsolutions.com/
Thanks! Stefan
----- Original Message -----
Hi all,
Are you guys able to reproduce this issue? Any help required from my side do let me know.
Regards, Ashish
Sent from my iPad
On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh < ashishkumar.singh@altair.com
wrote:
Hi,
Below are the steps.
Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system.
Single Big File Import:
1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql.
Cat all files present inside testdata directory in to on single file by using following command:
cat * > ../merged_data
Load this merged file in to database using script single_import.sql from mclient
Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.
Multiple small File Import: 1. First create a database using Monetdb October2014 build called pbsworks 2. Create data base schema using schema provided in file schema.sql. 3. Load files in to database using multiple_import.sql using mclient
Regards,
Ashish
From: Anthony Damico < ajdamico@gmail.com > Reply-To: " users-list@monetdb.org " < users-list@monetdb.org > Date: Monday, 17 November 2014 3:51 pm To: " users-list@monetdb.org " < users-list@monetdb.org > Cc: " developers-list@monetdb.org " < developers-list@monetdb.org > Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
please provide a list of every command that you executed to reproduce the problem. you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!!
On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh < ashishkumar.singh@altair.com > wrote:
Thanks for looking in to it,
I have uploaded required test files to you at below link:
https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu
Regards, Ashish
From: Anthony Damico < ajdamico@gmail.com > Reply-To: " users-list@monetdb.org " < users-list@monetdb.org > Date: Saturday, 15 November 2014 2:40 pm To: " users-list@monetdb.org " < users-list@monetdb.org > Cc: " developers-list@monetdb.org " < developers-list@monetdb.org >
Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
could you provide a minimal reproducible example? the less code that re-creates the problem, the better. thanks
http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducibl...
On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh < ashishkumar.singh@altair.com > wrote:
Additional information here,
If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch.
Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone!
Regards, Ashish
From: Ashish Singh < ashishkumar.singh@altair.com > Date: Friday, 14 November 2014 7:41 pm To: " developers-list@monetdb.org " < developers-list@monetdb.org > Cc: " users-list@monetdb.org " < users-list@monetdb.org > Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
Is any body else seeing this problem with October,2014 release?
I wish developer team can provide some insights in this area.
Regards, Ashish
From: Ashish Singh < ashishkumar.singh@altair.com > Date: Thursday, 13 November 2014 11:22 pm To: " developers-list@monetdb.org " < developers-list@monetdb.org > Cc: " users-list@monetdb.org " < users-list@monetdb.org > Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release
All,
With October,2014 release we are stucked in to very strange performance issue while importing data on virtualised environments.
On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing ~ 2000 files as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm just to make sure its not an issue with a particular instance setup.
On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing copy into command through JDBC connection.
Can you guys help me with pointers on this issue, I will be happy to share any debug info.
Regards, Ashish Kumar Singh Mobile: +91-9972045095 Skype: toashishk@gmail.com
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
participants (2)
-
Ashish Kumar Singh -
Stefan Manegold