- users-list - monetdb.org

[MonetDB-users] Change schema for already existing tables
by Travis Brady 22 Jan '08

22 Jan '08

Hello, Is it possible to change the schema for tables that have already been created and populated? I'd like to avoid having to fully recreate everything as the load process will likely take a few days at least. thank you, Travis Brady

5 5

[MonetDB-users] Problems with COPY INTO and CHAR(N) columns containing empty strings or all spaces
by Bob Ippolito 22 Jan '08

22 Jan '08

We've had problems with MonetDB SR5 using COPY INTO with data that may be the empty string or all spaces. We've been able to work around this by surrounding all of the values in our copy files with "quotes" (I read tablet.c to try and diagnose the issue and found that as a workaround). I'm not familiar enough with the source and build process yet to submit a patch. We are using '\t' as the separator and the fields that we've had problem with are CHAR(2) or CHAR(16) if that matters. -bob

2 3

[MonetDB-users] Optimal filesystem/platform choices for MonetDB?
by Bob Ippolito 22 Jan '08

22 Jan '08

We've started doing a big import of data into SR5 and I've noticed that it's creating *A LOT* of files. So far we've got about 620k files for 6.7 GB of bat. I expect that a full import should have at least 20 times what's imported so far. It seems that loading data is relatively slow compared to PostgreSQL (we're using COPY) and I'm guessing it's due to all of these files. Right now we're using ext3 on amd64 ubuntu linux. Should we be looking at other platforms? -bob

2 3

[MonetDB-users] Core dump on first connect by mclient in BUNinplace
by Bob Ippolito 22 Jan '08

22 Jan '08

So we've been testing out a big ETL and it crashed again, and now every time we start mserver5 it will seemingly start up ok but as soon as mclient connects it will crash. I've generated a core dump. The db farm is 3.6GB and the core dump is 15MB. I can send either to someone if they need it to look further into the issue. The platform is 2.6.20-15-generic (root@yellow) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #2 SMP on amd64. It's running an unmodified SR5 superball that we compiled ourselves. I tried looking at strace and the --debug option with all of the flags on but it didn't produce anything that looked very interesting. The backtrace is here: Core was generated by `//mochi/bin/mserver5 --dbname=mochiads_stats --dbinit=include sql; --dbfarm=/da'. Program terminated with signal 11, Segmentation fault. #0 0x00002b88a35bf285 in BUNinplace (b=0x12458e8, p=0x0, h=<value optimized out>, t=0x1246eb0, force=1 '\001') at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1555 1555 ATOMreplace(b->ttype, b->theap, BUNtloc(b, p), t); (gdb) bt #0 0x00002b88a35bf285 in BUNinplace (b=0x12458e8, p=0x0, h=<value optimized out>, t=0x1246eb0, force=1 '\001') at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1555 #1 0x00002b88a35bfb75 in void_inplace (b=0x12458e8, id=1, val=0x1245a20, force=1 '\001') at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1671 #2 0x00002b88a35bfc33 in void_replace_bat (b=0x12458e8, u=0x1246d08, force=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1688 #3 0x00002aaab26c9a5a in load_bat (sname=<value optimized out>, tname=<value optimized out>, bname=<value optimized out>, type=<value optimized out>, sz=<value optimized out>, cnt=0x1214400) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/bat/bat_storage.mx:364 #4 0x00002aaab26c9b83 in create_col (tr=<value optimized out>, c=0x1245878) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/bat/bat_storage.mx:456 #5 0x00002aaab26c44a9 in load_table (tr=0xd3dec8, s=0xd3df38, rid=149) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/store.mx:511 #6 0x00002aaab26c5523 in load_trans (tr=0xd3dec8, id=1128) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/store.mx:845 #7 0x00002aaab26c6ff4 in store_init (debug=<value optimized out>, store=<value optimized out>, logdir=<value optimized out>, dbname=<value optimized out>, stk=0) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/store.mx:1318 #8 0x00002aaab26aab79 in mvc_init (debug=0, store=store_bat, stk=0) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/server/sql_mvc.mx:246 #9 0x00002aaab2665c97 in SQLinit (c=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/backends/monet5/sql_scenario.mx:225 #10 0x00002b88a2c7d42a in initScenario (c=0x604980, s=0x2b88a2e934d0) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/mal/mal_scenario.mx:264 #11 0x00002b88a2c58d82 in MSscheduleClient (command=0xb0a1b8 "LIT", challenge=0x41000fd0 "KdhtXMmr29n", fin=0xb10270, fout=0xb03030) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/mal/mal_session.mx:342 #12 0x00002aaab0e78247 in doChallenge (cmd=<value optimized out>, in=<value optimized out>, out=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/modules/mal/mserver.mx:482 #13 0x00002aaab0e79036 in SERVERlistenThread (Sock=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/modules/mal/mserver.mx:564 #14 0x00002b88a43122a5 in start_thread () from /lib/libpthread.so.0 #15 0x00002b88a4ff261d in clone () from /lib/libc.so.6 #16 0x0000000000000000 in ?? () -bob

2 2

Re: [MonetDB-users] MonetDB/SQL 64-bit
by Martin Kersten 22 Jan '08

22 Jan '08

Stefan Manegold wrote: > Hi Peter, > > unfortunately, this still does not help us much --- at least not me (but I'm > everything but a Windows expert) ... > the 4 tutorial sessions on MonetDB website have been designed to uncover installation errors quickly and to get a first working session. Is tutorial 1 running? > I know, though, that on Windows both server & client need to be started via > the proper batch scripts (or by clicking on the respective icons) to ensure > evironment and paths are set correctly. > yes, the commands execute a script, which sets paths correctly. locate them to see what is required to properly run both. Especially, if you have manually moved folders around or changed permissions. > To help you any further, (sorry for repeating myself), we'd need to know > the exact error message, or at least the paths/names of DLLs that are not > found. > > Furthermoe, could you please continue mailing to neten(a)cwi.nl (as requested > monet(a)cwi.nl > by the "Rply-To:" in the mail header of bot my first and this reply; or even > better to Monetdb-users(a)lists.sourceforge.net)? --- thus, more people than > just me get to see your problem and can help answering and solving it... > > Stefan > > On Tue, Jan 22, 2008 at 12:51:19AM -0600, Peter Haluska wrote: > >> Hi Stefan, >> >> installation has worked. It looks like that it is having problems with >> system paths while trying to run server file(can not find dlls) or server >> batch file. Thanks. >> >> -Peter >> >> >> >> On Jan 21, 2008 6:24 PM, Stefan Manegold <Stefan.Manegold(a)cwi.nl> wrote: >> >> >>> Dear Peter, >>> >>> thank you very much for your interest in MonetDB! >>> >>> To be of any help for you, it is necessary for us to know exactly what >>> does >>> (not) work for you. >>> >>> Does it install? >>> Can you start a server? >>> Can you start a client and connect to the server? >>> Can you load data? >>> Can you run a query? >>> >>> Hence, please let us know exactly which steps you took to download, >>> install, >>> start, etc. MonetDB/SQL, where you get stuck, whether you get any error >>> message(s), and if so, which one(s)... >>> >>> Kind regards, >>> Stefan >>> >>> >>> On Mon, Jan 21, 2008 at 06:02:05PM -0600, Peter Haluska wrote: >>> >>>> Hi, we are testing your software MonetDB/SQL >>>> 64-bit< >>>> >>> http://monetdb.cwi.nl/projects/monetdb//download.php?target=http://prdownlo… >>> >>> > on >>> >>>> our new system 'Windows Server 2008 RC1 Enterprise Edition' with 64-bit >>>> support server and we can not make it to work. >>>> Any suggestions? Thanks. >>>> >>>> -Peter >>>> > >

1 0

[MonetDB-users] Unexpected results with COALESCE and aggregates
by Bob Ippolito 21 Jan '08

21 Jan '08

MonetDB SR5 behaves unexpectedly when we've tried to use COALESCE with aggregates, here's an example: sql>CREATE TABLE tmp(a int); sql>SELECT SUM(a) FROM tmp; +-------+ | sum_a | +=======+ | 0 | +-------+ sql>SELECT MIN(a) FROM tmp; +-------------+ | min_a | +=============+ | null | +-------------+ sql>SELECT COALESCE(MIN(a), 0) FROM tmp; sql> For the first SELECT I would expect NULL, the second SELECT is okay, but I expect a result for the third! Here's what PosgreSQL does: bob=# CREATE TABLE tmp(a int); CREATE TABLE bob=# SELECT SUM(A) FROM tmp; sum ----- (1 row) bob=# SELECT MIN(a) FROM tmp; min ----- (1 row) bob=# SELECT COALESCE(MIN(a), 0) FROM tmp; coalesce ---------- 0 -bob

2 2

[MonetDB-users] timestamptz issues with non-GMT
by Bob Ippolito 21 Jan '08

21 Jan '08

We've been playing around with MonetDB-Mars-SuperBall-SR5 for the past couple weeks and we've been having some issues with timestamps. We unfortunately don't do all of our calculations on GMT so we've hit a couple walls because it seems that the MonetDB 5 mtime module doesn't consider timezones: (Yes I know the compiler seems to optimize out the original type if you look at the EXPLAIN output, but this happens for timestamptz coming from tables too) sql>set time zone interval '-08:00' hour to minute; sql>SELECT CAST(TIMESTAMPTZ '2008-01-01 00:00+00:00' AS DATE); +--------------+ | single_value | +==============+ | 2008-01-01 | +--------------+ PostgreSQL has the expected behavior, which accounts for the local time zone when converting timestamps to dates. bob=# SET TIMEZONE TO 'PST8PST8'; SET bob=# SELECT '2008-01-01 00:00+00:00'::timestamptz::date; date ------------ 2007-12-31 (1 row) This is the best I've been able to do, but it's of course ugly and it seems like it could break if the implementation gets fixed: sql>SELECT CAST(TIMESTAMPTZ '2008-01-01 00:00+00' + CURRENT_TIMEZONE AS DATE); +----------------------+ | sql_add_single_value | +======================+ | 2007-12-31 | +----------------------+ Is the "best" solution to just use the GMT timezone (or the TIMESTAMP type) and hardcode our timezone offset everywhere? sql>SELECT CAST(TIMESTAMP '2008-01-01 00:00' - SEC_INTERVAL '28800' AS DATE); +----------------------+ | sql_sub_single_value | +======================+ | 2007-12-31 | +----------------------+ -bob

1 0

[MonetDB-users] "bpm_storage.h" missing when trying to install "sql" module
by xxiao 19 Jan '08

19 Jan '08

Hi, I have sucessfully installed the following modules of MonetDB monetdb version 1.21.0 monetdb-clients version 1.21.0 monetdb4 version 4.21.0 When I continue to install "sql" module, the "make" complains missing file "bpm_storage.h". The platform is Solaris10/SPARC. Anyone here has a hint what has gone wrong? Thank you so much for your attention. Xiang -- View this message in context: http://www.nabble.com/%22bpm_storage.h%22--missing-when-trying-to-install-%… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] mserver5 question
by Claude Monet 18 Jan '08

18 Jan '08

OS: CentOS 4.6 I am only able to initiate mserver5 as su. What should I do to be able to start the server in a user account? Thanks

2 4

Re: [MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 18 Jan '08

18 Jan '08

I thought I would try and ask these questions again, since there was no response. - How does how MonetDB handles its virtual memory? Does it swap out entire BATs at a time? Or just smaller blocks? What type of caching algorithms does it use? - If I configure a large swap, MonetDB is not using it. I guess it allocates based on actual physical memory. If I want to run a test to let the OS handle virtual memory and have MonetDB try and use more of system virtual memory, how do I do that? I am interested in seeing how this affects performance when the database is too large to fit in memory. - Are there any other tuning settings I should set when using a database that is much larger than memory? Thanks in advance, Rahul > ----- Original Message ----- > From: "Rahul Chopra" <rchopra(a)linuxmail.org> > To: "Communication channel for MonetDB users" <monetdb-users(a)lists.sourceforge.net> > Subject: Re: [MonetDB-users] tuning advice for TPC-H > Date: Thu, 10 Jan 2008 03:14:54 +0800 > > > > I am still trying to get better performance when doing TPC-H tests > for a database more than twice the size of physical memory. > > I am not sure how MonetDB handles its virtual memory, but it sure > seems to be going to disk an awful lot. Does it swap out entire > BATs? Or just smaller blocks? What type of caching algorithms does > it use? > > Also, one of the things I wanted to try is to setup a really big > swap partition on a separate device with write-back cache enabled > and have MonetDB use it (Basically, see if Linux will virtual > memory better than MonetDB does, and take advantage of putting swap > on a separate device for speed). When I try and do that though, > MonetDB seems to just be trying to stay within the confines of > physical memory available. Is there something I can configure to > try and get it to use system virtual memory? Has anyone tried out > anything like that? > > -Rahul > > > > > > > > > > > > > > I am trying to run some TPC-H queries to get a sense for how > > > > large of databases MonetDB can handle. I am currently trying > > a > > factor 20 database. I notice that on query 9, it is taking > > over > > two hours (still running). In "top" it claims that > > virtual > memory > for the process started at 27 GB or so, and is > > now down > to 10 GB. > What is noteworthy is that CPU usage is > > pretty small, > at 2%. I > previously observed 80-100% with a > > factor 1 database. > > > > > > Does this mean you got the sf-20 loaded? What fixed your problem with > > > loading? > > > > Yes, I got it loaded. Since I had broken up the file into smaller > > chunks, I just restarted on the last chunk that it failed on > > before the crash. Eventually, it went through all of the tables. > > > > > > > > The low cpu usage indicates that MonetDB is waiting for IO. I'll later > > > this week try to repeat your experiment on a similar system. Then I'll > > > have more input on how to improve things. > > > > After writing the email I noticed that system CPU wait time was > > at 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is > > waiting on IO. Perhaps it is just my hardware. > > > > I did disable the 2 GB swap partition and rerun. After doing so, > > execution time did improve somewhat. I am not sure if this means > > one should always disable swap on a MonetDB system, since the > > system drive is slower than the data drives. I will try another > > experiment and let you know the results. > > > > > = > College For Financial Planning > Nation's most successful CFP® Program with 35+ years of experience. > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=0badaaf35b41b0f3ca7504b… > > > -- > Powered by Outblaze > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Las Vegas, Nevada Home for Sale Dave Radcliffe will help you find your Las Vegas new home for sale, dream condo, townhome, or real estate investment property. Search here for all the homes for sale in Las Vegas, Nevada. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=95c6cb18131e4c352f8c920… -- Powered by Outblaze

2 1

[MonetDB-users] Distributed database
by Michaela Buergle 17 Jan '08

17 Jan '08

Hi there, I'm new to MonetDB and trying to figure out if it can be used for a db distributed over several computers. Have any of you tried to do that? Is there any documentation available? Thanks a lot micha

4 7

[MonetDB-users] create database on different drive
by Jos van Dongen 16 Jan '08

16 Jan '08

Hi all, I’m having a few difficulties on a Windows server: - I want to move the dbfarm to another drive and have altered the gdk_farm setting in the etc\monetdb5.conf file: gdk_dbfarm=G:\MonetDB5\dbfarm. MonetDB server however keeps creating a new MonetDB dir in the ‘documents and settings’ directory, the last place where I want to store a database. Do I need to alter other settings, or is there another config file? - How do you create a new database (on the different drive) and connect to that one instead of the standard ‘demo’ db? Met vriendelijke groet/kind regards, Jos van Dongen Tholis Consulting T: 06-51169606 F: 0842-231136 M: jvdongen(a)tholis.com www.tholis.com No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.19.2/1224 - Release Date: 14-1-2008 17:39

3 2

[MonetDB-users] make failed for buildtools/mel
by xxiao 15 Jan '08

15 Jan '08

Hi, I am trying to build MonetDB on SPARC/Solaris 10 platform from scratch. I started from buildtools. bootstrap and configure succeed. make of Mx succeed. make of mel failed. error messages: make[3]: Leaving directory `/tmp/MonetDB-CVS/buildtools/Mx' make[2]: Leaving directory `/tmp/MonetDB-CVS/buildtools/Mx' Making all in mel make[2]: Entering directory `/tmp/MonetDB-CVS/buildtools/mel' make[2]: *** No rule to make target `all'. Stop. make[2]: Leaving directory `/tmp/MonetDB-CVS/buildtools/mel' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/tmp/MonetDB-CVS/buildtools' any help is greatly appreciated! -- View this message in context: http://www.nabble.com/make-failed-for-buildtools-mel-tp14841558p14841558.ht… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Incorrect version (SR4 instead of SR5) linked to from download page
by Bob Ippolito 10 Jan '08

10 Jan '08

The Super Source Tarball referenced on http://monetdb.cwi.nl/projects/monetdb/Download/index.html is SR4, when it should be SR5. I wasted a couple hours trying to make SR4 build correctly on Mac OS X 10.5 when I finally found out that I was building an old version. I re-built with SR5 and everything seems to be working fine so far. -bob

2 1

[MonetDB-users] Sqlalchemy support to be in trunk soon
by m h 10 Jan '08

10 Jan '08

Just thought I'd give any python developers here a heads up. I've recently gotten commit access to sqlalchemy and am planning on pushing Monetdb support in. Hopefully the 0.4.3 release will include it as "experimental". For motivated python users who want to try it know, the source is found at bug 874[0] cheers, matt 0 - http://www.sqlalchemy.org/trac/ticket/874

1 0

Re: [MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 09 Jan '08

09 Jan '08

I am still trying to get better performance when doing TPC-H tests for a database more than twice the size of physical memory. I am not sure how MonetDB handles its virtual memory, but it sure seems to be going to disk an awful lot. Does it swap out entire BATs? Or just smaller blocks? What type of caching algorithms does it use? Also, one of the things I wanted to try is to setup a really big swap partition on a separate device with write-back cache enabled and have MonetDB use it (Basically, see if Linux will virtual memory better than MonetDB does, and take advantage of putting swap on a separate device for speed). When I try and do that though, MonetDB seems to just be trying to stay within the confines of physical memory available. Is there something I can configure to try and get it to use system virtual memory? Has anyone tried out anything like that? -Rahul > > > > > > > > I am trying to run some TPC-H queries to get a sense for how > > > large of databases MonetDB can handle. I am currently trying a > > > factor 20 database. I notice that on query 9, it is taking over > > > two hours (still running). In "top" it claims that virtual > > memory > for the process started at 27 GB or so, and is now down > > to 10 GB. > What is noteworthy is that CPU usage is pretty small, > > at 2%. I > previously observed 80-100% with a factor 1 database. > > > > Does this mean you got the sf-20 loaded? What fixed your problem with > > loading? > > Yes, I got it loaded. Since I had broken up the file into smaller > chunks, I just restarted on the last chunk that it failed on before > the crash. Eventually, it went through all of the tables. > > > > > The low cpu usage indicates that MonetDB is waiting for IO. I'll later > > this week try to repeat your experiment on a similar system. Then I'll > > have more input on how to improve things. > > After writing the email I noticed that system CPU wait time was at > 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is > waiting on IO. Perhaps it is just my hardware. > > I did disable the 2 GB swap partition and rerun. After doing so, > execution time did improve somewhat. I am not sure if this means > one should always disable swap on a MonetDB system, since the > system drive is slower than the data drives. I will try another > experiment and let you know the results. > = College For Financial Planning Nation's most successful CFP® Program with 35+ years of experience. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=0badaaf35b41b0f3ca7504b… -- Powered by Outblaze

1 0

[MonetDB-users] Info reg the Physical Tables..
by sateesh 08 Jan '08

08 Jan '08

Hi All, If i create two logical tables which has the same column then when MonetDB actually creates the Physical Tables does it create two tables for this Column.. For Ex: I created these two Tables at the Logical Level.. TAB_1 col_1 varchar(100), col_2 int TAB_2 col_3 varchar(256), col_2 int Now I home MonetDB will create these physical tables.. For TAB_1 PHYSICAL_TAB_1 PHYSICAL_TAB_2 OID , COL_1 OID , COL_2 How abt for TAB_2 Does it create PHYSICAL_TAB_3 PHYSICAL_TAB_4 OID , COL_3 OID , COL_2 or Just PHYSICAL_TAB_3 OID , COL_3 Thanks Sateesh -- View this message in context: http://www.nabble.com/Info-reg-the-Physical-Tables..-tp14692698p14692698.ht… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Noob question: Why can I only initiate mserver5 as root?
by Claude Monet 08 Jan '08

08 Jan '08

Logged as root mserver5 works perfectly. Yet when I login as a user and call mserver5: >mserver5 --dbinit="include sql;" I get the error: !FATAL: GDKlockHome: Database lock '.gdk_lock' denied Any help is greatly appreciated

3 2

Re: [MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 08 Jan '08

08 Jan '08

> > > > I am trying to run some TPC-H queries to get a sense for how > > large of databases MonetDB can handle. I am currently trying a > > factor 20 database. I notice that on query 9, it is taking over > > two hours (still running). In "top" it claims that virtual memory > > for the process started at 27 GB or so, and is now down to 10 GB. > > What is noteworthy is that CPU usage is pretty small, at 2%. I > > previously observed 80-100% with a factor 1 database. > > Does this mean you got the sf-20 loaded? What fixed your problem with > loading? Yes, I got it loaded. Since I had broken up the file into smaller chunks, I just restarted on the last chunk that it failed on before the crash. Eventually, it went through all of the tables. > > The low cpu usage indicates that MonetDB is waiting for IO. I'll later > this week try to repeat your experiment on a similar system. Then I'll > have more input on how to improve things. After writing the email I noticed that system CPU wait time was at 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is waiting on IO. Perhaps it is just my hardware. I did disable the 2 GB swap partition and rerun. After doing so, execution time did improve somewhat. I am not sure if this means one should always disable swap on a MonetDB system, since the system drive is slower than the data drives. I will try another experiment and let you know the results. = Series 7 NASD Exam Training Series 7 and related NASD Exam Books, software and web based training. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=16e765a220119782741a407… -- Powered by Outblaze

1 0

[MonetDB-users] MonetDB scalability
by Djordje Djokovic 08 Jan '08

08 Jan '08

2 1

[MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 08 Jan '08

08 Jan '08

I am trying to run some TPC-H queries to get a sense for how large of databases MonetDB can handle. I am currently trying a factor 20 database. I notice that on query 9, it is taking over two hours (still running). In "top" it claims that virtual memory for the process started at 27 GB or so, and is now down to 10 GB. What is noteworthy is that CPU usage is pretty small, at 2%. I previously observed 80-100% with a factor 1 database. I have 8 GB RAM, with 2 GB RAM swap. System swap is on a slower partition than the data drive, which is a relatively fast RAID 5 disk array. Might I be better off disabling system swap altogether when using MonetDB? Are there any other configuration changes I should make or MonetDB best practices I should follow? Thanks, Rahul = Mercedes Homes- San Antonio real estate Mercedes Homes builds homes and communities in the San Antonio area. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=9c24f83a902bfb70ef8d7bc… -- Powered by Outblaze

2 1

[MonetDB-users] indexes
by Rahul Chopra 07 Jan '08

07 Jan '08

Hello, I was just curious, what happens behind the scenes when an index is created in MonetDB? I noticed it is instantaneous... just wondering why. Is the BAT file reordered (for a single column index) kind of like for a clustered index, or is a separate index created? Thanks, Rahul = Truelife Financial Get the loan you deserve today regardless your credit history. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=cc5aa117b4038bf76c9b23d… -- Powered by Outblaze

2 1

[MonetDB-users] loading data
by Rahul Chopra 07 Jan '08

07 Jan '08

I ran into some problems loading data for doing some TPC-H tests. After a successful test for a factor 1 database, I moved on to test a factor 20 database. mserver5 appeared to crash when loading partsupp, orders, or lineitem. On a hunch, I ran split on these to ensure that each file was less than 2 GB each, splitting partsupp and orders into two files, and orders into 10. After doing this, partsupp and orders loaded without any problems. However, lineitem would cause mserver5 to crash after loading 2 or 3 of the 1.5 GB chunks. Also, in restarting mserver5, I noticed that CPU usage was sometimes heavy for a while, before starting loading. I am guessing it is doing some recovery or cleanup? Is loading an issue? Since it was busy upon restart, do I need to wait a while before loading a new file to give the system some time to do any housekeeping or something? I am using the latest released sources (not from HEAD), compiled on CentOS 5.1 (64 bit). Thanks, Rahul = Live Answering Service Answering. Live operators 24/7. Rates start at $39/100 calls. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=202d1706cc4cee71bfb381f… -- Powered by Outblaze

2 1

Re: [MonetDB-users] gdk_mem_bigsize and gdk_vm_minsize
by Rahul Chopra 03 Jan '08

03 Jan '08

> > On Fri, Jan 04, 2008 at 12:09:05AM +0800, Rahul Chopra wrote: > > Can you please explain gdk_mem_bigsize and gdk_vm_minsize? I > > searched around on the site and I am not really clear on what the > > optimal settings for these are. > > Memory blocks allocated via GDKmalloc and friends larger then > gdk_mem_bigsize will be mmaped using virtual swap space (vmalloc). This > is done to avoid fragmentation. So this is just for an individual block, and not for total allocated memory? Is a "block" allocated via entire BATs? Or are blocks allocated in segments when read from a single BAT? > > BATs have heaps, these can be malloced (GDKmalloc), virtual mmapped > using swap space (GDKvmalloc) or mmapped (needs a filename). > Blocks (heaps) larger then vm_minsize will be vmalloced. > The backends (sql/xquery) control when we switch to mmapped with files. > > > > > I am currently using the defaults, on a system with 8 GB of RAM. > It depends on your os if thats okay. Thanks. I am using Linux (64 bit), CentOS 5.1. My main concern was if setting these too low somehow restricts the amount of memory used and leads to more swapping. I am not familiar enough with the system to understand all of this, but with 8 GB of ram, should I be using a larger gdk_mem_bigsize than the config default of 262144? The config file mentions these being set to 1/2 of available memory, but default values appear commented out. Are those the values that are actually used? Or, if these are commented out, are these dynamically set? -Rahul > > Niels > > > > Thanks, > > > > Rahul > > > > > > = > > Limousine and Luxury Bus Wine Tours > > Private Napa & Sonoma wine-tasting tours via luxury limousine or bus. > > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=2c5439db6084d9969b4f63e… > > > > > > -- Powered by Outblaze > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2005. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > -- > > Niels Nes, Centre for Mathematics and Computer Science (CWI) > Kruislaan 413, 1098 SJ Amsterdam, The Netherlands > room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 > url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Mercedes Homes - In FL, TX, NC, & SC Mercedes Homes builds homes in Florida, Texas, and the Carolinas. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=e0639e1f90664efd0041110… -- Powered by Outblaze

2 1

Re: [MonetDB-users] killing queries
by Rahul Chopra 03 Jan '08

03 Jan '08

Thanks for the information, I understand. Other than killing the client application, is there a way to separately connect to the server and gracefully kill the query or connection? > ----- Original Message ----- > From: "Niels Nes" <Niels.Nes(a)cwi.nl> > To: "Communication channel for MonetDB users" <monetdb-users(a)lists.sourceforge.net> > Subject: Re: [MonetDB-users] killing queries > Date: Thu, 3 Jan 2008 17:47:45 +0100 > > > On Fri, Jan 04, 2008 at 12:06:37AM +0800, Rahul Chopra wrote: > > How can I safely kill queries? > > > > I notice if I ctrl-c out of mclient when executing a long run > > query, mserver5 still seems to be using a lot of memory, maybe it > > has not deallocated it all? Also, if I start a new mclient > > session queries now run slower. Of course, one should always > > avoid ctrl-c, but in case we do have multiple queries running and > > we wish to kill a long running one for some reason, is there a > > way to safely do so? (Like, start a new mclient session, and > > execute some kind of admin commands). > Resources taken because of killed clients should disappear. This can > however take some time as the cleanup is only done within the > interpreter loop. Also we keep the used persistent tables in memory. > They only get freed when running out of memory or when idle for some > time. > > I'm not sure which version your using, but also the current released > version has some memory leaks which maybe the reason for your memory > growth and slowdown. > > Niels > > > > TIA > > > > > > = > > Espresso Machines - Free Shipping > > Espresso Machines, coffee beans and pods, cappuccino cups and > > spoons. Free shipping. No sales tax. > > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=8427826bf39d56eb7c331a1… > > > > > > -- Powered by Outblaze > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2005. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > -- > > Niels Nes, Centre for Mathematics and Computer Science (CWI) > Kruislaan 413, 1098 SJ Amsterdam, The Netherlands > room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 > url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Area Homes Realty Online Home Search Area Homes Realty is a full service residential real estate company servicing Eastern Massachusetts. Our website is a valuable resource of information whether you are buying or selling. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=6d65a7e04b5c96a53badb07… -- Powered by Outblaze

2 1