- developers-list - monetdb.org

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_builder.mx, MonetDB_5-2, 1.26, 1.26.2.1
by Stefan Manegold 29 Oct '07

29 Oct '07

On Mon, Oct 29, 2007 at 07:09:42AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv18604/src/mal > > Modified Files: > Tag: MonetDB_5-2 > mal_builder.mx > Log Message: > pushStr shouldn't do a GDKstrdup, as it may return an equal string from > the constant cache, ie only if the returned variable holds the given string > (poniters are equal) we do a string dup. This solves another memory leak. > > > Index: mal_builder.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_builder.mx,v > retrieving revision 1.26 > retrieving revision 1.26.2.1 > diff -u -d -r1.26 -r1.26.2.1 > --- mal_builder.mx 28 Aug 2007 19:56:41 -0000 1.26 > +++ mal_builder.mx 29 Oct 2007 07:09:40 -0000 1.26.2.1 > @@ -260,10 +260,12 @@ > ValRecord cst; > > cst.vtype= TYPE_str; > - cst.val.sval= GDKstrdup(val); > + cst.val.sval= val; gcc -DHAVE_CONFIG_H -I. -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal -I../.. -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal -I../optimizer -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal/../optimizer -I../scheduler -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal/../scheduler -I../modules/kernel -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal/../modules/kernel -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/include/MonetDB -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/include/MonetDB/common -I/ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/include/MonetDB/gdk -DLIBMAL -Wall -Wextra -std=c99 -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 -D_REENTRANT -c mal_builder.c -fPIC -DPIC -o .libs/libmal_la-mal_builder.o cc1: warnings being treated as errors /ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal/mal_builder.mx: In function 'pushStr': /ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/mal/mal_builder.mx:263: warning: assignment discards qualifiers from pointer target type make[5]: *** [libmal_la-mal_builder.lo] Error 1 Stefan > cst.len= strlen(cst.val.sval); > _t = defConstant(mb,TYPE_str,&cst); > > + if (getConstant(mb,_t).val.sval == val) > + getConstant(mb,_t).val.sval = GDKstrdup(val); > return pushArgument(mb, q, _t); > } > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 4

[Monetdb-developers] MonetDB(5) and modules
by Fabian Groffen 26 Oct '07

26 Oct '07

I noticed that MonetDB5 uses modules, but compiles them as shared libraries. Maybe other targets do the same here. autogen appears to have (had) some partial support for modules, and the attached patch fixes/re-enables that support. In short it passes -module to libtool during link phase, such that the object is generated as loadable module. Why did I look at this? Mainly because on Darwin there is a difference between those objects that are loaded by dlopen(3) (loadable modules) and those that are used by the link editor ld(1) (shared libraries). The latter have type DYLIB whereas the former have type MH_BUNDLE. A so-called "bundle" doesn't have an install_name, as it can't be used by the link editor. Now initially all went well during compilation until I hit this error: /Library/Gentoo/usr/bin/ld: ../atoms/.libs/lib_color.so is input for the dynamic link editor, is not relocatable by the static link editor again This message is correct on Darwin, as batcolor.mx links against lib_color.so. libtool already detected this case: *** Warning: Linking the shared library lib_batcolor.la against the loadable module *** lib_color.so is not portable! ** Warning, lib lib_color.so is a module, not a shared library ** And there doesn't seem to be a static archive available ** The link will probably fail, sorry I don't recall ever seeing problems with these loads (apart from on Panther which didn't grok it at all) on my (as of today out-of-date) OSX 10.4, so I'm not sure how relevant it is to actually fix this. It looks like support to load shared libraries via dlopen actually was added, so we need not to worry about it at all. Actually, the man-page says it all: "dlopen() can load dynamic libraries and bundles." "In Mac OS X 10.4, dlopen was rewritten to be a native part of dyld." So far for sharing my experiences in this area. As far as I know, 10.3 support for MonetDB was shot already long ago (as it should), so maybe we should remove the remaining bits that could/can trigger -module libtool invocation, as it *will* cause trouble.

1 0

[Monetdb-developers] problems with XQuery aggregation
by Henning Rode 25 Oct '07

25 Oct '07

hej monetdb developers, i tried to write a simple aggregation query: let $cands := doc("x.xml")//c for $eid in distinct-values($cands/@id) return <candidate id="{$eid}" count="{count($cands[@id eq $eid])}"/> however, executing the above query turns out to be rather difficult. i always killed the process after waiting for more than half an hour without getting any result. x.xml is 17MB large and contains 300000 candidates nodes c. i run it on Monet Database Server V4.19.0 where the query crashed ERROR = !ERROR: GDKload: cannot mmap(): name=10/1030, ext=buns.priv !OS: Cannot allocate memory !ERROR: GDKload failed: name=10/1030, ext=buns.priv !ERROR: CMDleftjoin: operation failed. and on the newer stable release MonetDB Server v4.20.0 on a machine with 16GB memory where the query just runs endless. apparently the query compiler does not recognise the necessary join and aggregation here. is that a known problem? btw. sorting the candidates by @id value is done in seconds: let $cands := doc("x.xml")//c for $c in $cands order by $c/@id return $c best -henning

3 4

[Monetdb-developers] Segmentation Fault upon sql.start()
by Stefan de Konink 22 Oct '07

22 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 After loading in a bunch of data the monetdb server was Killed due to memory starvation. With swap MonetDB performed the huge amount of one insert in 10 seconds, so that wasn't an option too. I restarted it, and tried to 'load the data again'. Backtrace: gdb ./mserver5 GNU gdb 6.7 Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu"... Using host libthread_db library "/lib/libthread_db.so.1". (gdb) run Starting program: /opt/monetdb/bin/mserver5 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff275fe000 [Thread debugging using libthread_db enabled] [New Thread 0x2afd85f0f2a0 (LWP 23929)] # MonetDB Server v5.0.0 # Copyright (c) 1993-2007 CWI, all rights reserved # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # dbname:osm # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /opt/monetdb/etc/monetdb5.conf) [New Thread 0x40800950 (LWP 23932)] >include sql; >sql.start(); Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x2afd85f0f2a0 (LWP 23929)] 0x00002aaaaab40a8f in load_column () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so (gdb) bt #0 0x00002aaaaab40a8f in load_column () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #1 0x00002aaaaab41a71 in load_table () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #2 0x00002aaaaab433cc in load_trans () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #3 0x00002aaaaab46e15 in store_init () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #4 0x00002aaaaab1b95b in mvc_init () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #5 0x00002aaaaaad0b50 in SQLinit () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #6 0x00002afd8389f476 in initScenario () from /opt/monetdb/lib/libmal.so.0 #7 0x00002aaaaaad0ac8 in SQLsession () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #8 0x00002afd8387842d in runMALsequence () from /opt/monetdb/lib/libmal.so.0 #9 0x00002afd83879dfd in runMAL () from /opt/monetdb/lib/libmal.so.0 #10 0x00002afd83871dc8 in MALengine () from /opt/monetdb/lib/libmal.so.0 #11 0x00002afd8389f082 in runScenarioBody () from /opt/monetdb/lib/libmal.so.0 #12 0x00002afd8389f0d2 in runScenario () from /opt/monetdb/lib/libmal.so.0 #13 0x00002afd838719e0 in MSserveClient () from /opt/monetdb/lib/libmal.so.0 #14 0x0000000000402c14 in main () (gdb) It work doesn't anymore... I can conclude, inserting a lot of data start to become a huge waste of time. If required I can recompile with the -g option and a lower amount of optimization. Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHHQTsYH1+F2Rqwn0RCjWKAJ9u/OEu8Sk6HTSY+v431bLjmylrCwCffsNN Rpf0Fs4XjrakYkdyEwkknWQ= =0UHS -----END PGP SIGNATURE-----

2 2

[Monetdb-developers] python development
by Sjoerd Mullender 22 Oct '07

22 Oct '07

For all you Python developers, a new rule is being enforced from now on: Python source files are not allowed to contain TAB characters or trailing white space. This is enforced at cvs commit time. -- Sjoerd Mullender

1 0

[Monetdb-developers] man-pages
by Fabian Groffen 22 Oct '07

22 Oct '07

We should put or html-ified (autogenerated) man-pages for merovingian, monetdb and mclient on our website, as easy to read documentation.

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/testing Mtest.py.in, MonetDB_1-20, 1.309.2.8, 1.309.2.9
by Fabian Groffen 22 Oct '07

22 Oct '07

On 21-10-2007 16:36:26 +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB/src/testing > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv16749/src/testing > > Modified Files: > Tag: MonetDB_1-20 > Mtest.py.in > Log Message: > export TST_FIVE, we sometimes need to know which server we test > > > Index: Mtest.py.in > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/testing/Mtest.py.in,v > retrieving revision 1.309.2.8 > retrieving revision 1.309.2.9 > diff -u -d -r1.309.2.8 -r1.309.2.9 > --- Mtest.py.in 19 Oct 2007 13:41:24 -0000 1.309.2.8 > +++ Mtest.py.in 21 Oct 2007 16:36:21 -0000 1.309.2.9 > @@ -1237,6 +1237,7 @@ > os.environ['TST_MODS'] = str(env['TST_MODS']) > if par['M5']: > env['TST_FIVE'] = "Five" > + os.environ['TST_FIVE'] = "Five" This looks like an upcoming IndentationError to me. > return rtrn > ### GetBitsAndOIDsAndModsAndStatic(env) #

2 2

[Monetdb-developers] Mserver5 issues / Perl patches
by Stefan de Konink 21 Oct '07

21 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, [If the first part should be in monetdb-users please inform me] Currently I am still using the same setup, because I do not have the time to reimport my data using after the nightly builds. After my first big import in SQL resulting in 4*10^7 rows, I decided to move on. And continue my work. I have started mserver5 again, include sql; [no problems] sql.start(); [this command took 10 minutes to be processed] After this 10 minutes, the command works instantly when running the server again. Is this some kind of integrity checking? In my last e-mail I asked about the Perl issues, Niels pointed that the Perl sources were out of sync. I took the liberty to try to come up with a fix, please check as attached. I think no copyright can be claimed on 3 changes, so if this is the right solution :) Please apply :) Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHGqTsYH1+F2Rqwn0RCnZqAJ9l4IMVeGOCMm7JJN0BijnCvEIrAQCdEDd0 8zEe0HfX1ygvsJdawvOC63Q= =RqZ8 -----END PGP SIGNATURE-----

2 1

[Monetdb-developers] Perl documentation and connectivity
by Stefan de Konink 20 Oct '07

20 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello, I'm trying the first lines of the Perl example presented: http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/The-Perl-Library.h… # connect to the database: my $dsn = 'dbi:monetdb:database=test;host=localhost;port=50000;language=mil'; my $dbh = DBI->connect( $dsn, undef, undef, # no authentication in MIL { PrintError => 0, RaiseError => 1 } # turn on exception handling ); skinkie@xen01 ~/private $ perl test.pl Start a simple Monet MIL interaction data sources: dbi:monetdb: DBI connect('database=test;host=localhost;port=50000;language=mil','',...) failed: RuntimeError Usage: mapi_connect(host,port,username,password,lang,dbname); at test.pl line 15 (I have traced this back to: MonetDB/CLI/MapiLib.pm) Is there some trivial thing I have missed? (I'm definitely not a Perl hero so forgive me for my incompetence.) Yours Sincerely, Stefan de Konink ps. Running: # MonetDB Server v5.0.0 # Copyright (c) 1993-2007 CWI, all rights reserved # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # dbname:demo ps2. I know test!=demo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHGU0jYH1+F2Rqwn0RCvlvAJ0XbNzyV8ySOyoNcKbmitNWUSwH1gCfaDma /kMe7PQ4nl9cRhMcKpun2Uw= =BP78 -----END PGP SIGNATURE-----

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/debug logdebug.c, XQuery_0-20, 1.72, 1.72.2.1
by Stefan Manegold 18 Oct '07

18 Oct '07

Sjoerd, not that I want to "worry" you, but just out of curiosity: did this one make it into the release? If not, I would not recomment to re-build the release once again --- unless Jan has very strong arguments that this is a strickt show-stopped (don't think so, since most tests work fine) ---, but just keep it for a potential bug-fix release. Stefan On Wed, Oct 17, 2007 at 04:29:31PM +0000, Jan Rittinger wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/debug > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv26773/debug > > Modified Files: > Tag: XQuery_0-20 > logdebug.c > Log Message: > -- Bugfix: Added missing 'break;'. > > > Index: logdebug.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/debug/logdebug.c,v > retrieving revision 1.72 > retrieving revision 1.72.2.1 > diff -u -d -r1.72 -r1.72.2.1 > --- logdebug.c 18 Sep 2007 11:46:58 -0000 1.72 > +++ logdebug.c 17 Oct 2007 16:29:28 -0000 1.72.2.1 > @@ -1494,6 +1494,7 @@ > PFatt_str (n->sem.pos_sel.part)); > > PFarray_printf (xml, " </content>\n"); > + break; > > case la_fun_1to1: > PFarray_printf (xml, > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1