- developers-list - monetdb.org

[Monetdb-developers] join performance issue!
by Stefan Manegold 25 Jul '06

25 Jul '06

Dear all, to prevent you from falling into the same trap as I did this afternoon when trying to analyze, measure & compare positional (fetch-)join performance, here's a warning and my detailed story: WARNING: When running 'join(BAT,BAT);' in MIL or 'algebra.join(BAT,BAT);' in MAL, be aware that this includes not only the actual join algorithm, but also a BATpropcheck() call to check and set properties of the join result (ALWAYS, i.e., also with the default --debug=0 !) NOTE: The latter can easily be up to 85% (*eighty-five* percent!) of the time you measure with a simple wall-clock around the 'join(BAT,BAT);' or 'algebra.join(BAT,BAT);' (see details below) ! Detailed Story: Preparing to invent and implement another cache-aware/-friendly positional (fetch-)join algorithm for some special cases, I was trying to assess the performance of the current positional (fetch-)join algorithms for the following three cases: 1) join(ld,r); 2) join(lo,r); 3) join(lu,r); In all three cases, r is a [V(o)ID, INT] BAT, i.e., has a dense non-materialized head and a randomly distributed integer tail. ld, lo, lu are all [INT, OID] BATs with a randomly distributed integer tail, and an OID head that is in ld: dense (and marked so in the BAT header) (but materialized) in lo: ordered (actually also dense, but only marked ordered, not marked dense in the BAT header) in lu: unordered (random shuffle of an originally dense sequense) Thus, the three case trigger the use following join implementations, respectively: 1) densefetchjoin 2) orderedfetchjoin 3) defaultfetchjoin I used the same size (count) four BATs, and the join is a perfect 1:1 hit; hence the result is as big as the inputs. Here are the wall-clock times (in micro seconds) for BATs of 10000000 tuples MonetDB 4.13.1 (CVS HEAD) default compilation (gcc -g -O2) 64-bits and 64-bit OIDs 2 GHz AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ 2 GB RAM 1) join(ld,r); 2065162 us 2) join(lo,r); 2090277 us 3) join(lu,r); 3058936 us A close look under the hood (i.e., in BATjoin in src/gdk/gdk_relop.mx) reveals the following split timings for the actual join (batjoin()) and the property checking (BATpropcheck()): batjoin() BATpropcheck() join() 1) join(ld,r); 275331 us 1774275 us 2065162 us (abs) 13.33 % 85.91 % 100.00 % (rel) 2) join(lo,r); 303311 us 1770722 us 2090277 us (abs) 14.51 % 84.71 % 100.00 % (rel) 3) join(lu,r); 1268057 us 1775788 us 3058936 us (abs) 41.45 % 58.05 % 100.00 % (rel) Well, there are some lessons to learn: - Be aware of the BATpropcheck when you measure join performance! - The "mandatory" BATpropcheck is only used for BATjoin and BATleftjoin in src/gdk/gdk_relop.mx --- at least as far as I know. - The "mandatory" BATpropcheck in BATjoin and BATleftjoin is there for a good reason: With joins, it's hard to guess the properties of the result (are head and/or tail sorted, dense, and or key/unique?) in general, but the performance of MonetDB highly depends on the availability of these properties; hence, while making the join (considerably) more expensive, the investment (might) pay-off with later operation on the join result. - We should consider making this BATpropcheck in BATjoin and BATleftjoin configurable from the outside, e.g., by adding an extra optional argument to join() to request to skipping the BATpropcheck. I hope you find this information useful... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

[Monetdb-developers] (no subject)
by Dennis Rutjes 25 Jul '06

25 Jul '06

L.S, Sorry dat ik dit niet vanaf de SF bug entry kan doen, ik kan me momenteel niet aanmelden. Ik ben om monetDb te deployen op een windows machine(dual core 2 GIg), eesrt gebruikmakend van de release(msi) installer. Daarna kreeg ik na het importeren van 200 k records, de fout # Monet Database Server V4.12.0 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-win32/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !WARNING: BBPincref: range error 283 !WARNING: BBPincref: range error 284 !WARNING: BBPincref: range error 282 !WARNING: BBPincref: range error 285 Ik dacht misschien zijn er XP specifieke dll's die mee gecompileert dienen te worden. MicrovisualC++ EXPRESS gedownload(gratis tegenwoordig met windows SDK SP2 :-), met nmake in de NT directory gedraaid(Ja alle vooraf gestelde eisen aan voldaan python etc) krijg ik de volgende melding. D:\Development\Projects_C++\Monet_CVS\MonetDB\NT>nmake NEED_MX=1 Microsoft (R) Program Maintenance Utility Version 8.00.50727.42 Copyright (C) Microsoft Corporation. All rights reserved. python ".\configure.py" "." "D:\Development\Projects_C++\Monet_CVS\Monet DB\NT" monetdb_config.h.in > monetdb_config.h echo #ifndef UNISTD_H > unistd.h echo #define UNISTD_H >> unistd.h echo #include "io.h" >> unistd.h echo #define open _open >> unistd.h echo #define read _read >> unistd.h echo #define write _write >> unistd.h echo #define close _close >> unistd.h echo #define getpid _getpid >> unistd.h echo #define umask _umask >> unistd.h echo #endif >> unistd.h if exist ".\..\RunMserver.bat.in" python ".\configure.py" "." "D:\Develo pment\Projects_C++\Monet_CVS\MonetDB\NT" ".\..\RunMserver.bat.in" > RunMserver.b at if exist ".\..\RunMapiClient.bat.in" python ".\configure.py" "." "D:\Dev elopment\Projects_C++\Monet_CVS\MonetDB\NT" ".\..\RunMapiClient.bat.in" > RunMap iClient.bat if exist ".\..\RunMtest.bat.in" python ".\configure.py" "." "D:\Developm ent\Projects_C++\Monet_CVS\MonetDB\NT" ".\..\RunMtest.bat.in" > RunMtest.bat if exist ".\..\RunMapprove.bat.in" python ".\configure.py" "." "D:\Devel opment\Projects_C++\Monet_CVS\MonetDB\NT" ".\..\RunMapprove.bat.in" > RunMapprov e.bat "D:\EPF\Microsoft Visual studio 8\VC\BIN\nmake.exe" /nologo /f ".\..\Mak efile.msc" "prefix=D:\Development\Projects_C++\Monet_CVS\MonetDB\NT" all if not exist "src" mkdir "src" copy ".\..\src\Makefile.msc" "src\Makefile" 1 file(s) copied. cd "src" && "D:\EPF\Microsoft Visual studio 8\VC\BIN\nmake.exe" /nologo "prefix=D:\Development\Projects_C++\Monet_CVS\MonetDB\NT" all if not exist "common" mkdir "common" NMAKE : fatal error U1073: don't know how to make '".\..\..\src\common\Makefile. msc"' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: '"D:\EPF\Microsoft Visual studio 8\VC\BIN\nmake.exe"' : return code '0x2' Stop. D:\Development\Projects_C++\Monet_CVS\MonetDB\NT> Is daar wat aan te doen?, ik ben niet zo'n C++ wizz. Met vriendelijke groet, Dennis Rutjes

1 0

[Monetdb-developers] Problem editing the .mx files.
by Luis Zarrabeitia 24 Jul '06

24 Jul '06

Hello. I need to pipe data into a monetdb database (something along the lines of 'myprocess | MapiClient -s "COPY ...."'. It doesn't seem to be possible out-of-the-box, so I wanted to poke around in the code to find out why and how to fix it (if needed), and hopefully submit a patch. I've been having a problem with that (playing with the code). I don't know how to 'compile' the modified .mx files in the sql module. I have both bison and flex installed, but the makefile uses the commands 'x' and 'l' to process the .mx, and I don't have any idea of what x and l are. So, in a nutshell, my question is: "How do I recompile the sql module after I modify the .mx files? What are 'x' and 'l'?" Regards, Zarrabeitia.

3 4

[Monetdb-developers] Time, Money, Keeping you from earning the Degree you deserve?
by money12542＠aol.com 19 Jul '06

19 Jul '06

1 0

[Monetdb-developers] native XML database and XML:DB driver
by Berny Linnemann 18 Jul '06

18 Jul '06

Hello, I thought lets send a new posting to specify my goal. I want to use monetDB asa native XML database embedded in a java application. Thats why I would like to know where I can find the textfile as mentioned in the article found in the next link: http://www.monetdb.nl/TechDocs/APIs/JDBC/node9.html In my first posting I mentioned I was tying to find the textfile in some files I downloaded. That were XQuery binaries and source files. Perhaps I was looking at the wrong location and could somebody tell where so find the needed textfile. With kind regards, Berny

2 1

[Monetdb-developers] Getting the correct jar files
by Berny Linnemann 18 Jul '06

18 Jul '06

Hello, I have got a problem with my attempt getting the correct JAR files to use the XML:DB driver as mentioned in this link: http://www.monetdb.nl/TechDocs/APIs/JDBC/node9.html In this article a location is specified they explain how to create the needed jar files. But I can't find the textfile ther refer to in this article. I downloaded a few binary and source but couldn't find the needed textfile. So what do I need to do getting this textfile? Berny

2 1

[Monetdb-developers] Get xquery timing, Error when running examples, etc
by kla gw 17 Jul '06

17 Jul '06

Hi, I just started to use MonetDB. My PC is Windows XP. I installed the MonetDB-XQuery-0.10.2.msi and few weeks later after the newer version come out, I uninstalled from Control Panel & install 0.12.0. I have several questions: 1. How to run the query from the command prompt and get the timing? Currently, everytime I want to run the query, I have to start XQuery Server (start - all programs - monetdb xquery - XQuery Server). After that I save the xquery into .xq file, then run it by right click - open with MPFClient. I've tried the following in XQuery Server but it gives me error: MonetDB>MapiClient -T -lx test.xq; !ERROR: "MapiClient -T -lx test.xq;" ! ^ ! can't help you here, sorry. MonetDB> 2. I tried to run the example in the website, books1.xq and books2.xq and slashdot.xq. In the files, the xml file is from internet. But when I run the files, the following error message come out: ERROR: No such file or directoryI/O warning : failed to load external entity "http://monetdb.cwi.nl/XQuery/files/bib.xml" ERROR: shred:mainSHREDDER: xmlCreateURLParserCtxt(" http://monetdb.cwi.nl/XQuery/files/bib.xml") failed ERROR: CMDshred2bats: operation failed. ERROR: interpret_params: +(param 2): evaluation error. Is it connection problem? How to modify the connection settings so that I can download data from internet? 3. voc2.xq and voc5.xq give the following error: type error: [err:XPTY0004] Orderspec requires at most one atomic item (given: string*). 4. Where is the shredded document being stored? Is it in the place where I install the program or in My Documents? Because before I uninstall 0.10.2, I shredded some documents, and after I uninstall then install 0.12.0, the shredded xml documents are still there. Thanks, Klarinda

5 6

[Monetdb-developers] Another Question about use Aqua Datastudio
by china-davy 17 Jul '06

17 Jul '06

When I use Aqua Data Studio 4.7 as a SQL client,I must register a database first. If I want to connect a remote server,I must fill in all the blank in the Server Properties . For example:I must fill in the following information Name: Remote-monet-server Type: Production Login Name: monetdb Password:monetdb Datasource: this field I don not know how to write :( Can you tell me how to fill in this field? When I use "MonetDB",I can only connect localhost server, if I want to connetct the remote server, how to write the Datasource? "MonetDB(a)10.78.90.17" is right?? Davy China -- View this message in context: http://www.nabble.com/Another-Question-about-use-Aqua-Datastudio-tf1950599.… Sent from the monetdb-developers forum at Nabble.com.

4 7

[Monetdb-developers] Question about remote connection to MonetDB
by china-davy 16 Jul '06

16 Jul '06

Hi: I am a starter of MonetDB.I encountered a problem about conneting a remote MoneDB server via ODBC.The problem is discribed as follows: Given that the MonetDB server is running on a host(10.20.30.1),and,the Application program is running on a host (10.20.30.2).The MonetDB version is the windows version.I have tried to connet the server through the ODBC connect string in my appliction program,but failed,by the way,I am ensure the connect string is correct. My question is: 1.Can MonetDB server be connected via ODBC driver from a remote Application Clinet? 2.If it is so,how can I connet the MonetDB server.Wether need I do some configuration in the configure file of the MonetDB server? Waiting for your suggestion!Thanks! best regards! Davy China -- View this message in context: http://www.nabble.com/Question-about-remote-connection-to-MonetDB-tf1950030… Sent from the monetdb-developers forum at Nabble.com.

3 8

[Monetdb-developers] Your email requires verification verify#lQA7JuOWyKG66nv6H4P2htV9lmaPlRrT
by monet＠billionairesforbush.com 13 Jul '06

13 Jul '06

The message you sent requires that you verify that you are a real live human being and not a spam source. To complete this verification, simply reply to this message and leave the subject line intact. The headers of the message sent from your address are show below: >From monetdb-developers(a)lists.sourceforge.net Wed Jul 12 21:31:08 2006 Received: from billiona by host.billionairesforbush.com with local-bsmtp (Exim 4.52) id 1G0q2a-0001Qj-Mc for monet(a)billionairesforbush.com; Wed, 12 Jul 2006 21:31:08 -0400 Received: from localhost by host.billionairesforbush.com with SpamAssassin (version 3.1.3); Wed, 12 Jul 2006 21:31:08 -0400 From: "monetdb-developers(a)lists.sourceforge.net" <monetdb-developers(a)lists.sourceforge.net> To: monet(a)billionairesforbush.com Subject: Start earning the salary you deserve by obtaining the proper credentials! Date: Thu, 13 Jul 2006 01:19:08 -0300 Message-Id: ndIuH4okPn4Om.z1ennQI6kFWz3(a)lists.sourceforge.net X-Spam-Flag: YES X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on host.billionairesforbush.com X-Spam-Level: ****************** X-Spam-Status: Yes, score=18.1 required=5.0 tests=BAYES_99,FORGED_RCVD_HELO, FUZZY_MONEY,HTML_40_50,HTML_MESSAGE,HTML_MIME_NO_HTML_TAG, INVALID_MSGID,MIME_HTML_ONLY,RCVD_IN_NJABL_DUL,RCVD_IN_SORBS_DUL, RCVD_IN_XBL,UNPARSEABLE_RELAY,UPPERCASE_25_50 autolearn=no version=3.1.3 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----------=_44B5A25C.752037C9"

1 0