- developers-list - monetdb.org

Hash join implementation
by Xu,Wenjian 11 Jul '15

11 Jul '15

Hi Stefan, Thank you very much for your suggestions. They are very helpful. I am also trying to understand the hashjoin() implementation in gdk/gdk_join.c. I am aware of the classical *multi-pass partitioned hash join* as described in the literature: [1] Main memory hash joins on multi-core CPUs: tuning to the underlying hardware. Cagri Balkesen et al. ICDE13 [2] Optimizing main-memory join on modern hardware. Stefan Manegold et al. TKDE (2002) I was trying to follow the idea in these papers to understand the implementation of hash join in MonetDB but failed. Do you have any suggestions? Are there any other papers/documents I can refer to? Thanks. Best regards, XU Wenjian >Hi Wenjian, >you might want to consider studying the logical (relational) >and physical (MAL) plans, or debug your SQL queries; for >details see >https://www.monetdb.org/Documentation/Manuals/SQLreference/Runtime >You might also want to consider using the MAL debugger >directly; cf., >https://www.monetdb.org/Documentation/Manuals/MonetDB/debugger >Also, a combination of mserver5's --trace and ->gdk_debug=2097152 ("ALGOMASK") >might be useful. >Best, >Stefan ----- On Jul 8, 2015, at 11:42 AM, Xu,Wenjian zeroxwj at gmail.com <https://www.monetdb.org/mailman/listinfo/developers-list> wrote: >>* Hi, *>> >>* My objective is to investigate that, given an SPJ query, how are BATs *>>* *physically* processed (changed) among relational algebra operators (e.g., *>>* select, join). I do not care about how query is translated into execution plan *>>* in MAL, what I care is the interaction of operators in the kernel (aka., gdk) . *>> >>* My current plan is to add some debug information in those operators (e.g., *>>* functions in src/gdk_join.c, src/gdk_select.c) and re-compile the system. Then, *>>* I will pose an SPJ query (through console) and check the system log to see how *>>* those operators are invoked in sequence. *>> >>* But this method seems inefficient and I would like to ask if there are any *>>* alternatives, for example, is it possible to debug the system *step-by-step* *>>* given an input query (or its logical query plan). *>> >>* Thank you very much for any kind help. *>> >>* Best regards, *>>* Wenjian *>> >> >*>*

1 0

Interaction of operators in the kernel.
by Xu,Wenjian 08 Jul '15

08 Jul '15

Hi, My objective is to investigate that, given an SPJ query, how are BATs *physically* processed (changed) among relational algebra operators (e.g., select, join). I do not care about how query is translated into execution plan in MAL, what I care is the interaction of operators in the kernel (aka., gdk) . My current plan is to add some debug information in those operators (e.g., functions in src/gdk_join.c, src/gdk_select.c) and re-compile the system. Then, I will pose an SPJ query (through console) and check the system log to see how those operators are invoked in sequence. But this method seems inefficient and I would like to ask if there are any alternatives, for example, is it possible to debug the system *step-by-step* given an input query (or its logical query plan). Thank you very much for any kind help. Best regards, Wenjian

2 1

Transaction-replication, monetdbd database properties and other changes on default
by Dimitar Nedev 23 Jun '15

23 Jun '15

Hello everyone, We've recently merged the transaction-replicaiton branch into the default. That means that you can try out the new, still experimental, transaction-replication if you build the MonetDB default branch. Since there is already a release branch for the next feature release (July2015), the transaction-replicaiton support will not be part of the next major release. In general, most changes affect a database instance if it runs as a slave, master instances should function just as before. So unless you flip the right switches (and it cannot happen accidentally*) you shouldn't notice any differences. Most code changes are in gdk_logger, where a few large functions (including logger_new) were split into smaller one to accommodate for the need of new read-only logger. That is, a logger that only reads transactions and persists them in the data store. In additional, you can now set any property to a database (mserver5) instance running under the MonetDB-Daemon (monetdbd). Previously only a few preset properties could be set via the monetdbd. The current preset properties will still function as before, both in terms of mapping monetdb-to-mserver5 names and allowed values. * I will provide more information how to turn on and try out the transaction-replication in a blog post on the website very soon. Best regards, Dimitar

1 0

New Jul2015 stable branch created
by Sjoerd Mullender 11 Jun '15

11 Jun '15

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2015. So in each of your stable checkout directories you can do hg pull hg update -rJul2015 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Oct2014-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 09 Jun '15

09 Jun '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP4 bugfix release Bug Fixes * 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. * 3707: var() possibly not working in debug builds * 3712: Concurrency issue on querying the SQL catalog * 3713: Long startup cost for simple session * 3715: Crash with two ALTER TABLE statements in a transaction * 3718: Adding and dropping a non existing tablename to/from a merge table is accepted without an error * 3719: Assertion failure in /MonetDB-11.19.11/gdk/gdk_bat.c:2841: BATassertHeadProps: Assertion `!b->H->key || cmp != 0' failed. * 3720: Incorrect results on joining with same table * 3723: Assertion failure in rel_bin.c:2548: rel2bin_groupby: Assertion `0' failed. * 3725: LEFT JOIN bug with CONST value * 3731: left shift for IP addresses not available to non-system users

1 0

Hashjoin performance with large vs small tables
by Roberto Cornacchia 12 May '15

12 May '15

Hi there, Triggered by a suspiciously slow join, I had a look at BATsubjoin, in particular at how it is decided which algorithm to use. I looked at a couple of cases with a join on strings. This is one (pseudo syntax): l.count ~16M l.T.sorted = 0 l.T.revsorted = 0 r.count = 1 r.T.sorted = 1 r.T.revsorted = 1 In this case, a hash table on r is created (because it's smaller). This join takes *430ms*. I forced swapping l and r, thus built the hash table on the larger bat, and then it takes *0.8ms*. Notice that though r.count = 1 is a bit of an extreme case, it does happen often in practice. And the case with a few tuples would not be very different. The question I wanted to ask is: is it always right to build the hash table on the small table? Perhaps some more heuristics can help? Like: if the smaller is very small, and the larger is not extremely large, then hash on the larger one? Roberto

2 12

Facing issues in implementing Embedded R in MonetDB-11.19.11
by Vishal Virmani 12 May '15

12 May '15

Hi I am implementing Embedded R in MonetDB version 11.19.11. My environment is VM with Ubuntu 14.04 Server. I have build MonetDB from source and R as well as from source. Running the command sudo mserver5 --version shows me the desired output. When i try to invoke a database using sudo monetdb start testdb1 command, where testdb1 is a database i successfully created, it repeatedly throws up an error...database 'testdb1' started up, but failed to open up a communication channel. Upon checking the associated merovingian log file i see further messages saying the object code to rapi.prelude missing. Any way to resolve this...? Regards Vishal Virmani @poisoncreed

1 0

Suggestion: Windows builds ...
by g4＠novadsp.com 11 May '15

11 May '15

Hello Might it be possible to include copies of the relevant header files for the support libraries (iconv through to zdll) in the GIT repo and/or tarballs? There is a considerable amount of pain involved in acquiring/building/installing matching versions of the lib/dll files on Windows (esp. on Windows!). Also libstream.dll does not advertise any version information which makes matching just that bit harder :) Thanks. Jerry.

1 0

Announcement: New Oct2014-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 07 May '15

07 May '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP3 bugfix release Build Environment * We now also create debug packages for Debian and Ubuntu. MonetDB Common * Replaced the rangejoin implementation with one that uses imprints if it can. Bug Fixes * 3466: UPDATE statements fails with "GDKerror: MT_mremap() failed" * 3602: Surprising overload resolution of generate_series * 3613: SQL data dictionary contains columns names which are also special keywords. This causes unexpected/unneeded SQL query errors * 3645: Network address operators such as << and <<= do not work * 3647: missing BAT for a column leads to crash in gtr_update_delta * 3648: memory corruption on unclean connection shutdown with local temporary tables * 3650: Naming of persistent BATs is fragile * 3653: PREPARE crashes mserver if unbound variable is function parameter * 3655: SQL WHERE -1 in (-1) issue? * 3656: error message after calling fitsload() * 3660: Incorrect Results for Comparison Operators on inet Datatype * 3661: Ship debug symbols for pre-built binaries * 3662: UPDATE row with row value constructor crashes monetdb server * 3663: Incorrect result ROW_NUMBER in subquery * 3664: SQLstatementIntern missing parameter when using jsonstore * 3665: inter-session starvation issue, particularly affects sys.queue * 3666: casting text column to inet truncating text column and resulting inet for first occurrence only * 3667: insert of negative value for oid column aborts mserver5 process with assertion failure * 3669: ALTER TABLE <tbl_nm> ADD CONSTRAINT <tbl_uc1> UNIQUE (col1, col2, col3) causes Assertion failure and abort * 3671: ODBC-Access on Windows 2012 does not work - E_FAIL * 3672: libbat_la-gdk_utils.o: relocation R_X86_64_PC32 against `MT_global_exit' can not be used when making a shared object * 3676: merovingian hangs trying to exit * 3677: Crash in BATgroup_internal (caused by 87379087770d?) * 3678: Ruby driver installation ignores prefix * 3680: Prepared statements fail on execution with message 'Symbol type not found' * 3684: Wrong query result set WHERE "IS NULL" or "NOT IN" clauses uses in combination with ORDER, LIMIT and OFFSET * 3687: 'bat.insert' undefined * 3688: Crash at exit (overrun THRerrorcount?) * 3689: No more connections accepted if a single client misbehaves * 3690: find_fk: Assertion `t && i' failed. * 3691: conversion of whitespaces string to double or float is accepted without an error during insert * 3693: algebra.join undefined (caused by non-existing variables in the plan) * 3696: Inconsistent behavior between dbl (SQL double) and flt (SQL real) data types and across platforms * 3697: mserver5[26946]: segfault at 0 ip 00007f3d0e1ab808 sp 00007f3cefbfcad0 error 4 in lib_sql.so[7f3d0e180000+16c000] * 3699: segfault again! (during last week I found 3 segfault bugs already) * 3703: INSERT INTO a MERGE TABLE crashes mserver5 * 3704: Unknown identifier from subquery * 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. * 3706: Assertion failure in gdk_bat.c: BATassertHeadProps: Assertion `!b->H->sorted || cmp <= 0' failed. * 3709: "BATproject: does not match always" on abusive use of ALTER TABLE SET READ ONLY

1 0

C++ compiling Fixes
by Bastian Köcher 27 Apr '15

27 Apr '15

Hi, I want to write an optimizer for MonetDB in C++. So I included the necessary headers into my project, but I ran into some compile errors and warnings. I created a patch which solves some of these problems. Regards, Bastian

1 0