- developers-list - monetdb.org

Columnar Storage layout of MonetDB
by Senthil Vidhiyakar 21 Mar '17

21 Mar '17

I want to know the file layout of columnar data storage in MonetDB. I know each column is stored in separate files. Actually I want to know the exact layout as in a block diagram to get a clear picture of the storage layout. Helps are appreciated. Regards, Senthil Vidhiyakar. S

4 5

Announcement: New Dec2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Mar '17

03 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP2 bugfix release Bug Fixes * 4034: argnames array in rapi.c has fixed length (that was too short) * 6080: mserver5: rel_bin.c:2391: rel2bin_project: Assertion `0' failed. * 6081: Segmentation fault (core dumped) * 6082: group.subgroup is undefined if group by is used on an expression involving only constants * 6111: Maximum number of digits for hge decimal is listed as 39 in sys.types. Should be 38 as DECIMAL(39) is not supported. * 6112: Crash upgrading Jul2015->Jun2016 * 6130: Query rewriter crashes on a NULL pointer when having a correlated subquery * 6133: A crash occurs when preparing an INSERT on joined tables during the query semantic phase * 6141: Getting an error message regarding a non-GROUP-BY column rather than an unknown identifier * 6177: Server crashes * 6186: Null casting causes no results (silent server crash?) * 6189: Removing a NOT NULL constraint from a PKey column should NOT be allowed * 6190: CASE query crashes database * 6191: MT_msync failed with "Cannot allocate memory" * 6192: Numeric column stores wrong values after adding large numbers * 6193: converting to a smaller precision (fewer or no decimals after decimal point) should round/truncate consistently * 6194: splitpart returns truncated last part if it contains non ascii caracters * 6195: Cast from huge decimal type to smaller returns wrong results * 6196: Database crashes after generate_series query * 6198: COALESCE could be more optimized * 6201: MonetDB completely giving up on certain queries - no error and no result * 6202: querying a table with an ordered index on string/varchar column crashes server and makes server unrestartable! * 6203: copy into: Failed to import table Leftover data 'False' * 6205: Integer addition overflow * 6206: casting strings with more than one trailing zero ('0') to decimal goes wrong * 6209: Aggregation over complex OR expressions produce wrong results * 6210: Upgrading a database from Jun2015 or older crashes the server * 6213: SQLsmith causes server to crash * 6217: Segfault in rel_optimizer (sqlsmith) * 6218: grouped quantiles with all null group causes following groups to return null * 6224: mal_parser: cannot refer to types containing an underscore

1 0

WARNING for users of the default branch
by Sjoerd Mullender 02 Mar '17

02 Mar '17

Databases created with the default branch before changeset f054b7310be7 (Mar 1, 2017) are *not* compatible with the default branch from that changeset on. The changeset is a merge of fixes from the Dec2016 branch, and one of the fixes introduced a version number update of the BBP.dir file. This update was incompatible with an earlier update of the version number that was exclusive to the default branch. There are two possible solutions: - dump your database(s) with a version from before the chnangeset, and restore with a version after. - manually update the version number in the BBP.dir file and ignore the possibility that the bug that was fixed may be in your database. Manually fixing the BBP.dir file involves incrementing the number on the first line of the file from 25118 to 25119. Needless to say, this is not recommended. Remember, there are no guarantees when using the default branch. In fact, there are also no guarantees when using any other branch. The only thing we promise is that we will do our best to make sure the actual releases work and can be upgraded properly. -- Sjoerd Mullender

1 0

monetdb.org system maintenance
by Sjoerd Mullender 01 Mar '17

01 Mar '17

Tomorrow, March 2, we will be upgrading the hardware for the system on which all monetdb.org infrastructure runs. Expect some downtime. -- Sjoerd Mullender

1 0

Announcement: New Dec2016-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 26 Jan '17

26 Jan '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP1 bugfix release Bug Fixes * 4039: Slow mserver5 start after drop of tables (> 1 hour) * 4048: Segfault on vacuum with parallel updates * 6079: pushselect optimizer bug on MAL snippet * 6140: INNER JOIN gives the results of a CROSS JOIN * 6150: Query giving wrong results, extra records are appearing * 6175: The program can't start because python27.dll is missing from your computer. * 6178: AVG + GROUP BY returns NULL for some records that should have results * 6179: mergetable optimizer messes up sample * 6182: sys.shutdown triggers assertion in clients.c * 6184: Incorrect result set - Extra records in result set

1 0

python3 client bug+fix
by Bas Bloemsaat 23 Dec '16

23 Dec '16

Hi, Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3. The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply. Regards, Bas Bloemsaat --- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1

2 2

Announcement: New Dec2016 Feature release of MonetDB suite
by Panagiotis Koutsourakis 21 Dec '16

21 Dec '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016 feature release MonetDB5 Server * Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y. * Removed command bbp.getHeat(). * Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead. * Removed function BKCappend_reverse_val_wrap: it was unused. * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. * Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces Build Environment * New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB. * With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore. * Removed obsolete code associated with long gone static linking option. * Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture. . * The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github (https://github.com/gijzelaerr/pymonetdb) the other clients are maintained by CWI/MonetDB on our own server (https://dev.monetdb.org/hg/monetdb-java, https://dev.monetdb.org/hg/monetdb-perl, https://dev.monetdb.org/hg/monetdb-php) MonetDB Common * The tnokey values must now be 0 if it is not known whether all values in a column are distinct. * The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique. * Implemented conversion to str from any type (not just the internal types). * VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure. * BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8. * BATattach now copies the input file instead of "stealing" it. * Removed the lastused "timestamp" from the BBP. * Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function. * Removed unused functions BBPhot and BBPcold. * Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop. * Removed the macro BUNfirst. It can be replaced by 0. * Changed BATroles by removing the argument to set the name of the head column. * The head column is now completely gone. MonetDB is completely "headless". * The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase. * BATs now only have a single (logical) name. * The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column. * BATkey now works on the TAIL column instead of the HEAD column. * Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL. * Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID). * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. SQL * Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y. * The experimental recycler code is moved to the attic. * The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.) Bug Fixes * 3357: Implement setQueryTimeout() * 3445: Add support for database name to dotmonetdb file * 3597: SQL to MAL listing looses types * 3973: JDBC hangs * 3976: Performance enhancement to LIKE without wildcards * 4005: Correlated update causes incorrect null constraint violation * 4016: merge table only optimises for point query * 4040: sys.storage call can take a long time * 4047: Segfault when updating a dropped table * 4050: Database corruption when running low on inode * 4057: missing bulk operations between constant and bat * 4061: SIGSEGV in candscan_lng * 4066: Deadlocked monetdbd * 6068: Error message about incompatible BBP version should be clearer * 6069: query with union all silently crashes * 6070: setting negative session query timeout should not be possible/allowed * 6071: where clause with cast and floor fails to sigsegv * 6072: Bind to UPD delta column does not get/show type information in EXPLAIN * 6073: Missing type information for constants in MAL explain * 6074: SET ROLE command does not work * 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed. * 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed. * 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed. * 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed. * 6084: Merge table point to wrong columns if columns in partition tables are deleted * 6108: monetdb5-sql sysv init script does not wait for shutdown * 6114: segfault raised in the query rewriter due to a null pointer * 6115: Assertion hit in the codegen * 6116: Codegen does not support certain kind of selects on scalar subqueries * 6117: Assertion hit in the query rewriting stage during the push down phase * 6118: SIGSEGV in strPut due to shared heap * 6119: Assertion hit in the MAL optimiser on a complex query * 6120: QUANTILE() treats NULL as if it is zero * 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result * 6123: Assertion hit in the codegen #2 * 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result * 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression * 6126: batcalc.== can't handle void BATs * 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom

1 0

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka)
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded >joweria@ubuntu:~$ joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded But when i try to reach another database as shown below, the module is not loaded joweria@ubuntu:~$ mserver5 --dbpath=/home/joweria/Spartial/mydbfarm/Gisbase !FATAL: GDKlockHome: Database lock '.gdk_lock' denied joweria@ubuntu:~$ sudo su [sudo] password for joweria: root@ubuntu:/home/joweria# mserver5 --dbpath=/home/joweria/ Spartial/mydbfarm/Gisbase # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'Gisbase', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded > Also when i run with my user priviledges

1 0

Re: developers-list Digest, Vol 52, Issue 1
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded 2016-12-01 13:00 GMT+02:00 <developers-list-request(a)monetdb.org>: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) > 2. Re: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 1 Dec 2016 12:11:49 +0200 > From: Joweria Nakibuuka <joweria.nakibuuka(a)unitn.it> > To: developers-list(a)monetdb.org > Subject: RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > Message-ID: > <CANbNUqbw+N3=tb929XArQ8q+AiCG-o5DYNR9Ndk2POChHpdi1g@ > mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Admin, > I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with > 128bit integers. i recently installed MonetDB5 default branch from the > mercurial repository. > > I tried to rebuild and enable geom module but to no avail, it shows no > error, after compilation it shows that the module is enabled,However when i > run the mserver the monetDB/GIS module is not loaded as expected. I am > stuck and i don't seem to know how to proceed forward in order to run my > spatial queries > > Are there any setting i need to do explicitly to load this module? > > the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 > --enable-geom=yes --enable-pyintegration=yes --with-geos=yes > > and then after compilation under the LDFLAGS-enabled componets > geom is one of the enabled componet but when i run the mserver5 the module > is not loaded as expected. > > I rebuild monetDB again and for first time i run the mserver5 commans > and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is > loaded) but when i closed the session (\q) and ran the same command again > the module was not loaded. i tried exporting paths to no avail. what could > be the problem? please i need your kind help. > > Thank you > > Regards, > Joweria nakibuuka >

2 2

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

Dear Admin, I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with 128bit integers. i recently installed MonetDB5 default branch from the mercurial repository. I tried to rebuild and enable geom module but to no avail, it shows no error, after compilation it shows that the module is enabled,However when i run the mserver the monetDB/GIS module is not loaded as expected. I am stuck and i don't seem to know how to proceed forward in order to run my spatial queries Are there any setting i need to do explicitly to load this module? the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 --enable-geom=yes --enable-pyintegration=yes --with-geos=yes and then after compilation under the LDFLAGS-enabled componets geom is one of the enabled componet but when i run the mserver5 the module is not loaded as expected. I rebuild monetDB again and for first time i run the mserver5 commans and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is loaded) but when i closed the session (\q) and ran the same command again the module was not loaded. i tried exporting paths to no avail. what could be the problem? please i need your kind help. Thank you Regards, Joweria nakibuuka

2 1