- users-list - monetdb.org

Re: users-list Digest, Vol 89, Issue 2
by Kalaiarasi Gowtham 13 Jan '20

13 Jan '20

Hi Jenni, Please, find the below steps I have followed to install monetdb *Installation of MonetDB in CentOS 7* *Installing dependencies* # yum update # yum install python autoconf automake libtool bison openssl openssl-devel gettext-devel git # git clone https://github.com/MonetDB/MonetDB.git # cd /MonetDB # ./bootstrap # cd .. # mkdir testdir # cd testdir *To enable py3integration python version 3 should be installed* # yum install centos-release-scl # yum install rh-python36 # scl enable rh-python36 bash *check python version* # python -V then # ../MonetDB/configure –enable-py3integration=yes # make # make install Thanks and regards, Kalai On Wed, 8 Jan 2020 at 16:31, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: pymonetdb issue (ferreira) > 2. Re: Monetdb (Ying Zhang) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 7 Jan 2020 17:22:59 +0000 > From: ferreira <pedro.ferreira(a)monetdbsolutions.com> > To: Communication channel for MonetDB users <users-list(a)monetdb.org>, > ALEX OSSIPOV <alexo15(a)comcast.net> > Subject: Re: pymonetdb issue > Message-ID: > <0e74a920-b2ad-2993-b1dc-fcd7a49105bb(a)monetdbsolutions.com> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Hello Alex, > > I just looked at pymonetdb source code and indeed it does not support > IPv6 yet. I'm going to create an issue to request for it. The reason is > because in the past, the MAPI protocol (which MonetDB uses for socket > connections) didn't support IPv6, so the clients (Python, Ruby, JDBC, > etc.) never had to implemented them. However they can now add this option. > > Best regards, > > Pedro > > On 12/22/19 5:30 AM, ALEX OSSIPOV wrote: > > Hi All > > I installed pymonetdb v 1.2.1 and trying to connect to remote > > server?MonetDB v11.35.3 (Nov2019) listening on IPV6 only. I have got > > next error in python client: > > >>> mapi_connection.connect(hostname="<server_name>", port=xxxxx, > > username="monetdb", password="monetdb", database="demo", > > language="sql", unix_socket=None, connect_timeout=-1) > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > File "/usr/local/lib/python3.7/site-packages/pymonetdb/mapi.py", line > > 143, in connect > > self.socket.connect((hostname, port)) > > socket.gaierror: [Errno -2] Name or service not known > > >>> > > My pc client is also using ipv6 to connect to MonetDB server. I also > > have jdbc connection that works fine. > > Question: > > Does it mean pymonetdb does not support ipv6 only environment? > > What would be the reason if it has nothing to do with protocol? > > > > Thanks > > Alex > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list >

2 2

Monetdb
by Kalaiarasi Gowtham 07 Jan '20

07 Jan '20

Hi, Currently, we are trying to start the database created in monetdb using the following command monetdb start <dbname> monetdbd: internal error while starting mserver 'database <dbname> did not initialise the sql scenario', please refer to the logs Getting logs as follows [root@localhost mydbfarm]# tail -f merovingian.log 2020-01-03 16:03:14 MSG merovingian[13157]: database <dbname> has shut down 2020-01-03 16:03:14 ERR control[13157]: !monetdbd: an internal error has occurred 'database <dbname> did not initialise the sql scenario' 2020-01-03 16:03:18 ERR merovingian[13157]: client error: database <dbname> did not initialise the sql scenario 2020-01-03 16:03:18 MSG merovingian[13157]: database <dbname> (13263) has exited with exit status 0 Please,help us to solve this issue. Thanks Kalai

2 1

Announcement: New Nov2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Dec '19

20 Dec '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019-SP1 bugfix release (11.35.9) Bug Fixes * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6724: Prepare confuses types when more than one argument is used * 6726: Python aggregation does not create aggr_group when aggregating over all rows * 6765: GRANT SELECT privilege on a subset of table columns results in access denied error when selecting the same columns from the table * 6790: Count distinct giving wrong results * 6791: str_to_time('11:40', '%H:%M') creates wrong time value * 6792: JSON path compiler accepts invalid input * 6793: cast(interval second value to int or decimal) is wrong (by a factor of 1000), cast(interval month value to decimal or floating point) fails * 6794: external name fits.listdir not bound (sys.listdir) Fatal error during initialization: * 6796: Incorrect crash time reported by monetdb tool after crash of mserver5 * 6798: json.text off by one error * 6804: DNS resolution of 0.0.0.0 fails on recent Ubuntus

1 0

resetting credentials from mserver5?
by Jonathan Speiser 20 Dec '19

20 Dec '19

Hello, I had tried changing the password on my monetdb server (MonetDB 5 server v11.29.3 "Mar2018") but after rebooting it seems I am locked out from mclient. Is there any way I can change the monetdb user password from mserver5? Thank you for your help Jon

2 1

starting database 'tpch'... FAILED
by DEMBELE Simon Pierre 16 Dec '19

16 Dec '19

-------- Original Message -------- Subject: starting database 'tpch'... FAILED Date: 16/12/2019 12:58 From: DEMBELE Simon Pierre <simon-pierre.dembele(a)ensma.fr> To: mailman-owner(a)monetdb.org Bonjour, Hello I need help; Why can solve this issue starting database 'tpch'... FAILED start: starting 'tpch' failed: database 'tpch' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details Thank -- Simon

2 2

MERGE TABLE - CANNOT DROP PARTITION
by Ariel Abadi 10 Dec '19

10 Dec '19

Hi! I'm still having issue dropping internal tables from the MERGE TABLE. I used to have this problem on a previous version. I have created a new DB, imported all the data... and now I need to delete a partition (NOT a table). I'm still facing the same problem, which is that I cannot remove it from the merge table. (as you can see below).. Is there anything I can do? Rgds Ariel Welcome to mclient, the MonetDB/SQL interactive terminal (Apr2019) Database: MonetDB v11.33.3 (Apr2019), 'mapi:monetdb://dt-mn:54321/cli_dg' Type \q to quit, \? for a list of available commands auto commit mode: on sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful

2 1

CREATE DATABASE
by Ariel Abadi 09 Dec '19

09 Dec '19

Hi! Im trying to start a new fresh database .. I am creating with the following commands: monetdbd create /opt/starbi/mm/ monetdbd start /opt/starbi/mm/ monetdb create cli_dg monetdb start cli_dg When I try to start the new database... the database is NOT starting. starting database 'cli_dg'... FAILED start: starting 'cli_dg' failed: database 'cli_dg' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details *Here is the log:* 2019-12-09 15:59:07 MSG cli_dg[385]: # loading sql script: 72_fits.sql 2019-12-09 15:59:07 ERR cli_dg[385]: #main thread:!ERROR:ParseException:SQLparser:3F000!CREATE PROCEDURE: external name fits.listdir not bound (sys.listdir) 2019-12-09 15:59:07 ERR cli_dg[385]: ParseException:SQLparser:3F000!CREATE PROCEDURE: external name fits.listdir not bound (sys.listdir)Fatal error during initialization: 2019-12-09 15:59:07 ERR cli_dg[385]: ParseException:SQLparser:3F000!CREATE PROCEDURE: external name fits.listdir not bound (sys.listdir) 2019-12-09 15:59:11 MSG merovingian[370]: database 'cli_dg' (385) has exited with exit status 1 2019-12-09 15:59:11 ERR control[370]: (local): failed to fork mserver: database 'cli_dg' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details Can you please help!?

4 5

COPY INTO ... LOCKED
by Roberto Cornacchia 02 Dec '19

02 Dec '19

Hi, The documentation about the LOCKED modifier for COPY INTO mentions: *When using the LOCKED mode, it is the responsibility of the database user to make sure that there is currently only a single client connection accessing the database* I was assuming (and the syntax suggests, I think) that the LOCKED refers to the table being copied into, not to the entire database. That is, only that table is updated without the overhead of logging. But the documentation imposes the very restrictive condition that only one connection is active on the database. That seems to suggest that the entire database is being updated without logging, not only that table. Could you please clarify? Can I do COPY INTO LOCKED safely with more than one connection, as long as only 1 connection touches that table? Roberto

1 0

Issue in using Monet/Python loader
by Kalaiarasi Gowtham 29 Nov '19

29 Nov '19

Hi, While trying to use Monet/Python Loader function,we are getting the following error *pymonetdb.exceptions.OperationalError: 'pyapi.eval_loader' undefined in: pyapi.eval_loader(0x7f34f01f2c60:ptr, "{_emit.emit( { 'event_date': '2019-10-10', 'status': 1})};":str);* *Followed the below steps for installation* *Python installation steps:* We have installed python 3.6 1)yum install autoconf 2)yum install automake 3)yum install libtool 4)yum install openssl 5)yum install openssl-devel 6)yum install python-devel 7)Configured environment path settings for python export PYTHONPATH=$PATH:/usr/local/lib/python3.6/site-packages *Monet installation steps:* 1)git clone https://github.com/MonetDB/MonetDB.git 2)cd /MonetDB 3)./bootstrap 4)cd.. 5)mkdir testdir 6)cd testdir 7)../MonetDB/configure --enable-pyintegration=yes While configuring we get the following *py3integration is enabled* 8)make 9)make install 10)monetdbd create /path/to/mydbfarm 11)monetdbd start /path/to/mydbfarm 12)monet stop -a 13)monet set embedpy=true 14)monet start -a 13)Enabled numpy(pip install numpy) *Created a loader by using the below python script* import pymonetdb import sys import os connection = pymonetdb.connect(username="xyz", password="xyz123", hostname="ipaddress", database="testdb") cursor = connection.cursor() cursor.execute("CREATE LOADER myloader() LANGUAGE PYTHON {_emit.emit( { 'event_date': '2019-10-10', 'status': 1})};") #create loader cursor.execute("COPY LOADER INTO store FROM myloader();") #append the row from loader to table connection.commit() *While running the above script getting the following error* pymonetdb.exceptions.OperationalError: 'pyapi.eval_loader' undefined in: pyapi.eval_loader(0x7f34f01f2c60:ptr, "{_emit.emit( { 'event_date': '2019-10-10', 'status':1})};":str); Please, help us to fix this issue. Thanks & Regards Kalaiarasi A

2 1

Announcement: New Nov2019 Feature release of MonetDB suite
by Sjoerd Mullender 28 Nov '19

28 Nov '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019 feature release (11.35.3) MonetDB5 Server * The server "console" has been removed, as has the --daemon option. The server now doesn't read from standard input anymore. The way to stop a server is by sending it a TERM signal (on Linux/Unix) or by sending it an interrupt signal (usually control-C -- on all systems). * Implemented a function bat.diffcand to calculate difference of two candidate lists. * The mtime module was completely rewritten, the atom types date, daytime, and timestamp were changed. Upgrade code for BATs containing these types has been implemented. The old daytime type used a 32 bit integer to record milliseconds since the start of the day. The new daytime type uses a 64 bit integer to record microseconds since the start of the day. The old date type recorded days since or before the year 1. The new daytime type records the day of the month and the number of months since the year -4712 separately in a single 32 bit integer of which only 26 bits are used. Dates now use the proleptic Gregorian calendar, meaning the normal Gregorian callendar has been extended backward, and before the year 1, we have the year 0. Both the old and new timestamp types are a combination of a daytime and a date value, but since those types have changed, the timestamp type has also changed. The new date type has a smaller range than the old. The new date range is from the year -4712 to the year 170049. During conversion of date and timestamp columns, the dates are clamped to this range. * The tzone and rule atom types have been removed. They were not used by any code, and they were defined in a non-portable way. * Added "mapi_ipv6" property to monet_options to force ipv6 address binding only. This property is inherited while forking from monetdbd if it is also set there. * Removed (bat)calc.between_symmetric and changed (bat)calc.between by adding a number of extra arguments, all of type :bit: symmetric, low inclusive, high inclusive, nils false. Merovingian * Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively. * Added ipv6 property to monetdbd properties to force IPv6 addresses binding only. By default this property is false to allow IPv4 addresses as well. MonetDB Common * BATcalcbetween and all its variants now have an extra bool parameter "anti" to invert the test. * All forms of BATcalcbetween and VARcalcbetween have extra bool arguments for low inclusive, high inclusive and nils false. The latter causes the result to be false instead of nil if the value being checked is nil. SQL Frontend * Removed functions json.text(string) returns string and json.text(int) returns string. Their MAL implementation didn't exist, so they were meaningless. * There are new aggregate functions sys.median_avg and sys.quantile_avg that return the interpolated value if the median/quantile doesn't fall exactly on a particular row. These functions always return a value of type DOUBLE and only work for numeric types (various width integers and floating point). * Added sys.deltas ("schema" string, "table" string, "column" string) returns table ("values" bigint) system function which returns a single column with 6 values: a flag indicating if the column's upper table is cleared or not, the count of the RDONLY, RD_INS and RD_UPD_ID deltas of the column itself, the number of deleted values of the column's table, as well as the level of the current transaction in the transaction level tree. It should be used for debugging purposes only. * Added "VALUES row_list" statement as a top SQL projection statement. * The implementation of in-expression now follows a join-based approach instead of using iterative union/selects. This greatly improves performance for large in-value-lists. Furthermore the old approach has large in-value-lists generate large MAL plans. This is now no longer the case. * Strings are now limited to 1GB, double-quoted tokens are limited to 2kB. These sizes are bytes of (UTF-8 encoded) input data. Bug Fixes * 3533: SQL aggregate functions avg(), sum() and median() return an error when used on a column with datatype interval second * 6134: Query produces error: HEAPalloc: Insufficient space for HEAP of 1168033427456 bytes. * 6613: LATERAL crash /.../rel_bin.c:1473: rel2bin_table: Assertion `0' failed. * 6683: Bug in subselect * 6686: Bug in subselect (count function) * 6688: Bug in subselect (or condition) * 6689: Trying to improve the performance of SQL queries with a large list of members in IN clause. * 6695: timestamp transformation * 6700: Monetdb Bugs in Subselect statements: * 6722: window functions issues * 6740: while upgrading the database from verison (MonetDB-11.27.13) to (MonetDB-11.33.3) we are unable to bring up the database * 6754: in mclient a strang msg is reported after issueing command: set schema sys; * 6755: Assertion failure in rel_bin.c * 6756: Error in optimizer garbageCollector on merge tables select * 6757: Double free or corruption (out) * 6758: SIGSEGV in __strcmp_sse2_unaligned() * 6759: Python JSON loader creates invalid data type for strings * 6761: Error: Program contains errors.:(NONE).multiplex * 6762: mserver5 crashes on (re-)start * 6764: mserver5 crashes with corruption, double free, invalid size or invalid pointer * 6766: Missing bulk implementation for get_value and next_value calls * 6769: ProfilerStart is not threadsafe * 6771: R-devel * 6772: TRACE includes information from all active sessions * 6773: json.filter returns corrupted string when selecting JSON null value * 6774: PROD aggregation gives wrong result * 6775: NOT IN with an AND containing an OR gives wrong result * 6776: Creating a table with a full outer join query gives type with wrong digits on the joined key. * 6779: Using Windows Messages translation for errno error codes. * 6780: Wrong value of the rank function * 6781: Insert after index creation crash * 6783: AVG changes scale of its results * 6784: function sys.isauuid(string) should return false if string value cannot be converted to a UUID

1 0

UDF bulk implementation with some arguments as scalars
by Roberto Cornacchia 05 Nov '19

05 Nov '19

Hi, When defining a UDF, e.g. CREATE FUNCTION myfunc(a int, b int) RETURNS int we can have: - a scalar implementation command myfunc(a:int, b:int) :int - a bulk implementation command myfunc(a:bat[:int], b:bat[:int]) :bat[:int] - a mixed implementation (e.g. a is a column, b is scalar): command myfunc(a:bat[:int], b:int) :bat[:int] I'm trying to understand what the restrictions are, if any, for the third case. Is it expected to be allowed in (a combination of) the following cases? - Interleaved bulk/scalar arguments command myfunc(a:bat[:int], b:int, c:bat[:int]) :bat[:int] - Table-returning function command myfunc(a:bat[:int], b:int) (:bat[:int], :bat[:int]) - Table-returning function, used with a sub-select in input SELECT * FROM myfunc( (SELECT a, 10 FROM mytable) ); In particular I can't get the last one to work. Even though the sub-select has a constant as the second column, this constant seems to be first replicated into a full column and then given to myfunc() as a column. When the constant parameters are a few, that becomes a waste of time/space. Plus the annoyance of selecting the first value of a constant column for the actual internal implementation, which of course expects a scalar for those arguments. Am I missing something? Is there a way to make that work as I would expect? Thanks, Roberto

4 5

Report a bug
by lhy1149 04 Nov '19

04 Nov '19

Hello, I don't know if it's appropriate, but because I can't register monetdb's bugzilla, I'm going to report a bug here. This problem causes the insert statement to hang and mserver5 takes up 100% of the CPU time. Here is the code that caused the problem: Create table a(a int, b int, id bigserial); -- comment any of the next 2 lines will resolve the problem Create ordered index a_pk on a(id); Create index a_idx1 on a(a); Insert into a(a) values(1); -- this line will hang Enviroment: macOS Catalina 10.15 MonetDB v11.33.11 I hope the development team can check if there is this problem. Thank you

3 2

MERGE TABLE - CANNOT DROP PARTITION
by Ariel Abadi 28 Oct '19

28 Oct '19

Hi! Can someone help me? Im trying to REMOVE a table from a merge table, when I run the command, it states "operation successful", but the table is NOT removed. Here you can see an example... I have run it several times, but the table is still there :( [image: image.png]

4 7

Guide to installing from source on a FC30 machine with SELinux enabled.
by Arjen P. de Vries 10 Oct '19

10 Oct '19

Dear all, For what it's worth, here a report on building MonetDB from source on a Linux FC30 system with SELinux enabled. Thanks to Sjoerd for some crucial hints in getting it to work, and Roberto for the encouragement to not give up :-) First, build as usual: ./bootstrap ./configure --disable-debug --disable-developer --disable-assert --enable-optimize make -j Now, skip the usual `make install` and build rpm packages instead - this includes a single rpm that applies (almost) all the SELinux policies MonetDB needs. (Missing ones should be reported as bugs.) make rpm Install the created rpm packages in two steps, such that all directories that need policies have indeed been created (the SELinux rpm is under the noarch tree): find rpmbuild/RPMS/`uname -m` -name \*.rpm | xargs sudo dnf --disablerepo="*" install --skip-broken -y find rpmbuild/RPMS/noarch -name \*.rpm | xargs sudo dnf --disablerepo="*" install --skip-broken -y Start the server: sudo systemctl status monetdbd You can always uninstall the rpm packages with some more shell magic (replace x86_64 by `uname -m`): find rpmbuild/RPMS -name \*.rpm | sed -e 's/.*\/$MonetDB.\+fedora30$$\.x86.64\|\.noarch$\?\.rpm/\1/g' | xargs sudo dnf --disablerepo="*" -y rm This was sufficient for my, but if you are struggling with SELinux warnings/errors, it may be useful to know the following basics: You can check assigned filecontexts using ls -alRZ /var/monetdb5 These should align with the file contexts specified in monetdb.fc: grep monetdb5 /usr/share/doc/MonetDB-selinux/monetdb.fc Subsequently install those filecontexts and apply them as follows; e.g., for `.merovingian_properties` you would do: sudo semanage fcontext -a -t monetdbd_etc_t /var/monetdb5/.merovingian_properties sudo restorecon -v /var/monetdb5/.merovingian_properties Remaining issues are the following: + `make -j rpm` does not seem to work correctly. + `MonetDB-client-test` package does not want to install with the above command due to the --disablerepo flag. + SELinux complained about access to resource `cpu` - fixed by using the suggested `audit` in the error message. Good luck, hope it saves some people time! Arjen -- ==================================================================== ICIS, office M1.02.07 Radboud University Mercator 1 Faculty of Science Toernooiveld 212 arjen(a)cs.ru.nl NL-6525 EC Nijmegen, The Netherlands +31-(0)24-365 2354 ===================== http://www.informagus.nl/ ====================

1 0

trace not session-isolated?
by Roberto Cornacchia 09 Oct '19

09 Oct '19

On our production servers we keep a pre-defined number of sessions open to a database. Incoming queries are distributed among these sessions. I create an mclient connection to that database and I run a trace (any query). The trace displays all functions created by the queries belonging to the other sessions. All together concatenated (or maybe interleaved, I'm not sure) in the output of my trace. I expect this is not meant, is it?

1 0

Announcement: New Apr2019 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '19

26 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019 feature release (11.33.3) MonetDB5 Server * Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first. * Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8. * Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible. * The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter. Build Environment * We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line. * We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old). SELinux Support * There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories. Merovingian * Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it. Client Package * The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q. Mapi Library * Removed function mapi_cache_shuffle. * Removed function mapi_stream_query. Stream Library * The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.). * Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h. * Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream). * Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped. * Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'. MonetDB Common * During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts. * Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed). * Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending). * Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter). * The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'. * Function PROPdestroy now takes a BAT* parameter. * A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value. SQL Frontend * Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments. * Implemented SQL2003 natural logarithm function: ln(num_expr). * Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument). * Implemented length(blob) and comparison between blobs. * Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base. * Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm). * Removed unused global SQL variable "history". * Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong. * Allow usage of WITH clauses in insert, update, delete and merge statements. * Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] } * Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident * Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5) * Improved and extended storagemodel functionality. Changes include: * Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema. * Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns. * In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns. * System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize(). * System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya. * System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate. * System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead. * Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names. * Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table. * Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns. * Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value). * The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;). * Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);). * Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation. * Implemented X'...' style binary string literals. * Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation. * Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release. * Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending). * Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements. * The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>] * Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified. * The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated. * Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition. Bug Fixes * 2403: stream: http read support in streams * 2416: GDK: file/dir creation mask * 2478: SQL: rename columns * 2496: SQL: implement greatest / least * 3384: Auxiliary window functions * 3416: log() SQL function only supports natural logarithm * 3448: Make lbatomic_ops optional if installed * 3520: Add support for scalar function LN(num) * 3530: sigabort on window aggr function * 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename * 3743: DELETE FROM merge table not supported. * 3832: Cannot use expressions in GROUP BY clause * 3946: Expose C log2 function to SQL * 4056: mapi_timeout is not implemented * 6181: comments cause error with return in function * 6326: Eliminate unused UNION tables * 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB * 6475: Remove unused SQL global variable "history" * 6588: Table aliasing are not supported in SQL update queries * 6591: Name mangling in the SQL catalog results in ambiguity * 6608: Sqlitelogictest error message in group by with coalesce * 6609: Rename Schema in MonetDB * 6636: sys.queue() extension: allow DBA to see all running queries * 6647: Add suport to Python 3 on Windows * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6671: Error when running user function on merge table with remote part * 6674: R UDF with Date type fails to convert RAPI * 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids) * 6678: Binding NULL parameter to parametrized query results in syntax error on execution * 6680: Copy cannot open CSV file if systemctl enabled * 6684: Inserting into a replica table crashes the server * 6685: adding a view to a merge table gives unexpected behaviour * 6690: Unable to fetch empty string with ODBC driver * 6696: Re-use of MAL explain plans with complex type coercions

2 1

stethoscope not redirecting time column correctly
by Jiading Guo 17 Sep '19

17 Sep '19

Hi, When setting stethoscope dumping to disk by `stethoscope -d <db-name> -o <output-file>`, the time column is printed in the console instead of the file. In `MonetDB/clients/mapiclient/stethoscope.c` L113, it directly prints to stdout by `printf("\"%s\", ", ev->time);`. Is it on purpose? Also, I can't follow this message[1] to profile MAL because it seems that the APIs have been changed. Where can I find the newest document? Thanks, Jiading Guo [1]: https://www.monetdb.org/pipermail/users-list/2013-May/006369.html

2 2

!SQLException:sql.bind:42000!Column type mismatch
by Jiading Guo 09 Sep '19

09 Sep '19

Hi, I'm doing some tests about bind operation with v11.33.11 (Apr2019-SP1) and here's how to reproduce: ``` mclient -d test sql>create table unary(a int); sql>insert into unary(a) select value from generate_series(1, 101, 1); mclient -d test -l mal mal>sql.init(); mal>s := sql.mvc(); mal>a := sql.bind(s, "sys", "unary", "a", 0); ``` Then I got `!SQLException:sql.bind:42000!Column type mismatch`. I traced it down to `sql/backends/monet5/sql.c:mvc_bind_wrap`, `int coltype = getBatType(getArgType(mb, pci, 0));` coltype is set to 255 here, which confused me about what this coltype is used for. Also in `sql/backends/monet5/sql.mal`, the comment says the access kind (last argument) is divided into `0, base table`, `1, inserts` and `2, updates`. In what scenarios are these values used? Is `0` for read only operations? What about delete operations? Thanks for your help. Best regards, Jiading Guo

2 2

Latest release (11.33.11) still doesn't honor numactl
by Gotwisner, Dave 04 Sep '19

04 Sep '19

I have just upgraded MonetDB on an 88 core CentOS box - 2 cpu modules. I was also one of the people who originally found the numa issue. The system doesn't behave as I would expect. Here is the sequence of commands I execute to start up MonetDB (overkill, but creating a clean database for the test): 1. numactl --cpunodebind=1 --membind=1 monetdbd create /srv/zfsdata/monetdbTesting/nTC 2. numactl --cpunodebind=1 --membind=1 monetdbd set listenaddr=0.0.0.0 /srv/zfsdata/monetdbTesting/nTC 3. numactl --cpunodebind=1 --membind=1 monetdbd set discovery=false /srv/zfsdata/monetdbTesting/nTC 4. numactl --cpunodebind=1 --membind=1 monetdbd set control=false /srv/zfsdata/monetdbTesting/nTC 5. numactl --cpunodebind=1 --membind=1 monetdbd start /srv/zfsdata/monetdbTesting/nTC 6. numactl --cpunodebind=1 --membind=1 monetdb -p 50000 create nTC 7. numactl --cpunodebind=1 --membind=1 monetdb -p 50000 release nTC If I launch MonetDB using the monetdbd command as "numactl -cpunodebind=1 -membind=1", that means that we are assigning only 44 cores for use with MonetDB (mserver5). I would expect that either command 1 or 5 is the one that spawns mserver5. What I then see with numactl --show --pid=`pidof mserver5` is: policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 cpubind: 0 1 nodebind: 0 1 membind: 0 1 I would have expected that if we launch mserver5 via the monetdbd wrapper (as is what's documented in the tutorials), that the numa settings would be honored. Note also, I would expect 44 threads (plus a few extra) be set rather than 88 (plus a few extra). We are seeing 96 threads. Looking at the useful part of the /proc/`pidof mserver5`/status file gives the following: Cpus_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00fffffc,00000fff,ffc00000 Cpus_allowed_list: 22-43,66-87 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 Which implies it IS putting the program on 44 cores, but still creating 96 threads. It also shows that it isn't honoring the memory constraint? Any suggestion on how to correctly launch so it honors our numa requirements? Thanks, Dave

2 1

Possible problem with C++ based CREATE FUNCTION
by Gotwisner, Dave 04 Sep '19

04 Sep '19

I am trying to create a function that works around concurrency issues that are documented, and that we are seeing, when doing simultaneous "create table" statements and bulk insert statements (or selects) off of other tables. To do this, I am implementing a function that uses a boost::shared_mutex that we call around transactions. We are using a custom version of the boost library, so it is installed in a non-standard place: /usr/local/boost-1.54.0 for development, and /opt/xxx/ for production. >From the documentation, it appears that I can use #pragma LDFLAGS and CFLAGS to point to the non-standard directories. It appears to honor the CFLAGS option, but not the LDFLAGS option. If I use the standard /usr/lib64 libraries, calling my function works (but with the wrong library). Adding the #pragma in, I get the following link error: Failed to open shared library: libboost_thread.so.1.54.0: cannot open shared object file: No such file or directory. If I modify LD_LIBRARY_PATH before starting MonetDB, the problem goes away. Ideally, we don't want to have to add all of our relevant directories into the environment, but would expect the #pragma to work. Here is the function definition: CREATE FUNCTION rwlocker(input INT) RETURNS INT LANGUAGE CPP { #pragma LDFLAGS -L/usr/local/boost-1.54.0/lib -lboost_thread #pragma CFLAGS -I/opt/tc3/include -I/usr/local/boost-1.54.0/include #include <boost/thread/shared_mutex.hpp> // use lock() for schema changes, lock_shared() for inserts and queries static boost::shared_mutex readerWriterMutex; result->initialize(result, 1); int val = input.data[0]; switch(val) { case 1: // shared lock readerWriterMutex.lock_shared(); break; case 2: // shared unlock readerWriterMutex.unlock_shared(); break; case 3: // unique lock readerWriterMutex.lock(); break; case 4: // unique unlock readerWriterMutex.unlock(); break; default: // Error break; } result->data[0] = val; }; Invocation is easiest with: select rwlocker(12); which will just output a 12, bypassing the reader/writer locking. Any thoughts on why #pragma LDFLAGS doesn't appear to work right? We are using 11.33.3 on CentOS 7.3.1611 Thanks, Dave

2 2

Announcement: New Apr2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Sep '19

02 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019-SP1 bugfix release (11.33.11) Build Environment * Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched. MonetDB Common * We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available. SQL Frontend * Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries. * Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR. Bug Fixes * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6697: Duplicate expressions not eliminated with long CASE statement * 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed * 6703: SQL optimizer enters loop and goes into stack overflow * 6706: prepare doesn't recognize merge statement * 6712: Where clause with cast ignores sub-select * 6713: COPY INTO FROM 'file.xz' does not work * 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit) * 6715: Assertion failure in rel_bin.c for MERGE INTO command * 6716: COPY INTO does not load UTF8 encoded text * 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash) * 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation] * 6721: Possibly incorrect call to pcre_exec * 6725: Prepare statement on non-existing function crashes * 6728: SELECT DISTINCT not removing duplicates * 6729: Creating a table with duplicate column name should not be possible * 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always" * 6736: Unexpected query result with merge tables and view after upgrade * 6738: issue with window functions and hugeint type coercion * 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi * 6744: CREATE SEQUENCE seq; fails * 6748: MonetDB "forgets" columns after ALTER TABLE statements * 6749: mserver5 restart aborts/segfaults after dropping column with constraint * 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL

1 0

Crash - !mvc_init: unable to create system tables
by Nobody Noone 30 Aug '19

30 Aug '19

Hi, Would anyone have any advice on recovering. I was running a bunch of COPY SELECT * FROM x INTO '/y.z.csv' USING DELIMITERS ',' , '\n' , '"'; The log said there was already an existing file in the dir. The db was not responding so I stopped it. Then tried to start it and I get; !mvc_init: unable to create system tables #main thread:!ERROR:SQLException:SQLinit:42000!Catalogue initialization failed Fatal error during initialization: SQLException:SQLinit:42000!Catalogue initialization failed # MonetDB 5 server v11.33.3 (Apr2019) database 'y' (53342) has exited with exit status 1 ERR control[33899]: (local): failed to fork mserver: database 'y' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details Any advice on how to recover? Thanks:)

1 0

Performance issue related to dropping (empty) tables
by Gotwisner, Dave 27 Aug '19

27 Aug '19

I am seeing a performance issue related to schema upgrades. This is from a C++ program that uses the MAPI library. We try and keep 30 days worth of data, imported at a fairly high data rate with lots of columns. We use multiple threads to ingest data (bulk insert), and the number of threads is stored in a configuration table. For 20 days worth of data, this works out to around 9000 tables (plus merge tables). We have roughly 6 sets of tables per hour, each of which has 4 tables and a merge table connecting them. Because this is a schema change, we wrap the entire upgrade in a transaction. For the purpose of the test, we had no data in the tables, but only had empty tables. If we change our setting from 4 parallel tables to 2, we end up dropping 9000 tables. About half of the "drop table" commands take 0 seconds. The rest take anywhere between 4 and 12 seconds each. Here is a plot of the time per delete, y access is seconds, x is the statement. The X access is NOT time based: [cid:image001.png@01D5568E.A5389030] Any idea on how we can reduce the time? The total time to delete 9000 tables was 480 minutes. Thanks, Dave

2 4

memory control on large results
by Daniel Zvinca 15 Aug '19

15 Aug '19

Hello, I plan to use the embedded library (custom build) for a project that will likely address very large tables. While the control over statements is handled to the end user, I need an advice how to control memory usage on fetching huge amount of data. Specifically, I already skipped the default "monetdb_result_fetch" and implemented a compressed storage (which optional can be stored on disk), but I wasn't able to see a way of releasing the associated memory for already processed columns (persistent or transient bat). Unless the bat contents is shared between different res_col's, I would like to be able to release that memory to the system before proceeding to the next column. Any suggestion? Thank you, Dan

1 0

Embedded Python Initialization Issue
by Sharma, Sreejith 14 Aug '19

14 Aug '19

All, I am using an Python based UDF and have enabled embedpy=true for DB with Python version 2.7.5. This works fine with CentOS 7, but the same doesn't wok on Red Hat Ent Linux version. Is there any dependency on this? Is it due to OS version? Am getting below error though Python path is proper in Red Hat OS version server. Embedded Python is enabled but an error was thrown during initialization. Can anyone provide some help on this? Both uses same version of MonetDB server version. MonetDB server on which function works fine [user@lnx1535 ~]$ cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" MonetDB server on Azure where function is not working [user@alcazimdbd01 ~]$ cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.5 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.5" PRETTY_NAME="Red Hat Enterprise Linux" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.5 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.5" Regards, Sreejith

2 1

Type of integers used to calculate a SUM
by Jan, Di-An 13 Aug '19

13 Aug '19

With hugeint enabled, it seems that the SUM of INT (32-bit) is done using HUGEINT (128-bit) while the SUM of BIGINT (64-bit) is done using BIGINT (64-bit). That seems weird. Is SUM supposed to use the largest type or use the same type as the input or something else? Database: MonetDB v11.33.3 (Apr2019) sql>CREATE TABLE test (bi BIGINT, i INT); sql>INSERT INTO test VALUES (0x7000000000000000, 0x70000000); sql>INSERT INTO test VALUES (0x7000000000000000, 0x70000000); sql>INSERT INTO test VALUES (1, 3); sql>INSERT INTO test VALUES (2, 4); sql>SELECT SUM(i) FROM test; -- larger than an INT -- +------------+ | L3 | +============+ | 3758096391 | +------------+ 1 tuple sql>SELECT SUM(bi) FROM test; -- larger than a BIGINT -- overflow in calculation. sql>EXPLAIN SELECT SUM(i) FROM test; +------------------------------------------------------------------------------------------------------+ | function user.s10_22():void; | | X_1:void := querylog.define("explain select sum(i) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:hge := aggr.sum(X_17:bat[:int]); | | exit X_74:bit; | | sql.resultSet("sys.L3":str, "L3":str, "hugeint":str, 128:int, 0:int, 7:int, X_18:hge); | | end user.s10_22; | +------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(bi) FROM test; +-------------------------------------------------------------------------------------------------------+ | function user.s12_22():void; | | X_1:void := querylog.define("explain select sum(bi) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:lng] := sql.bind(X_4:int, "sys":str, "test":str, "bi":str, 0:int); | | X_17:bat[:lng] := algebra.projection(C_5:bat[:oid], X_8:bat[:lng]); | | X_18:lng := aggr.sum(X_17:bat[:lng]); | | exit X_74:bit; | | sql.resultSet("sys.L3":str, "L3":str, "bigint":str, 64:int, 0:int, 7:int, X_18:lng); | | end user.s12_22; | +-------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(i AS HUGEINT)) FROM test; +-----------------------------------------------------------------------------------------------------------------------+ | function user.s20_22():void; | | X_1:void := querylog.define("explain select sum(cast(i as hugeint)) from test;":str, "default_pipe":str, 13:int); | | barrier X_75:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:bat[:hge] := batcalc.hge(X_17:bat[:int]); | | X_19:hge := aggr.sum(X_18:bat[:hge]); | | exit X_75:bit; | | sql.resultSet("sys.L4":str, "L4":str, "hugeint":str, 128:int, 0:int, 7:int, X_19:hge); | | end user.s20_22; | +-----------------------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(bi AS BIGINT)) FROM test; +-----------------------------------------------------------------------------------------------------------------------+ | function user.s8_22():void; | | X_1:void := querylog.define("explain select sum(cast(bi as bigint)) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:lng] := sql.bind(X_4:int, "sys":str, "test":str, "bi":str, 0:int); | | X_17:bat[:lng] := algebra.projection(C_5:bat[:oid], X_8:bat[:lng]); | | X_18:lng := aggr.sum(X_17:bat[:lng]); | | exit X_74:bit; | | sql.resultSet("sys.L4":str, "L4":str, "bigint":str, 64:int, 0:int, 7:int, X_18:lng); | +-----------------------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(i AS INT)) FROM test; +-----------------------------------------------------------------------------------------------------------+ | function user.s10_22():void; | | X_1:void := querylog.define("select sum(cast(i as int)) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:hge := aggr.sum(X_17:bat[:int]); | | exit X_74:bit; | | sql.resultSet("sys.L4":str, "L4":str, "hugeint":str, 128:int, 0:int, 7:int, X_18:hge); | | end user.s10_22; | +-----------------------------------------------------------------------------------------------------------+ sql>SELECT SUM(CAST(i AS INT)) FROM test; +------------+ | L4 | +============+ | 3758096391 | +------------+ 1 tuple Also, the cast in SUM(CAST(bi AS BIGINT)) seems to be optimized out while SUM(CAST(i AS HUGEINT)) has one more BAT operation than SUM(i) even though they seem to do the same thing, and the cast in SUM(CAST(i AS INT)) seems to have no effect. Di-an Jan

1 0

Books about Monetdb?
by Nobody Noone 09 Aug '19

09 Aug '19

Hi, Are there any books published about Monetdb? This listserv, monetdb.org and stackoverflow.com seem to be all I can find. Thanks!

2 1

understand how monetdb choose the best query plan
by DEMBELE Simon Pierre 08 Aug '19

08 Aug '19

Hello, J'm new in the use of monetdb, i want to understand how monetdb choose the best query plan. Is the model cost is used for that or what. Please if someone can clarify me. Thank yours time. -- Simon

2 2

Questions on upgrading tables based upon the system tables.
by Gotwisner, Dave 08 Aug '19

08 Aug '19

I am writing code to migrate from one version of our schema to another (such as if we add tables or add columns). The from version is whatever is physically being used, and the new version is compiled into data structures in our code. I can successfully create or remove tables, and add and drop columns. We have multiple tables that we then bind to a merge table, so the schemas must be exact. Our tables have a primary key. For the table and column based changes, querying sys.schemas, sys.tables, and sys.columns gives us everything we need. I can also figure out how to add the primary key on the new table (via alter table commands). However, I can't figure out how to find the set of columns in the primary key, to be able to build up the primary key. Here is the table definition for our tables that go into the merge table (%04d goes from 0000 to the number of threads we send bulk data with): CREATE TABLE if not exists data_availability__%04d ( interval_start BIGINT, source INT, PRIMARY KEY(interval_start, source) ); alter table data_availability add table data_availability__%04d; Looking at system tables gives the following information (before adding this table): sql>select * from sys.tables where name = 'data_availability'; +------+-------------------+------+------+------+-------+------+-------+-------+ | id | name | sche | quer | type | syste | comm | acces | tempo | : : : ma_i : y : : m : it_a : s : rary : : : : d : : : : ctio : : : : : : : : : : n : : : +======+===================+======+======+======+=======+======+=======+=======+ | 8889 | data_availability | 8800 | null | 3 | false | 0 | 0 | 0 | +------+-------------------+------+------+------+-------+------+-------+-------+ 1 tuple sql>select * from sys.keys where table_id = 8889; +------+----------+------+----------------------------------------------+------+--------+ | id | table_id | type | name | rkey | action | +======+==========+======+==============================================+======+========+ | 8887 | 8889 | 0 | data_availability_interval_start_source_pkey | -1 | -1 | +------+----------+------+----------------------------------------------+------+--------+ 1 tuple sql>select * from sys.key_types; +-------------+---------------+ | key_type_id | key_type_name | +=============+===============+ | 0 | Primary Key | | 1 | Unique Key | | 2 | Foreign Key | +-------------+---------------+ 3 tuples sql>select * from sys.idxs where table_id = 8889; +------+----------+------+----------------------------------------------+ | id | table_id | type | name | +======+==========+======+==============================================+ | 8888 | 8889 | 0 | data_availability_interval_start_source_pkey | +------+----------+------+----------------------------------------------+ 1 tuple sql>select * from sys.index_types; +---------------+-----------------------+ | index_type_id | index_type_name | +===============+=======================+ | 0 | Hash | | 1 | Join | | 2 | Order preserving hash | | 3 | No-index | | 4 | Imprint | | 5 | Ordered | +---------------+-----------------------+ 6 tuples sql>select * from sys.columns where table_id = 8889; +------+----------------+--------+-------------+------------+----------+---------+-------+--------+---------+ | id | name | type | type_digits | type_scale | table_id | default | null | number | storage | +======+================+========+=============+============+==========+=========+=======+========+=========+ | 8885 | interval_start | bigint | 64 | 0 | 8889 | null | false | 0 | null | | 8886 | source | int | 32 | 0 | 8889 | null | false | 1 | null | +------+----------------+--------+-------------+------------+----------+---------+-------+--------+---------+ 2 tuples The above data is for the merge table that we will bind the (new) table to. I can get the name and id of the primary key (which is also an index). I could not find a table that gives a column -> key binding. Is there such a table? If not, how do I get this information from MonetDB? Thanks, Dave

2 1

Question on querying two specific columns from sys.columns
by Gotwisner, Dave 07 Aug '19

07 Aug '19

I am trying to query the default and null columns from sys.columns. Both of these are sql key words, so queries either don't work or return the wrong value. Doing a "select default from ss.columns;" returns "syntax error, unexpected DEFAULT in: "select default"" error. Doing a "select null from sys.columns;" returns null for all records, even though it is a Boolean column with true/false values. I am currently working around this by doing "select *" and pulling out the relevant fields, but, I want to protect against a future release possibly changing the column ordering or changing the number of columns in this table (without also requiring an update to our code). Is there a way to escape the column names in a way that gives me what I want? Thanks, Dave

2 1

Re: Alter question about MonetDB
by Ying Zhang 06 Aug '19

06 Aug '19

Hai, This question should have been sent to the users mailing list, so I added that e-mail address. I can confirm this problem with an Apr2019 refreshed yesterday. Please see the 1st attachment. Also, after having restarted both mclients, I can still only query “tbl5” using the mclient on the left side. Please see the 2nd attachment. Can you please fire a bug report for this? Thanks! Jennie > On 6 Aug 2019, at 08:55, 角 <3089454366(a)qq.com> wrote: > > Hi,All: > I am a MonetDB user from China. I have some troubles when I was using MonetDB. I use mclient and enter this sql : "alter table t1 rename to t2；", while this mclient is alive,I use anthor mclient to connect the same database as the former and I enter this sql : "select * from t2" then I get an error : no such table. So how can I use alter command to make all mclients take effect? > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list

1 0

Question on missing BAT file
by Sharma, Sreejith 06 Aug '19

06 Aug '19

All, Is there a way to figure out which BAT files constitutes a specific table in MonetDB? I know the location in sys.storage represents it, but I see one of the tail file is showing missing which is not available neither in sys.storage not in DB folder. We get this error during migration from older version (MonetDB 11.27.3) to MonetDB 11.33.3 version. 2019-07-31 10:43:54 MSG TSV_PROD_DB_A[161617]: !ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV/TSV_PROD_DB_A/bat/10/45/104502.tail Regards, Sreejith

2 2

Fwd: Re: Managing Access to Monetdb Schemas
by ferreira 02 Aug '19

02 Aug '19

-------- Forwarded Message -------- Subject: Re: Managing Access to Monetdb Schemas Date: Fri, 2 Aug 2019 11:06:21 +0200 From: ferreira <pedro.ferreira(a)monetdbsolutions.com> To: Nobody Noone <nobody.noone.nobody.me(a)gmail.com> Hello! I found it strange why we don't support GRANT/REVOKE privileges statements on schemas. I looked into the SQL standard, and there is no mention of it. You can use roles to limit schema access for each user. Here you have an example: CREATE ROLE my_role; CREATE SCHEMA my_schema AUTHORIZATION my_role; CREATE USER my_user with password 'p1' name 'User with role' schema "my_schema"; GRANT my_role to my_user; You can add more fine grained access to each table with table privileges as you mentioned. As for specific rights such as DDL, we don't support it as of now. You can create a feature request on Bugzilla, but as it is not paid, it will have low priority. Best regards, Pedro On 7/31/19 4:40 PM, Nobody Noone wrote: > New to Monetdb, From a DBA perspective I’m having trouble > understanding how to manage access rights to schemas and the tables > therein. My department has many senior analysts who should have admin > rights to the tables within the schema (DML - select, insert, update, > delete, trunc, etc; DDL create table, drop table, create seq, etc). > There are also some users that need select only in a given schema. How > do I manage this without having to grant access per table? Is that > where roles come into play? Say super users get privilege_code_id= 79 > (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be > assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am > not seeing those in the roles? I do see the ability when the schema is > created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I > don’t want to do is have to run a query that searches all tables in a > schema, grant admin DDL to them. > Thanks! > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list

1 0

Cast a string column with empty strings to int
by Ying Zhang 01 Aug '19

01 Aug '19

Hai, I have e.g. the following string column containing either numbers or empty strings: CREATE TABLE tst (s string); INSERT INTO tst VALUES ('1234'), (''), (‘4567'); I’d like to cast ’s' to an INT column. The following query works: sql>SELECT CAST(s AS INT) FROM (SELECT CASE WHEN s='' THEN NULL ELSE s END AS s FROM tst) AS tmp; +------+ | L4 | +======+ | 1234 | | null | | 4567 | +------+ 3 tuples But the following query doesn’t: sql>SELECT CASE WHEN s='' THEN NULL ELSE CAST(s AS INT) END FROM tst; conversion of string '' to type int failed. sql>SELECT CASE WHEN s='' THEN CAST(NULL as INT) ELSE CAST(s AS INT) END FROM tst; conversion of string '' to type int failed. This is probably because batcalc.int is applied on the whole “s” column as shown by the query plan: sql>EXPLAIN SELECT CASE WHEN s='' THEN NULL ELSE CAST(s AS INT) END FROM tst; +--------------------------------------------------------------------------------------------------------------------+ | mal | +====================================================================================================================+ … | C_7:bat[:oid] := sql.tid(X_6:int, "sys":str, "tst":str); | | X_10:bat[:str] := sql.bind(X_6:int, "sys":str, "tst":str, "s":str, 0:int); | | X_19:bat[:str] := algebra.projection(C_7:bat[:oid], X_10:bat[:str]); | | X_21:bat[:bit] := batcalc.==(X_19:bat[:str], "":str); | | X_24:bat[:bit] := batcalc.isnil(X_21:bat[:bit]); | | X_30:bat[:bit] := batcalc.ifthenelse(X_24:bat[:bit], false:bit, X_21:bat[:bit]); | | X_35:bat[:int] := batcalc.int(X_19:bat[:str]); | | X_36:bat[:int] := batcalc.ifthenelse(X_30:bat[:bit], nil:int, X_35:bat[:int]); | … Is my query wrong or the plan? Thanks, Jennie

2 2

Question with wildcarded expressions for LIKE clause
by Gotwisner, Dave 01 Aug '19

01 Aug '19

I am trying to do some constraints off the system tables to select a unique pattern of tables. We have tables like the following (I can't give the actual table names): * Config (a single record exists here, no merge tables use it) * Data_0000 (a small number of records exist in each of these) * Data_0001 * Data_0002 * Data_0003 * Data (this is a merge table over the last 4) * Test_table (a small number of records exist here, no merge tables use it) * Main__yyyy_mm_dd__hh__0000 (millions of records, yyyy, mm, dd, hh are all numbers, we have one set of tables per hour, and keep significantly over 1 weeks' worth of data) * Main__yyyy_mm_dd__hh__0001 * Main__yyyy_mm_dd__hh__0002 * Main__yyyy_mm_dd__hh__0003 * Main__yyyy_mm_dd__hh (this is a merge table over the last 4) If I do a select * from sys.tables where schema_id in (select id from schemas where name = 'frodo') and type in (select table_type_id from table_types where table_type_name = 'TABLE') I get the right thing - all non merge tables. I then want to further break it down into the tables that end in __[0-9][0-9][0-9][0-9] and those that don't (two separate queries for two separate purposes). Consider frodo the name of our schema. Adding and name like 'data\_\_____' appears to work. This should give me the 4 data_* tables. Adding and name like '%\_\_____' gives me all non-merge tables, including config and test_table. I'm assuming % is the wild card for 0 or more, and _ is the wild card for 1. This matches some documentation I have found. The \ are required because I need to explicitly look for 2 of the underscore character. Am I doing this correctly? Are these the right wildcarding characters? MS SQLServer gives a # as a match for a single digit, is there an equivalent? Or, is this just a bug in MonetDB? Thanks, Dave

3 5

Managing Access to Monetdb Schemas
by Nobody Noone 31 Jul '19

31 Jul '19

New to Monetdb, From a DBA perspective I’m having trouble understanding how to manage access rights to schemas and the tables therein. My department has many senior analysts who should have admin rights to the tables within the schema (DML - select, insert, update, delete, trunc, etc; DDL create table, drop table, create seq, etc). There are also some users that need select only in a given schema. How do I manage this without having to grant access per table? Is that where roles come into play? Say super users get privilege_code_id= 79 (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am not seeing those in the roles? I do see the ability when the schema is created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I don’t want to do is have to run a query that searches all tables in a schema, grant admin DDL to them. Thanks!

1 0

How to manage access rights
by Nobody Noone 30 Jul '19

30 Jul '19

New to Monetdb, From a DBA perspective I’m having trouble understanding how to manage access rights to schemas and the tables therein. My department has many senior analysts who should have admin rights to the tables within the schema (DML - select, insert, update, delete, trunc, etc; DDL create table, drop table, create seq, etc). There are also some users that need select only in a given schema. How do I manage this without having to grant access per table? Is that where roles come into play? Say super users get privilege_code_id= 79 (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am not seeing those in the roles? I do see the ability when the schema is created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I don’t want to do is have to run a query that searches all tables in a schema, grant admin DDL to them. Thanks!

1 0

Issue while migrating to MonetDB v11.33.3 (Apr 2019)
by Sharma, Sreejith 25 Jul '19

25 Jul '19

All, We are migrating our MonetDB database from MonetDB v11.27.13 (Jul2017-SP4) to MonetDB v11.33.3 (Apr 2019). We tried to bring up the DB instance with MonetDB v11.33.3 (Apr 2019), but getting below error though the same when we tried with previous version MonetDB v11.31.13 (Aug2018-SP2) it worked. Could you help us to identify the issue? 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: #main thread:!ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV_TEST/TSV_PROD_DB_T/bat/10/45/104502.tail 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: #main thread:!OS: No such file or directory 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: !FATAL: BBPinit: BBPcheckbats failed 2019-07-24 04:00:21 MSG TSV_PROD_DB_T[88767]: !ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV_TEST/TSV_PROD_DB_T/bat/10/45/104502.tail 2019-07-24 04:00:21 MSG TSV_PROD_DB_T[88767]: !OS: No such file or directory 2019-07-24 04:00:23 MSG merovingian[88722]: database 'TSV_PROD_DB_T' (88767) has exited with exit status 1 2019-07-24 04:00:24 ERR control[88722]: (local): failed to fork mserver: database 'TSV_PROD_DB_T' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints 2019-07-24 04:02:18 MSG control[88722]: (local): served status list Regards, Sreejith

2 1

(Без темы)
by Илья Хайбуллин 25 Jul '19

25 Jul '19

How can I write a workable join by null-safe eqauls? I have two tables: create table monet1_test ("Id" bigserial, key int, value varchar(2000)); create table monet2_test ("Id" bigserial, key int, value varchar(2000)); insert into monet1_test select value "Id", value "key", value "value" from generate_series(1,60000); insert into monet2_test select value "Id", value "key", value "value" from generate_series(1,200000); I tried select count(1) from monet2_test t1 inner join monet1_test t2 on ((t1.key = t2.key) or ((t1.key is null) and (t2.key is null))); and got errors GDK reported error: MT_mmap: CreateFileMapping(0000000000001200, &sa, 4, 22, 1508704256, NULL) failed OS: The printer is out of paper. HEAPalloc: Insufficient space for HEAP of 95997921280 bytes. MT_mmap: CreateFileMapping(00000000000011B4, &sa, 4, 22, 1508704256, NULL) failed OS: The printer is out of paper. HEAPalloc: Insufficient space for HEAP of 95997921280 bytes. I create bug-report: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6737

3 7

Indexing support on Json DataType
by Chatchawan Triperm 22 Jul '19

22 Jul '19

Hi All, Does anyone know whether there is built-in indexing on Json datatype or not. I would like to keep my dynamic columns in Json but worry about indexing. Regards, Chatchawan

3 4

Unexpected query result with merge tables and view after upgrade
by Enrico Maria Carmona 22 Jul '19

22 Jul '19

2 2

Questions on embedding R in MonetDB and client pacakge in R
by Andor 18 Jul '19

18 Jul '19

Hello everyone there, I am a beginner user of MonetDB , mainly working with database from R, new to this email list. Is there a seach function somewhere in this list? I can only see the archieve webpage. I have some general user questions, and would greatly appreciate your comments and suggestions. the embedding R in MonetDB.The very latest materials I could find on this topic on MonetDB websites, dated backed in 10/31/2014, I am wondering where I can find more embedding R in MonetDB related information? tutorial, documents etc the client package working with MonetDB from R I only see monetDB.R but not MonetDBLite in the CRAN packages index, yet in MonetDB site , both packages are introduced. any comments on the MonetDBLite, will it be there in the future? Thanks! Andor

1 0

yum upgrade doesn't allow existing db use new window functions
by Andy Allaway 09 Jul '19

09 Jul '19

Hi, I ran: yum upgrade MonetDB-SQL-server5 MonetDB-client When I create a new db I can call: select first_value(col1) over (partition by col2) from analytics; as per the blog monetdb.org/blog/extended_sql_window_functions However a db that existed before the update returns: Function first_value not found. I can see first_value in sys.functions. Any suggestions? Thanks!

2 2

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 04 Jul '19

04 Jul '19

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

4 3

Question on communications path between mserver5 and the MonetDB API (libmapi)
by Gotwisner, Dave 04 Jul '19

04 Jul '19

We are transferring a lot of data as the result of some of our queries (20-30 columns of integers and floats) for about 50,000 records, aggregating these from multiple MonetDB instances (on different machines). For this pattern of query, the largest amount of time spent on the query (from the network onwards, ignoring Mserver5 lookup time) is on the data conversion. When running "perf top" on our application, we notice we are spending a significant amount of time (the second most samples per perf top item at 14%) converting the data returned by libmapi from "const char *" to int64_t, int32_t, double, etc. We know the type of the data, and we have very few string columns in the database. Is there an option to MonetDB that allows it to pass binary data rather than textual data back and forth on the wire? Is there an API that will give the data back in the native type rather than in the on-the-wire string type? Both client and server are always running on 64-bit X86 boxes running Linux (meaning, little endian is guaranteed)? We use serialized protobufs whenever we send data on the wire when communicating between our various programs, and the data stream is significantly smaller than sending integers back and forth in ascii. If not, would switching to ODBC / JDBC give us better performance in this area? Thanks, Dave

2 1

Q: Logical plan
by Hanfeng Chen 04 Jul '19

04 Jul '19

Hello, A quick question. Is the logical plan generated by the keyword "plan" is the plan before it is sent to the SQL optimizer *OR* the output plan of the optimizer? >From the online documentation ( https://www.monetdb.org/Documentation/Manuals/SQLreference/PlanSQL) this sentence "*The relational plan used in the SQL optimizer* can be obtained by prepending the query with the keyword PLAN." is not clear to me. Thanks, Hanfeng Chen

2 1

Using the Monet API to connect to MonetDB if MonetDB comes up AFTER our application
by Gotwisner, Dave 24 Jun '19

24 Jun '19

I am currently trying to update my code that uses the Mapi API to retry connections if we start an application when MonetDB is down. Our programs do not run interactively, so detecting the failure, terminating the program, and restarting the application is problematic. There are a couple of issues here that I can use help with. What we are doing is calling mapi_connect(). Mapi_connect returns what appears to be a pointer (it's non-zero). So we call mapi_error with this return value. If we are not connected, we get an MERROR, not an MTIMEOUT, which is what I would expect. With an MERROR failure, we call mapi_error_str() to see what kind of error. We then must do a string compare (actually, a strstr() call looking for a "Connection refused" pattern inside the string. Is there a way to get an enum or #define value instead, with a different function call? Sort of like looking at errno rather than calling strerror(errno) and doing a string compare on the result. This would be way more efficient if we could do this. Yes, errors should be rare, but determining the type of error (or protecting against a string text change on an RPM update) shouldn't be expensive. We then call mapi_explain() to stderr. Is there a routine that returns the value of the output of explain() to a std::string or a "const char *"? We have a logger that does stuff with the strings before logging (timestamping, __line__ __file__, putting error codes, etc. Having the multiline output of the explain() information would be very useful for us. The above process is executed multiple times until we connect. Since the failed connects return what appears to be a pointer, do we need to do anything to free the pointers up? It doesn't appear that we have a memory leak (I ran valgrind). Note that if the connect fails, calling mapi_destroy() crashes. From the documentation, "mapi_destroy()" does "Free handle resources", so it seems reasonable to call it after a failed (and non-null) mapi_connect(). It should probably do nothing, it should not crash. Also, it appears that if I connect multiple times without disconnecting (I had a bug in my code), I can't connect more than 65 times (simultaneous connections). Is this an API issue or a server issue? The reason I ask, is that if it a client issue, we will be having a single client that potentially does queries against hundreds of servers, and aggregates the result. If it is a server issue, is there a parameter that we can tweak to increase this? We do multiple queries in parallel, each on their own connections. Thanks, Dave

2 2

Guidelines for MonetDB in production environments
by Roberto Cornacchia 20 Jun '19

20 Jun '19

Hi all, I'm struggling with optimizing resource sharing of MonetDB in production environments (see also: https://www.monetdb.org/pipermail/users-list/2018-June/010276.html) We run MonetDB instances for several projects / customers on each of our servers. Each MonetDB instance is a docker container (used mainly because of ease of deployment and environment isolation). It is not unusual to have 5-10 MonetDB containers on the same server. In principle, Docker does not even have much to do with this, but it puts everything in a realistic context. ** Memory mserver5 checks the system memory and calibrates on that. When 10 instances are running, they all assume they have the whole memory for themselves. Docker allows to set limits on the container memory. It does that by using cgroups (so Docker just makes things easier, but it's really about cgroups). However, memory limits set by cgroups are not namespaced ( https://ops.tips/blog/why-top-inside-container-wrong-memory/#memory-limits-… ). This means that each container will still see the whole memory and will simply get killed when the container limit has been reached (definitely not a solution). So far, the only mechanism I know to obtain the correect behavior is to run actual VMs for MonetDB. But this is very cumbersome and I want to avoid that as much as possible. Should we let 10 instances believe they each have the whole memory, and let them fight for it? (well, that's what's happening now, and I know for sure it's bad). Perhaps the solution can be as easy as allowing an explicit max memory setting, together with some documentation on the consequences of using low / high values. ** CPU Again, Docker allows to set quotas per container. I think cgroups CPU limits are namespaced, so perhaps this would just work well, I haven't really tried yet. ** I/O Same issue. It would be ideal to be able to set priorities, so that mserver5 instances that do background work get a lower I/O priority than instances serving online queries. Also, recommendations on swap settings would be interesting. How much swap? How to tune swappiness kernel settings? I am very aware that there is no simple answer to most of these questions. Many variables are in the picture. Still, some general thoughts from the developers would be appreciated. I think I have read pretty much everything has ever been written about MonetDB, but when it comes to resource utilization I have always bumped into the very unrealistic assumption that each MonetDB instance has a whole server for itself. As I mentioned above, things could get already much better with simple improvements, like allowing to set the maximum memory usable by each instance. But more in general, I feel there is much need for some guidelines for production environments. Or at least, to start the discussion. Best regards, Roberto

3 5

Error Upgrading to 11.33.3 (Apr2019)
by Ariel Abadi 19 Jun '19

19 Jun '19

Hi! I have a database that It started on version Dec2016-SP5. I have been upgrading from version to version without any problem. Now, Im trying to do the last upgrade (this means from Aug2018-SP2 to Apr2019. But Im facing an error. This is what I see on the log: 2019-06-06 15:17:55 ERR cli_gm[2188]: #main thread:!ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 MSG cli_gm[2188]: !ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 ERR cli_gm[2188]: !FATAL: BBPinit: BBPcheckbats failed 2019-06-06 15:17:58 MSG merovingian[2180]: database 'cli_gm' (2188) has exited with exit status 1 2019-06-06 15:17:58 ERR control[2180]: (local): failed to fork mserver: database 'cli_gm' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints Is there anything I can do to solve this problem? Thks in advance Ariel PD.If I go back to version Aug2018-SP2 the database keeps working with no problem.

2 2

Questions about using COPY INTO when using GRANTS on tables
by Gotwisner, Dave 14 Jun '19

14 Jun '19

Our application was working quite well when using COPY INTO when inserting data into our tables as the default MonetDB user "monetdb" into the system schema (sys). We have subsequently added users and granted table access to the various users. At this point, INSERT works successfully, but COPY INTO does not. The error message for the COPY INTO statement indicated that we needed to use "ON CLIENT" in our load. That makes no difference. Here are the relevant commands to the system (note the user names, passwords, schema names, and tables names were modified for this example): Program that sets up our basic database does this: START TRANSACTION; CREATE USER admin WITH UNENCRYPTED PASSWORD '***' NAME 'Admin User' SCHEMA sys; CREATE SCHEMA test AUTHORIZATION admin; ALTER USER admin SET SCHEMA test; CREATE USER update WITH UNENCRYPTED PASSWORD '***' NAME 'Update User' SCHEMA test; COMMIT; Then, the program that creates the tables does the following: START TRANSACTION; CREATE MERGE TABLE x (...); GRANT ALL ON x TO "update"; CREATE TABLE x0 (...); GRANT ALL ON x0 TO "update"; CREATE TABLE x1 (...); GRANT ALL ON x1 TO "update"; CREATE TABLE x2 (...); GRANT ALL ON x2 TO "update"; CREATE TABLE x3 (...); GRANT ALL ON x3 TO "update"; COMMIT; And we insert (or try to) (4 threads ingest into their own table, the merge table is used to combine the result set on queries): START TRANSACTION; COPY 100000 OFFSET 2 RECORDS INTO x0 FROM '/dev/shm/MonetDB/foo.csv' (columnlist) ON CLIENT DELIMETERS ',','\n'; This returns an error: !/dev/shm/MonetDB/foo.csv': cannot retrieve files. Because of company policy, I can't provide the schema, but the columns in the csv file and the schema differ, which is why I am using the (...) and (columnlist) in the example. Again, when I omit the CREATE USER, CREATE SCHEMA, and GRANT clauses, everything works correctly. Any ideas on what is needed to make it work? Right now, the ingest application and MonetDB are on the same server. In the future, that might not be true. We are running 11.33.3 on CentOS 7. Thanks, Dave

2 1

FW: Cancel query
by Zingman, Jonathan 13 Jun '19

13 Jun '19

Yes, sys.queue(), sorry. A couple of clarifications. The cookbook shows mclient, but we're using the API and we assume you can do the same thing from the API as well. Also, we use autogenerated queries that can be hundreds if not thousands of characters long. Matching in a case independent way would be very resource intensive. It's fairly clear from the documentation that MonetDB doesn't stop immediately in the middle of whatever it is doing, so that's understood. But cancelling queries has to be part of a robust database system. My model of how this might work is that the handle that’s returned when we issue a query could be used to associate with some data returned by sys.queue(), preferably tag or start time since those should be unique and fairly short. Then we stash then in some structure, and if a cancel request comes in, we do a much simpler search of the sys.queue() table that's guaranteed to work. JONATHAN ZINGMAN | Manager – TrueCall Data | DIRECT: 510.809.0423 | CELL: 510.823.5638 | www.netscout.com -----Original Message----- From: users-list <users-list-bounces+jonathan.zingman=netscout.com(a)monetdb.org> On Behalf Of Martin Kersten Sent: Tuesday, June 11, 2019 1:56 PM To: users-list(a)monetdb.org Subject: Re: Cancel query This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe. On 11/06/2019 22:38, Ying (Jennie) Zhang wrote: > > >> On 11 Jun 2019, at 00:58, Zingman, Jonathan <Jonathan.Zingman(a)netscout.com> wrote: >> >> In the cookbook, you give a good way to find out the status of queries by using the sys.query() call, > > Hai, Jonathan, > > MonetDB doesn’t have a function called sys.query(). Did you mean sys.queue() instead? > >> which returns the list of queries tagged in various ways. Our question is whether there is any way to access, e.g., the tag value. In our application, we need to occasionally cancel queries from our UI. The UI has no direct knowledge of the query. While we might be able to find the query we want to cancel by comparing the query text, our automatically generated queries can be very long, and if there’s some internal reformatting by Monetdb, we might not get an exact match. So is there anything in the handle or elsewhere that we can use to identify the query we want to cancel? > > Correct me if I’m wrong, but I’m not aware of a feature with which you can query the qtag of a query. Some code study will be needed to see if it’s feasible to add such feature, and how. > > You can probably get away with string matching, since MonetDB doesn’t do much/any reformatting next to to-lowercase. Not sure how much text the “query” field of sys.queue() holds. > > In addition, please be aware that > i) sys.queue() is originally develop for debugging purpose… > ii) the minimal cancellation entity is a function, i.e., when MonetDB is busy executing a very long running function, a cancellation will only take effect *after* that function has finished => then MonetDB won’t continue executing the next function. Small correction here, with function we mean a MAL function call. So, if the system is executing a very time consuming join, it will *not* be interrupted, but rather at the end of it the query in which is rans is terminated. > > Regards, > Jennie > >> >> JONATHAN ZINGMAN >> Manager – TrueCall Data >> >> <image001.png> >> 2855 Telegraph Ave, Suite 200 >> Berkeley, CA, 94705-1169 >> >> DIRECT: 510.809.0423 >> CELL: 510.823.5638 >> http://www.netscout.com >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma… > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma… > _______________________________________________ users-list mailing list users-list(a)monetdb.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma…

2 1

Cancel query
by Zingman, Jonathan 11 Jun '19

11 Jun '19

In the cookbook, you give a good way to find out the status of queries by using the sys.query() call, which returns the list of queries tagged in various ways. Our question is whether there is any way to access, e.g., the tag value. In our application, we need to occasionally cancel queries from our UI. The UI has no direct knowledge of the query. While we might be able to find the query we want to cancel by comparing the query text, our automatically generated queries can be very long, and if there's some internal reformatting by Monetdb, we might not get an exact match. So is there anything in the handle or elsewhere that we can use to identify the query we want to cancel? JONATHAN ZINGMAN Manager - TrueCall Data [logos] 2855 Telegraph Ave, Suite 200 Berkeley, CA, 94705-1169 DIRECT: 510.809.0423 CELL: 510.823.5638 www.netscout.com<http://www.netscout.com/>

3 2

MonetDB - Case Sensitive
by Suresh Devarakonda 11 Jun '19

11 Jun '19

Hi, I am new to the MonetDB and interested to know about its case sensitivity. We noticed that objects are being created in lower case if they are not specified in quotes so it appears to me that default case for objects created in Monet is LOWERCASE. But, we got a requirement that objects should be created in upper case even though they are not specified in quotes in DDL statements. In other words, can we make the default to UPPERCASE for all the objects when they created. Please let us know if that is possible. Your early response is very much appreciated. ~Suresh D

2 1

Installation error on AWS Linux
by Gerald Sangudi 04 Jun '19

04 Jun '19

Hello, I'm trying to install MonetDB on AWS Linux. The first command succeeds: sudo yum install https://dev.monetdb.org/downloads/epel/MonetDB-release-epel.noarch.rpm The second command fails: sudo yum install MonetDB-SQL-server5 MonetDB-client Error: *[ec2-user@ip-172-31-23-149 ~]$ sudo yum install MonetDB-SQL-server5 MonetDB-clientLoaded plugins: extras_suggestions, langpacks, priorities, update-motdhttps://dev.monetdb.org/downloads/ <https://dev.monetdb.org/downloads/>epel/2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not FoundTrying other mirror. One of the configured repositories failed (MonetDB 2 - x86_64), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this:* It seems to me there's a reference to epel/2 which is out of date. Should that be epel/7? Is it possible to get an updated rpm? Thanks, Gerald

2 2

REPLICA, CLUSTER
by Ariel Abadi 29 May '19

29 May '19

Hi! Does anybody implemented a REPLICA DB, or a CLUSTER with several nodes for monetdb? I couldnt find too much information in how to implement it. Any information you can provide, will help a lot! Thks in advance Ariel

1 0

MonetDB performance on large table with 20K columns
by Malware Research 16 May '19

16 May '19

Hi, I posted a question on StackOverflow <https://stackoverflow.com/questions/55951563/monetdb-performance-on-a-large…> regarding s specific performance issue. I’m not sure if the right people are monitoring S/O from time to time so I thought I’ll post it here too. Here it is in case you do not want to follow the link: I'm testing MonetDB as a solution for a data-science project. I have a table of 21K columns - all but three are features described as float (32bit) and 6.5M rows (which may or may not become larger, perhaps up to 20M rows). My aim is to use the integrated Python on MonetDB to achieve the ability to train without exporting the data from the DB every time. In addition, queries on specific columns are necessary so the columnar storage can be a significant advantage. I have compiled MonetDB 11.31.13 to gain the embedded Python support. OS is CentOS 7. Storage is not SSD. 48 core server with ~300GB of memory. I created an (unique) index on the table (without analyze). I noticed that when I SELECT * FROM [TABLE_NAME] SAMPLE 50; it takes a long long time to complete. I then tried: SELECT f1, f2, ..., f501 from [TABLE_NAME] SAMPLE 50; SELECT f1, f2, ..., f1001 from [TABLE_NAME] SAMPLE 50; SELECT f1, f2, ..., f2001 from [TABLE_NAME] SAMPLE 50; ... SELECT * from [TABLE_NAME] SAMPLE 50; I ran the queries locally with mclient and used time to measure the amount of time it took and I noticed two things: 1. There is a period where a single core is taking 100% CPU. The more columns the longer it takes to complete. Only when it finishes I can see all cores working, data being consumed, etc... In addition, during that time, the query does not appear in the result of select * from sys.queue(); Eventually, the time needed to get 50 rows from the table was almost 4 hours. 2. The amount of columns is doubled but between each step in the test the amount of time it takes to get a result is tripled. So my questions is: Is this behaviour expected or does it reflect something I did wrong? The data requested from the table should be around 4MB (50 * 21000 * 4Bytes), so this reflects a significant time waiting for such a small amount of data. Help is appreciated! Help is very much appreciated!

2 1

MonetDB mercurial source doesn't compile on macOS Mojave
by Ying Zhang 16 May '19

16 May '19

Dear all, When trying to compile the MonetDB mercurial source on macOS Mojave, it ends with the error: /Users/jennie/monet/Apr2019/common/stream/stream.c:819:10: error: implicit declaration of function 'fdatasync' is invalid in C99 [-Werror,-Wimplicit-function-declaration] && fdatasync(fileno(fp)) < 0 ^ 1 error generated. Does anyone know a fix or workaround for this? Thanks a lot! Jennie

1 1

BPP version problem
by Corbit, Dann 02 May '19

02 May '19

Mdblaunch.bat seems to work: C:\Program Files\MonetDB\MonetDB5>mserver5 --dbpath=dbfarm\voc\ --set mapi_port=50000 # MonetDB 5 server v11.33.3 (Apr2019) # Serving database 'voc', using 12 threads # Compiled for x86_64-pc-winnt/64bit # Found 31.927 GiB available main-memory. # Copyright (c) 1993 - July 2008 CWI. # Copyright (c) August 2008 - 2019 MonetDB B.V., all rights reserved # Visit https://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded > But neither of these batch files work. C:\Program Files\MonetDB\MonetDB5>MSQLserver.bat MonetDB/Python Disabled: Python 3.7 installation not found. !FATAL: BBPinit: incompatible BBP version: expected 061041, got 061033. This database was probably created by a too old version of MonetDB. Press any key to continue . . . C:\Program Files\MonetDB\MonetDB5>M5server.bat MonetDB/Python Disabled: Python 3.7 installation not found. !FATAL: BBPinit: incompatible BBP version: expected 061041, got 061033. This database was probably created by a too old version of MonetDB. Press any key to continue . . . They complain about the server version. I tried deleting the database folder and restarting. The database was rebuilt. But I still get the errors. How do I fix this error? I do not have real database systems. I only have the voc sample data.

3 3

Trying to link IBM Cognos and MonetDB via ODBC
by SERET Paul 25 Apr '19

25 Apr '19

Dears, I get the following error while trying to link IBM Cognos Analytics software to MonetDB via ODBC (sorry, the error message is in french, I translate below) : [cid:image001.png@01D4FB81.C8C4CA40] QE-DEF-0285 The logon failed QE-DEF-0325 The logon failed for the following reason: RQP-DEF-0068 Unable to connect to at least one database during a multi-database attach to 1 database(s) in: 'testDataSourceConnection' UDA-SQL-0031 Unable to access the 'testDataSourceConnection' database. Check that the connection parameters to the database are configured correctly. For example, ensure that the data source connection contains the signon information, such as a password, to connect to the database. UDA-SQL-0107 A general exception has occurred during the operation 'attach(error code -49)'. The specified driver has not been loaded because of system error 127 : The specified procedure cannot be found. (MonetDB ODBC Driver, C:\Program Files (x86)\MonetDB\MonetDB ODBC Driver\lib\libMonetODBC.dll). The last section of the message sounds strange to me, because the DLL seems to be correctly installed from the most recent ODBC driver installer (2019/01/04) : [cid:image002.png@01D4FB84.B9159040] It looks like there is an issue inside the DLL program. Is there a known issue with this DLL or the driver ? IBM Cognos mentions that the ODBC driver must be compliant with the 3.5 standard. I guess this is the case for MonetDB ODBC driver, isn't it ? Could you please have any clue to get this sorted out ? Thanks in advance, best regards Paul SERET

1 0

How can I get a parameterized bulk version of a scalar C UDF to work in SQL?
by aris 25 Apr '19

25 Apr '19

Hi, Say that I have a family of simple scalar transforming function f(x ,p) => y parameterized by p which are implemented as C UDF's and exposed as SQL UDF's. Is it possible to create a bulk version of this? So the scalar C signature is simplified to something like char* f(int* output, int* input, const char** func_parameter); the mal stuff module m; command f(input:int, func_parameter:str):int address f; and in sql we have CREATE FUNCTION f(input INT, func_param STRING) RETURNS INT EXTERNAL NAME m.f; Now for the bulk version I would like to use in C: char* f(bat* output, bat* input, const char** func_parameter); in MAL: module batm; command f(input:bat[:int], func_parameter:str):bat[:int] address f; in SQL: reuse the scalar SQL signature. But that doesn't seem to work. When I compile this and call the udf in a sql query on an actual column, the plan applies the scalar version on each tuple instead of calling the bulk version once. However if I remove the parametrization, i.e. removing the func_parameter from the signature, the bulk version is correctly called. Aris

1 0

Database crash issue
by Amit Ambekar 12 Apr '19

12 Apr '19

Hi, I am using MonetDB from last 2 years, but never faced such issue till now. I am not able to use database ,it is giving error as below: monetdbd: internal error while starting mserver database has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details' Regards, Amit Ambekar Data Engineer

2 2

Get stuck when trying to copy large amounts of data
by Yinjie Lin 02 Apr '19

02 Apr '19

Hi, I'm trying to copy large amounts of data into MonetDB, but the processes would get stuck in the middle. The following is what I did. I have 6 tables, all merge tables partitioned by range of time. And there are hundreds of data files, each has almost 10 million rows and is of size about 3.5 GB. For each table, I use the following command to copy data: `cat file.txt | xargs -n1 -P1 -I {} sh -c "mclient -p 50000 -s \"copy 10000000 records into tbx from stdin best effort\" - < '{}'" ` where file.txt contains paths of files to be loaded. So 6 mclient connections are created to copy data in parallel. But mclient processes would get stuck after loading more than 200 million rows each table. Then I have to kill the processes and restart the database. The environment is CentOS 7.3, 512 GB RAM. MonetDB is compiled using the latest master branch. So does anyone know why this happened? Does MonetDB have a limit on the size of data one instance can hold? Or if there is something wrong with my practice? Thanks in advance! Yinjie Lin

2 1

Performance issue with count distinct
by AUBERT Simon 29 Mar '19

29 Mar '19

Hello all, Here a simple issue I have with performance : 1/ I create a table A and populate it with a bulk load 2/I create a table B, I populate the table with bulk load 3/I create table C with a CTAS statement. I analyze the statistics and set the primary key 4/I try a "count distinct" type query (on all the table) on the PK Field. It is pretty slow, with a high sollicitation of my hard drive for several dozens of second. the "select count" is however fast. So I have these following questions : 1/why despite statistics the count distinct query is slow? I thought the idea of statistics was to get some metrics about the table? 2/if the field I make a count distinct on is the PK (or the set of field is the PK), it's exactly the same than a count non null. So why the time are not the same? 3/if these issues are design issue or bugs, how can I ask to make it solved? For information : Monetdb 11.31.13 on Windows 10 Best regards, Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

2 1

Program contains errors.:(NONE).multiplex
by Niklas B 29 Mar '19

29 Mar '19

Hi, I'm upgrading from an old MonetDB (11.26.0 source snapshot) to the latest proper release. However, my Python aggregate functions appear to have stopped working (Program contains errors.:(NONE).multiplex): Running mserver5 with: /usr/local/bin/mserver5 --daemon=yes --set embedded_py=true --dbpath=/data/monetdb-farm/ --set gdk_nr_threads=1 --set max_clients=6 Reproduce with: CREATE FUNCTION test_udf("conum" int) RETURNS STRING LANGUAGE PYTHON3 {{ }}; Create table test("id" uuid, "date" DATE, "number" INT); select test_udf("number") from test; Gives: Program contains errors.:(NONE).multiplex I'm running it via Docker (stock Debian) with only MonetDB installed via apt-get. Not sure if this is connected: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378 <https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378> Anyone having any similar issues? Tried compiling from source (April 2019 bransch) as well, no luck Regards, Niklas

3 2

sys.storage sorted attribute changes for primary key
by Jeremy Norris 29 Mar '19

29 Mar '19

Hello, I have a large table (~700 million rows, ~150 columns). Initially after import, the database is very fast on primary key lookups, however after a few thousand creates/updates/deletes on the database, "sorted" (as reported by sys.storage), flips from true to false. This results in major performance degradation. What are the conditions under which a database can lose it's sorted-ness on the primary key column? eg: sql>select schema, table, column, sorted from sys.storage where schema = 'test1' and table = 'cmdb' and column = 'pk_id'; +---------+-------+--------+--------+ | schema | table | column | sorted | +=========+=======+========+========+ | test1 | cmdb | pk_id | false | +---------+-------+--------+--------+ Thanks, -Jeremy Norris

2 1

Fwd: Question about partition bounds of updatable merge tables
by Yinjie Lin 06 Mar '19

06 Mar '19

Hi, I found that there is a new feature called updatable merge table in the new release (Updatable merge tables with partition predicates <https://www.monetdb.org/blog/updatable-merge-tables>). I think this makes merge table similar to *partitioned table* in other databases. But I found that the upper bound of range partitioning is an inclusive bound, while the upper bound of some databases like MySQL and PostgreSQL is an exclusive bound. So my question is: why is the upper bound declaration designed like this? Will it be possible if MonetDB be consistent with other databases? Besides, currently merge table doesn't support partition pruning, so all child tables will be scanned even if the filter condition contains the partitioning key. Is it possible if this feature will be added one day? Best Regards, Yinjie Lin

1 0

#!ERROR: GDKmove: cannot rename /mnt/g/monet/wal/sql/log to /mnt/g/monet/wal/sql/log.bak
by Michael Jansen 21 Feb '19

21 Feb '19

Hello, I have a crash after about 5 minutes when running in transaction replication mode. After the crash I cannot access the db. I am running: mserver5 --trace --dbpath=/data/monetdb/dbfarm/test_db/ --set gdk_keep_persisted_log_files=1000 --set gdk_logdir=/mnt/g/monet/wal --set sql_debug=4 I am using GlusterFS under /mnt/g to manage the replication. I am running on AWS Linux 2. and I have free space on in both the data and wal directories. (I assume this is enough space): Filesystem Size Used Avail Use% Mounted on 10.0.1.68:/gv0 8.0G 7.4G 669M 92% /mnt/g bash-4.2$ the db only has one test table: sql>create table t (n float, t text) ; operation successful sql>select * from t ; +--------------------------+------+ | n | t | +==========================+======+ | 1.1 | blah | | 1.1 | blah | | 1 | blah | +--------------------------+------+ 3 tuples any help is appreciated! -mj

1 0

Error when installing on Centos 7
by Trevor Andreas 25 Jan '19

25 Jan '19

Hello, I am getting an error trying to install MonetDB following the instructions here: https://dev.monetdb.org/downloads/epel/ The error I am getting is when running: yum install -y MonetDB-SQL-server5 MonetDB-client failure: repodata/repomd.xml from monetdb: [Errno 256] No more mirrors to try. https://dev.monetdb.org/downloads/epel/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found Looks like the repodata is missing for Centos 7? It looks to be there for Centos 6 https://dev.monetdb.org/downloads/epel/6/x86_64/repodata/ Thanks! Trevor

2 1

Invalid Credentials error
by AUBERT Simon 17 Jan '19

17 Jan '19

Good morning, I have used Monetdb for a few weeks now and this evening I cannot login after a full system restart (windows+monetdb). Environment : Windows 10 (well, it's my work computer, I don't have a lot of choice) Monetdb 5 server v11.31.11 I start monetdb by using C:\Program Files\MonetDB\MonetDB5\bin\mserver5.exe Then mclient on C:\Program Files\MonetDB\MonetDB5\mclient.bat (and usually it work just fine) I also tried with mclient.exe using the -x command to debug. Error message : C:\Program Files\MonetDB\MonetDB5\bin>mclient -X -d demo user(win32):monetdb password: MAPI = monetdb@localhost:50000 ACTION= mapi_reconnect ERROR = !InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' Of course, I use the default monetdb/monetdb credentials. I have no idea of what happens and moreover I have lost an entire schema (I see that if I connect through Tableau Desktop...). I have done an upgrade on the last version (11.31.13) and same result, impossible to connect. And same result. I spend the entire evening on that a Thank in advance. Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

2 3

table-returning function with subselect in input - no mitosis?
by Roberto Cornacchia 14 Jan '19

14 Jan '19

Hi, * Background: I have a custom string tokenizer that I invoke this way: SELECT * FROM tokenize( (SELECT id, s FROM my_strings) ); The output has of course a 1-to-N rows relation with the input. The function has a bulk implementation, so it takes M rows and it outputs M rows. * The question: No matter how large the input table my_strings is, mitosis does not seem to kick in. Is it not allowed in this case because it cannot assume that my custom function allows to concatenate partial results? Do you see any chance for me to enable mitosis with this scenario? Thanks, Roberto

1 1

Announcement: New Aug2018-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Jan '19

14 Jan '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP2 bugfix release (11.31.13) SQL Frontend * Disabled function sys.getcontent(url). Bug Fixes * 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered * 6645: optimizer treats all the functions with no or constant parameters as constant * 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s * 6651: Multi-column IN clause for subquery produces wrong results * 6653: CREATE TABLE accepts empty table/column name * 6654: Incorrect handling of 'TRUE' in compound select * 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message * 6660: GRANTing a ROLE is not idempotent * 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt. * 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489 * 6665: Creation of serial types does not accept negative numbers * 6666: COPY INTO from .. LOCKED doubles input data * 6668: The SAMPLE key word doesn't work in a subquery. * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value

1 0

Re: Performance degradation
by Jeremy Norris 12 Jan '19

12 Jan '19

Here's some additional observations on this puzzling issue: The slow table is "cmdb", the normal table is "cmdb2". Both have the same number of rows according to "SELECT count(*)" (73553310). However, sys.statistics shows something a bit different: sql>SELECT x.name, count(*) AS columns_with, z.count FROM sys.tables AS x, sys.columns AS y, sys.statistics AS z WHERE x.id = y.table_id AND y.id = z.column_id GROUP BY x.name, z.count; +-------+--------------+----------+ | name | columns_with | count | +=======+==============+==========+ | cmdb | 8 | 73553310 | | cmdb | 145 | 73553311 | | cmdb2 | 153 | 73553310 | +-------+--------------+----------+ 3 tuples sql> For the slow table, "cmdb", there are 145/153 columns that have an extra "row". This row count mismatch causes it to skip parallelization optimizations in "monetdb5/optimizer/opt_mitosis.c", line 212. A few related questions: 1) In the columnar model, is is valid for conceptual tuples to exist which don't have values for each column like this? (This seems strange as now there's a difference between an explicit NULL and "not existing"). How might this situation have happened? (Only transactional full-row inserts were applied to this DB). Can this be repaired? 2) Is the conditional in "monetdb5/optimizer/opt_mitosis.c", line 212 sufficiently robust? I may be misunderstanding the full context of this line, but it seems like we could still benefit from splitting/parallelization when r < rowcnt, no? Thanks, -Jeremy Norris

1 0

Performance degradation
by Herman Knief 10 Jan '19

10 Jan '19

Greetings, Running MonetDB v11.31.11 (Aug2018-SP1), we ran into an issue where queries that have been running quite well, suddenly slowed by about 5x. Looking at the explain, it seems we hit a condition that is preventing mitosis? Copying the tables related to the query seems to restore performance to the previous level, but would be very curious to understand what happened here. What metrics or stats should we be looking at? Changes we see in the plan are as follows: Bad example: sql.tid(X_4:int, "waltest":str, "cmdb":str); Good example: sql.tid(X_4:int, "waltest":str, "cmdb2":str, 0:int, 32:int); Looking at the storage info, the sizes (heap and column), type width and counts look the same between the two tables. Any thoughts or suggestions would be appreciated. - Herman -- Please excuse typos... I can't type.

1 0

Embedded Python module is not available in recent builds of Windows version
by Kevin Chen 21 Dec '18

21 Dec '18

Hi, I upgraded the MonetDB windows version to 2018Aug and found that the embedded Python module is no longer working. Even the startup script with “embedded_py=true”, the module was not loaded. There was no message showing “MonetDB/Python2 module loaded”. All my Python functions are failed. Please help. Thanks. Kevin

2 1

!FATAL: SQL type wrd missing
by Ariel Abadi 18 Dec '18

18 Dec '18

Hi! Please I need an urgent solution (if there is one) I have upgraded a database to the latest version, and when Im trying to start the DB, there is an error: 2018-12-18 10:58:06 ERR DB[29818]: !FATAL: SQL type wrd missing Can someone help me with any clue!???? Thks in advance Ariel

2 3

Re: grafana-monetdb-datasource installation problem
by Svetlin 03 Dec '18

03 Dec '18

Hi Ariel, The current version of the grafana-monetdb-datasource plugin cannot be used with the latest grafana release. There is a missing breaking dependency that has being introduced(see #10053 <https://github.com/grafana/grafana/issues/10053>) and we didn’t have time to patch that in. Secondly, you would need a http server as a mediator to forward the queries to monetdb. The grafana monetdb datasource plugin merely creates MonetDB compatible queries and forwards responses back to grafana core for visualization. I hope this helps. Cheers, Svetlin > On Dec 3, 2018, at 11:03 AM, Ying (Jennie) Zhang <ying.zhang(a)monetdbsolutions.com> wrote: > >> aabadi(a)starconnecting.com <mailto:aabadi@starconnecting.com>

2 1

grafana-monetdb-datasource installation problem
by Ariel Abadi 01 Dec '18

01 Dec '18

Does anybody has installed the monetdb datasource plugin for Grafana? Im running Grafana v5.3.4. The plugin was downloaded from https://github.com/MonetDBSolutions/grafana-monetdb-datasource And is not working... :'( Does anybody knows how to connect MonetDB within Grafana? Any ideas will be very much appreciated!!! Thks Ariel

2 1

Re: Issues trying to create a table...
by Stefan Manegold 28 Nov '18

28 Nov '18

Hi, Internally (as in in the MonetDB code base),the impact of the problem could imho be significantly reduced (though not fully avoided)by "simply" using different separator characters (or even strings?) when creating qnames,e.g. schema:table;column insteadof schema_table_column ... A unique bijective reversible mapping from schema,table,column to qname should avoid the problem entirely , but might require to use separator chracter(s) that are not valid in SQL identifiers, if such exist ... Best,Stefan -------- Original message --------From: Ying Zhang <Y.Zhang(a)cwi.nl> Date: 28/11/2018 07:33 (GMT+01:00) To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: Re: Issues trying to create a table... > On 28 Nov 2018, at 04:18, Herman Knief <hknief(a)gmail.com> wrote: > > Hi Folks, > I'm new to MonetDB and trying to setup a schema from a MariaDB export. > I have a particular table that reports a conflict with the create statement. I've stripped this down to just a particular field that seems to be causing the issue, but unsure where to go from here. > > sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script" string); > CREATE TABLE: zboot_sc_cat_item_delivery_plan_script conflicts Hai Herman, This is because something has already been defined before, whose complete qname is “zboot_sc_cat_item_delivery_plan_script”. This often happens with (but probably not limited to) schema_name+table_name+column_name. In MonetDB test suite, we have two examples to trigger this (type of) error: sql>create table foo (id integer, bar_id integer); operation successful sql>create table foo_bar (id integer); CREATE TABLE: sys_foo_bar_id conflicts and sql>create table foo_bar (baz int); operation successful sql>create table foo (bar_baz int); CREATE TABLE: sys_foo_bar_baz conflicts This is not a bug. You need to change your table and/or column definitions, like your alternatives below. Then CREATE TABLE will succeed. Regards, Jennie > > What's interesting is that either changing the name of the table or the name of the field and these statements work. There is something about this particular combo that is problematic. > sql>CREATE TABLE "sc_cat_item_delivery_plan1" ("script" string); > operation successful > sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script1" string); > operation successful > > I have a total of 2251 tables... all were created except for this one. I've tried to destroy and recreate the schema with no luck. Logs don't seem to indicate any issue and trying to use the debug on the client didn't get me anywhere. Thoughts? Suggestions? > > -- > Please excuse typos... I can't type. > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

2 1

Issues trying to create a table...
by Herman Knief 28 Nov '18

28 Nov '18

Hi Folks, I'm new to MonetDB and trying to setup a schema from a MariaDB export. I have a particular table that reports a conflict with the create statement. I've stripped this down to just a particular field that seems to be causing the issue, but unsure where to go from here. sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script" string); CREATE TABLE: zboot_sc_cat_item_delivery_plan_script conflicts What's interesting is that either changing the name of the table or the name of the field and these statements work. There is something about this particular combo that is problematic. sql>CREATE TABLE "sc_cat_item_delivery_plan1" ("script" string); operation successful sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script1" string); operation successful I have a total of 2251 tables... all were created except for this one. I've tried to destroy and recreate the schema with no luck. Logs don't seem to indicate any issue and trying to use the debug on the client didn't get me anywhere. Thoughts? Suggestions? -- Please excuse typos... I can't type.

3 2

Issue while connecting to MonetDB via Perl cpan driver
by Sharma, Sreejith 15 Nov '18

15 Nov '18

Hi, We have a Perl code via which we are trying to connect to one of the MonetDB instance. Here we are using DBD::monetdb Perl cpan driver for connectivity. when we run the snippet, it just tries to connect to DB and seems hung, looks like it's awaiting for user input for password. Also, on the server side logs we see an error (challenge error) can connection will not be established. Here is a sample snippet which isn't working. Could anyone help with an idea or code snippet which works with Perl DBI for MonetDB connectivity? [cid:image001.png@01D47D35.C23AD040] #!/usr/bin/perl # use strict; use warnings; use DBI(); print "Attempting to connect...\n"; my $dbh = DBI->connect('dbi:monetdb:host=XXXX;port=55555', 'USERNAME', 'Password', { RaiseError => 1 } ); print "Connected. Trying SQL...\n"; my $sth = $dbh->prepare('SELECT * FROM env'); $sth->execute; $sth->dump_results; print "Done\n"; exit 0; Regards, Sreejith

1 0

Error: cannot bind delete column
by Doherty, Adam 15 Nov '18

15 Nov '18

Hello all: I have run across a show-stopping error, "cannot bind delete column" which renders Monet unusable until I restart the service. A batch import of events runs every 10 minutes and has been unable to complete for the last 28 hours because the service is not accepting commands. I'm not sure where to look for the source of this error. Restarting the service correct the issue, but this is a problem as the Monet instance drives live Tableau dashboards using this imported data. Where do I need to start looking to track down the source of this error? I am currently running 11.29.3 on CentOS 7. Adam

2 1

Adding columns to tables via binary bulk loading
by Anderson, David B 12 Nov '18

12 Nov '18

All, Is it possible to add a column to an existing table via binary bulk loading? We have a table with ~1 billion rows and we would like to add columns to the table, but would prefer bulk loads for speed. Also, does anyone have a snippet of code to create the binary file format that bulk loading expects. I've tried a couple versions (simple int and double), but the server wouldn't recognize the files. Thanks, Dave

2 1

Unable to get sys.queue for all the users
by Murthy, Gautham 07 Nov '18

07 Nov '18

Hello, We are unable to get the information for all active users sessions when we use sys.queue as monetdb user. Also, the tried as user other than monetdb, it returned no access, We had provided the user select privilege on the sys.queue view. [monetadmin@lnx1487 scripts]$ /monet_binaries/MonetDB-11.27.13_PY/bin/mclient mapi:monetdb://lnx1487:50053/DB_BWS_S Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4) Database: MonetDB v11.27.13 (Jul2017-SP4), 'mapi:monetdb://lnx1487:50053/DB_BWS_S' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select "user","active",count(*) as "No. Of Connections" from sys.sessions() group by "user","active"; +-------------------+--------+--------------------+ | user | active | No. Of Connections | +===================+========+====================+ | DM_POS_BWS_STG_P6 | true | 1 | | DM_POS_BWS_STG_P6 | false | 3 | | monetdb | true | 1 | +-------------------+--------+--------------------+ 3 tuples (6.027ms) sql>select * from sys.queue; +------+---------+----------------------------+----------------------------+----------+---------+--------+--------------------------+ | qtag | user | started | estimate | progress | status | tag | query | +======+=========+============================+============================+==========+=========+========+==========================+ | 3630 | monetdb | 2018-11-07 17:07:34.000000 | null | null | running | 3630@0 | select * from sys.queue; | +------+---------+----------------------------+----------------------------+----------+---------+--------+--------------------------+ 1 tuple (1.610ms) sql>\q [monetadmin@lnx1487 scripts]$ Thank You, Gautham

4 3

Cannot setup monetdb on Centos 7
by GAUQUELIN, Philippe 05 Nov '18

05 Nov '18

Hi Need help on setup. Thanks for your answer Trying to install I got this message. The GPG keys listed for the "MonetDB 7 - x86_64" repository are already installed but they are not correct for this package. More details below: []# yum reinstall https://dev.monetdb.org/downloads/epel/MonetDB-release-epel.noarch.rpm Loaded plugins: fastestmirror, langpacks MonetDB-release-epel.noarch.rpm | 6.4 kB 00:00:00 Examining /var/tmp/yum-root-x4i3eb/MonetDB-release-epel.noarch.rpm: MonetDB-release-epel-1.3-1.monetdb.noarch Resolving Dependencies --> Running transaction check ---> Package MonetDB-release-epel.noarch 0:1.3-1.monetdb will be reinstalled --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Reinstalling: MonetDB-release-epel noarch 1.3-1.monetdb /MonetDB-release-epel.noarch 4.0 k Transaction Summary ======================================================================================================================================================================= Reinstall 1 Package Total size: 4.0 k Installed size: 4.0 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : MonetDB-release-epel-1.3-1.monetdb.noarch 1/1 Verifying : MonetDB-release-epel-1.3-1.monetdb.noarch 1/1 Installed: MonetDB-release-epel.noarch 0:1.3-1.monetdb Complete! [root@franalytic02 tmp]# []# rpm --import https://www.monetdb.org/downloads/MonetDB-GPG-KEY []# yum install MonetDB-SQL-server5 MonetDB-client Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: centos.crazyfrogs.org * epel: mirrors.ircam.fr * extras: centos.crazyfrogs.org * updates: centos.mirror.fr.planethoster.net Resolving Dependencies --> Running transaction check ---> Package MonetDB-SQL-server5.x86_64 0:11.31.11-20181011.el7 will be installed --> Processing Dependency: MonetDB5-server(x86-64) = 11.31.11-20181011.el7 for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libstream.so.11()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libmonetdb5.so.25()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libbat.so.17()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 ---> Package MonetDB-client.x86_64 0:11.31.11-20181011.el7 will be installed --> Running transaction check ---> Package MonetDB.x86_64 0:11.31.11-20181011.el7 will be installed ---> Package MonetDB-stream.x86_64 0:11.31.11-20181011.el7 will be installed ---> Package MonetDB5-server.x86_64 0:11.31.11-20181011.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Installing: MonetDB-SQL-server5 x86_64 11.31.11-20181011.el7 monetdb 854 k MonetDB-client x86_64 11.31.11-20181011.el7 monetdb 275 k Installing for dependencies: MonetDB x86_64 11.31.11-20181011.el7 monetdb 1.0 M MonetDB-stream x86_64 11.31.11-20181011.el7 monetdb 197 k MonetDB5-server x86_64 11.31.11-20181011.el7 monetdb 853 k Transaction Summary ======================================================================================================================================================================= Install 2 Packages (+3 Dependent packages) Total size: 3.1 M Installed size: 9.1 M Is this ok [y/d/N]: y Downloading packages: warning: /var/cache/yum/x86_64/7/monetdb/packages/MonetDB-stream-11.31.11-20181011.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID df0e54f3: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Importing GPG key 0x0583366F: Userid : "MonetDB Database System Packager <monet(a)cwi.nl>" Fingerprint: 213e 64dc d5dd c2c0 63cc 39fd 053c 3ed4 0583 366f Package : MonetDB-release-epel-1.3-1.monetdb.noarch (installed) >From : /etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Is this ok [y/N]: y Importing GPG key 0x0583366F: Userid : "MonetDB Database System Packager <monet(a)cwi.nl>" Fingerprint: 213e 64dc d5dd c2c0 63cc 39fd 053c 3ed4 0583 366f Package : MonetDB-release-epel-1.3-1.monetdb.noarch (installed) >From : /etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Is this ok [y/N]: y Public key for MonetDB-stream-11.31.11-20181011.el7.x86_64.rpm is not installed Failing package is: MonetDB-stream-11.31.11-20181011.el7.x86_64 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Philippe

2 1

Using capi.eval from MAL
by Renato Marroquín Mogrovejo 04 Nov '18

04 Nov '18

Hi all, I am trying to use the following patten from MAL[1] pattern eval(fptr:ptr,cpp:bit,expr:str,arg:any...):any... However, I am not sure how I should create the first parameter <fptr:ptr> from MAL. I am just trying to be able to call a C-UDF from MAL. From capi.c, it would seem that it is the client's context but I don't know how to get a handle of that. Any pointers are highly appreciated. Thanks! Renato M. [1] https://github.com/MonetDB/MonetDB/blob/2310e49a349ebcb8ff5b5c585928c8f4bd1…

2 4

Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
by Murthy, Gautham 04 Nov '18

04 Nov '18

Hello, We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data. sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$ [monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$ Thank You, Gautham

3 3

Append an array of values to a BUN
by Anton Kravchenko 02 Nov '18

02 Nov '18

Hi there, Is there a way to append an array of values to a BUN, instead one value at a time? sql_table *tbl = mvc_bind_table(m, sch, "atable"); sql_column *col = mvc_bind_column(m, tbl, "acolumn"); BAT *tmp1 = COLnew(0, TYPE_lng, (BUN)nrows, TRANSIENT); long long v_long[3] = {100000000009,100000000010,100000000011}; for (irow=0; irow<nrows; irow++) BUNappend(tmp1, &v_long[irow], FALSE); mvc_append_column(m->session->tr, col1, tmp1); mvc_commit(m, 0, tr->name); bat_destroy(tmp1); Thanks, Anton

2 5

Circular Memory Only Database
by Doherty, Adam 26 Oct '18

26 Oct '18

Hello all: I am wondering if anyone has a remote table strategy such that "hot data" is stored in a table that is backed by RAM. I have a table with approximately 1 billion events that grows by approximately 1,000 events a minute. Table is a merge table of merge tables, the parent merge table has a child merge table for every year of data under those child tables are source tables that make up data for areas. So Area1_2017, Area2_2017 -> Events_2017 Area1_2018, Area2_2018 -> Events_2018 Events_2017, Events_2018 -> Events I am finding that as I add more data the system seems to slow down. I am wanting to keep a cyclical table of say the last 6 months from the Events table in a table backed solely by RAM So a query to Recent_Events would remain very fast or at the very least constant. This should be achievable with tmpfs. I am currently running 11.29.3 on CentOS 7 on VMware 6.7 Server has 128GB of RAM and 16 cores. Adam

2 1

SQLSTATE Error not showing up in mclient Aug2018 version (Aug2018 (11.31.7))
by Sharma, Sreejith 22 Oct '18

22 Oct '18

All, Do we have a way to compile mclient along with server source code? We are not able to see SQLSTATE error code on concurrency with Aug2018 (11.31.7) server version along with mclient Aug2018. Would like to see if we need to specifically compile client too? ERROR = !COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead CODE: 40000 Regards, Sreejith

2 3

Re: users-list Digest, Vol 74, Issue 3
by Oxana Dolmatova 17 Oct '18

17 Oct '18

Hi Stefan, Thank you for the information, now I see the point. >Did you by any chance try any other DBMSs with your "non-standard" workload? >If so, how do those experiences and limitations compare to MonetDB? I did not run tests over conventional DBMSs because they have stricter limits compared to MonetDB (e.g. Postgres is limited to 1600 columns), but only over array DBMSs or math packages. Obviously, their column scalability is better, but they can not handle RA that efficiently. Best regards, Oksana On Tue, 16 Oct 2018 at 12:16, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Column limit (Oxana Dolmatova) > 2. Re: Column limit (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 15 Oct 2018 17:14:14 +0200 > From: Oxana Dolmatova <oxana.dolmatova(a)gmail.com> > To: E.Rozenberg(a)cwi.nl > Cc: users-list(a)monetdb.org > Subject: Re: Column limit > Message-ID: > <CAJJ2wAQi+4S+= > e9BfbStuJxD7izNh8z-mcBF8FYKxagjj5oRkA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Eyal, > > Sorry for the late reply. Yes, there is no problem with large number of > columns in sparse representation of arrays (i.e., dimension, dimension, > value). > However, in order to perform relational operations, this structure should > be pivoted first. When matrices are represented by tables, both algebras > can be applied to the same object, > so I'm specifically looking for the main bottleneck of processing hundreds > of thousands of columns in MonetDB. > > Best regards, > Oksana > > > On Wed, 10 Oct 2018 at 13:40, Eyal Rozenberg <E.Rozenberg(a)cwi.nl> wrote: > > > If you want to model an NxM matrix of elements of type MYTYPE in a DBMS > > (not just MonetDB), you would typically have a table with 3 columns: > > > > CREATE TABLE my_matrix ( > > "row" INTEGER NOT NULL, > > "col" INTEGER NOT NULL, > > "cell_value" MYTYPE NOT NULL, > > PRIMARY KEY ("row", "col") > > ); > > > > of length N x M. > > > > Anyways, I'm not the expert on Arrays in our group so I'll let others > > chime in. > > > > Eyal > > > > > > On 10/10/2018 13:01, Oxana Dolmatova wrote: > > > Hello, > > > > > > Thank you for all the replies and information! > > > The version is 11.23, plus I'm using the cloud platform > > with 4cpu-16ram-hpc. > > > The number of rows was just 1K for 15K attributes in a table. > > > I understand that MonetDB was made with no intention to create and > > > process tables with that many columns. But I'm interested in matrix > > > operations integration, so I would like to know what is the main > > > bottleneck to process such data (matrix-like sized) in the system. > > > > > > Best regards, > > > Oksana > > > > > > > > > On Tue, 9 Oct 2018 at 21:02, Eyal Rozenberg <eyalroz(a)technion.ac.il > > > <mailto:eyalroz@technion.ac.il>> wrote: > > > > > > I'll add two observations to what Zhang Ying said. > > > > > > 1. Internally in MonetDB, the index of a column (or a BAT) in > > MonetDB's > > > BBP (BAT Buffer Pool) is a signed 32-bit integer (on most > > platforms), so > > > there isn't a theoretical problem having more than 15,000 columns; > > but > > > MonetDB was not designed with that many tables in mind. The "long > > > dimension" is expected to be the length of columns. > > > > > > 2. When you have as many as 15,000 columns, you're not simply > finding > > > more attributes of the same items. So, for example, a person has a > > first > > > name, last name, year of birth, etc. etc. - but after several > dozens > > of > > > these you're probably adding columns such as "data relevant to this > > > person at point X", "data relevant to this person at point Y" and > so > > on. > > > > > > In your case, I'm guessing you have a lot of points and hence a lot > > of > > > columns with data for your different items. What you should > consider > > > doing instead is having an extra "coordinate" column. Thus, in the > > > example I gave, I would drop most of the columns, in favor of a > > second > > > table named "person_data". This table's columns would be > "person_id" > > (or > > > some other key columns), "sample_point" and "value". Thus instead > of > > M > > > columns and N records you would have N x M records with 3 columns. > > > > > > Eyal > > > > > > > > > > > > On 09/10/2018 16:50, Ying Zhang wrote: > > > > Hai Oksana, > > > > > > > > A.f.a.I.k., there is no explicit limit on the #columns you can > put > > > in a relation (did you mean ?table??). The reason that mserver5 > > > stopped responding is probably because it is busy with I/O. > > > > > > > > A long time ago, I once tried to create 1M tables, each with 3 > > > columns. After about a day or something, MonetDB has created ~400K > > > tables, and I could hear my HDD being constantly busy. MonetDB was > > > still working, but that was already a sufficient evidence that that > > > idea wasn?t going to work. > > > > > > > > So the question is actually what do you want to achieve? > > > > > > > > Regards, > > > > Jennie > > > > > > > > > > > >> On 9 Oct 2018, at 13:43, Oxana Dolmatova > > > <oxana.dolmatova(a)gmail.com <mailto:oxana.dolmatova@gmail.com>> > > wrote: > > > >> > > > >> Hello, > > > >> > > > >> Could you tell the limitation for number of columns in a > relation > > > and the main reason for that limitation. So far I could create > > > relations with up to 15000 columns. When the number of columns is > > > bigger mserver5 stops responding. > > > >> > > > >> Thank you and best regards, > > > >> Oksana Dolmatova > > > >> _______________________________________________ > > > >> users-list mailing list > > > >> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > >> https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > _______________________________________________ > > > > users-list mailing list > > > > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Oksana Dolmatova > > > > > > _______________________________________________ > > > users-list mailing list > > > users-list(a)monetdb.org > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > -- > Best regards, > Oksana Dolmatova >

1 0

Announcement: New Aug2018-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 17 Oct '18

17 Oct '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP1 bugfix release (11.31.11) Build Environment * On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object. Whole System * Some subtle dependencies between RPMs have been fixed. Bug Fixes * 6640: timestamp_to_str returning incorrectly adjusted results * 6641: race condition in SQL UDF with update * 6642: Hanging query * 6648: key property potentially wrong after type conversion * 6649: Projection inside within transaction gives wrong results

1 0

Column limit
by Oxana Dolmatova 15 Oct '18

15 Oct '18

Hello, Could you tell the limitation for number of columns in a relation and the main reason for that limitation. So far I could create relations with up to 15000 columns. When the number of columns is bigger mserver5 stops responding. Thank you and best regards, Oksana Dolmatova

6 7

monetdb discover not finding other db's
by Frik Lubbe 24 Sep '18

24 Sep '18

Hi all, I relatively new to MonetDB, but have many year exposure to database enignes. I have 2 x vm's, almost identical with the exception of IP's and host names. both are Ubuntu Server 18.04 LTS. I have installed MonetDB on both VM's and created a db farm and a database on each. When running 'monetdb discover' on any of the VM's it finds nothing. Any help is welcome please. Regards, Frik Outputs from 'monetdbd get all /data/monetdb/dbfarm/' *VM 1:* root@ubuntu-srv-001:~# monetdbd start /data/monetdb/dbfarm/ another monetdbd is already running root@ubuntu-srv-001:~# monetdbd get all /data/monetdb/dbfarm/ property value hostname ubuntu-srv-001 dbfarm /data/monetdb/dbfarm/ status monetdbd[1941] 1.7 (Aug2018) is serving this dbfarm mserver /usr/bin/mserver5 logfile /data/monetdb/dbfarm//merovingian.log pidfile /data/monetdb/dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 50000 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 root@ubuntu-srv-001:~# monetdb discover root@ubuntu-srv-001:~# *VM 2:* root@ubuntu-srv-002:~# monetdbd get all /data/monetdb/dbfarm/ property value hostname ubuntu-srv-002 dbfarm /data/monetdb/dbfarm/ status monetdbd[2373] 1.7 (Aug2018) is serving this dbfarm mserver /usr/bin/mserver5 logfile /data/monetdb/dbfarm//merovingian.log pidfile /data/monetdb/dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 60001 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.60001 controlsock /tmp/.s.merovingian.60001 root@ubuntu-srv-002:~#

1 0

SQLcopyinto.java
by Roberto Cornacchia 20 Sep '18

20 Sep '18

I'm trying to debug n issue I have with data disappearing after a database restart. Because part of the process for creating the table involves a mixed Mapi / JDBC approach as described in monetdb-java/example/SQLcopyinto.java, I started from this example. When I run it though, I get a final count of 0 (should be 100). Can anyone confirm? Mar2018, JDBC 2.28 Roberto

2 1

Join Order Benchmark on MonetDB
by Junxiong Wang 19 Sep '18

19 Sep '18

Hello, I am currently evaluating the performance of MonetDB on the Join Order Benchmark, proposed by Leis et al.(http://www.vldb.org/pvldb/vol9/p204-leis.pdf <http://www.vldb.org/pvldb/vol9/p204-leis.pdf>). There are a few queries where performance seems suboptmal, e.g., query 16b and 17e (https://github.com/gregrahn/join-order-benchmark <https://github.com/gregrahn/join-order-benchmark>): MonetDB takes around 400 seconds to evaluate query 17e with a single thread, more than 20 seconds with 48 threads, Postgres takes only around 9 seconds. I suspect that the query optimizer selects the wrong plan, this is the output of the plan command. | project ( | | | group by ( | | | | join ( | | | | | join ( | | | | | | join ( | | | | | | | join ( | | | | | | | | join ( | | | | | | | | | join ( | | | | | | | | | | table(sys.movie_companies) [ "movie_companies"."movie_id" NOT NULL as "mc"."movie_id", "movie_companies"."company_id" NOT NULL as "mc"."company_id" ] COUNT , | | | | | | | | | | select ( | | | | | | | | | | | table(sys.company_name) [ "company_name"."id" NOT NULL HASHCOL as "cn"."id", "company_name"."country_code" as "cn"."country_code" ] COUNT | | | | | | | | | | ) [ "cn"."country_code" = varchar(255) "[us]" ] | | | | | | | | | ) [ "mc"."company_id" NOT NULL = "cn"."id" NOT NULL HASHCOL ], | | | | | | | | | table(sys.title) [ "title"."id" NOT NULL HASHCOL as "t"."id" ] COUNT | | | | | | | | ) [ "t"."id" NOT NULL HASHCOL = "mc"."movie_id" NOT NULL ], | | | | | | | | table(sys.cast_info) [ "cast_info"."person_id" NOT NULL as "ci"."person_id", "cast_info"."movie_id" NOT NULL as "ci"."movie_id" ] COUNT | | | | | | | ) [ "ci"."movie_id" NOT NULL = "t"."id" NOT NULL HASHCOL , "ci"."movie_id" NOT NULL = "mc"."movie_id" NOT NULL ], | | | | | | | table(sys.name) [ "name"."id" NOT NULL HASHCOL as "n"."id", "name"."name" NOT NULL as "n"."name" ] COUNT | | | | | | ) [ "n"."id" NOT NULL HASHCOL = "ci"."person_id" NOT NULL ], | | | | | | table(sys.movie_keyword) [ "movie_keyword"."movie_id" NOT NULL as "mk"."movie_id", "movie_keyword"."keyword_id" NOT NULL as "mk"."keyword_id" ] COUNT | | | | | ) [ "t"."id" NOT NULL HASHCOL = "mk"."movie_id" NOT NULL, "ci"."movie_id" NOT NULL = "mk"."movie_id" NOT NULL, "mc"."movie_id" NOT NULL = "mk"."movie_id" NOT NULL ], | | | | | select ( | | | | | | table(sys.keyword) [ "keyword"."id" NOT NULL HASHCOL as "k"."id", "keyword"."keyword" NOT NULL as "k"."keyword" ] COUNT | | | | | ) [ "k"."keyword" NOT NULL = clob "character-name-in-title" ] | | | | ) [ "mk"."keyword_id" NOT NULL = "k"."id" NOT NULL HASHCOL ] | | | ) [ ] [ sys.min no nil ("n"."name" NOT NULL) NOT NULL as "L3"."L3" ] | | ) [ "L3"."L3" NOT NULL as "L4"."member_in_charnamed_movie" ] | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I was wondering whether anyone might have a hint about what the problem could be and how one could improve performance for those specific queries? Thanks a lot!

1 0

Measure the memory footprint of Monetdb from an SQL client
by Mahmoud Mohsen 12 Sep '18

12 Sep '18

Hello, Is there away to measure the physical or virtual memory consumption of Monetdb using an SQL query. Something similar to select * from storage() where table_name = name; I believe storage() helps with the disk consumption not main memory consumption, am I right? Thank you a lot in advance, Mahmoud Mohsen

2 1

COPY INTO FILE - Folder permission
by Ariel Abadi 10 Sep '18

10 Sep '18

Hi! We are running some "exports" using the COPY INTO FILE, the problem we are facing is that the file created by monetdb is Read Write ONLY by the user running the database. -rw------- 1 root monet 9477 Sep 7 15:08 exp.csv Is there any way so the created file has also RW permission to the group and NOT only the user. In this case, we are using an NFS share so we are not able to use setfacl. Does anybody has an idea how to solve this ? Thanks a lot in advance! Rgds Ariel

5 8

About Monetdb Query Plan
by Junxiong Wang 07 Sep '18

07 Sep '18

Hello, I am currently working on a project related to query optimization I have several questions, 1) Is it possible to force the query engine to use the specified join order? 2) When I change the table order in the “FROM” clause, the query time varies a lot (many orders of magnitude)? Does the table order in the “FROM” clause affect the query plan? 3) I want to know the details of MonetDB query plan. When I use “explain” command, I just get the MAL code. Are there any tools to visualize it, just like PostgreSQL “analyze" command? Thanks a lot.

2 1

About Monetdb Query Plan
by Junxiong Wang 06 Sep '18

06 Sep '18

Hello, I am currently working on a project related to query optimization I have several questions, 1) Is it possible to force the query engine to use the specified join order? 2) When I change the table order in the “FROM” clause, the query time varies a lot (many orders of magnitude)? Does the table order in the “FROM” clause affect the query plan? 3) I want to know the details of MonetDB query plan? But when I use “explain” command, I just get the MAL code. Are there any tools to visualize it, just like PostgreSQL “analyze" command? Thanks a lot.

1 0

How to enable SQLSTATE error code in mclient
by Murthy, Gautham 06 Sep '18

06 Sep '18

Hello, How do we enable SQLSTATE error code on mclient ? Thank You, Gautham

1 0