- users-list - monetdb.org

Re: users-list Digest, Vol 89, Issue 2
by Kalaiarasi Gowtham 13 Jan '20

13 Jan '20

Hi Jenni, Please, find the below steps I have followed to install monetdb *Installation of MonetDB in CentOS 7* *Installing dependencies* # yum update # yum install python autoconf automake libtool bison openssl openssl-devel gettext-devel git # git clone https://github.com/MonetDB/MonetDB.git # cd /MonetDB # ./bootstrap # cd .. # mkdir testdir # cd testdir *To enable py3integration python version 3 should be installed* # yum install centos-release-scl # yum install rh-python36 # scl enable rh-python36 bash *check python version* # python -V then # ../MonetDB/configure –enable-py3integration=yes # make # make install Thanks and regards, Kalai On Wed, 8 Jan 2020 at 16:31, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: pymonetdb issue (ferreira) > 2. Re: Monetdb (Ying Zhang) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 7 Jan 2020 17:22:59 +0000 > From: ferreira <pedro.ferreira(a)monetdbsolutions.com> > To: Communication channel for MonetDB users <users-list(a)monetdb.org>, > ALEX OSSIPOV <alexo15(a)comcast.net> > Subject: Re: pymonetdb issue > Message-ID: > <0e74a920-b2ad-2993-b1dc-fcd7a49105bb(a)monetdbsolutions.com> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Hello Alex, > > I just looked at pymonetdb source code and indeed it does not support > IPv6 yet. I'm going to create an issue to request for it. The reason is > because in the past, the MAPI protocol (which MonetDB uses for socket > connections) didn't support IPv6, so the clients (Python, Ruby, JDBC, > etc.) never had to implemented them. However they can now add this option. > > Best regards, > > Pedro > > On 12/22/19 5:30 AM, ALEX OSSIPOV wrote: > > Hi All > > I installed pymonetdb v 1.2.1 and trying to connect to remote > > server?MonetDB v11.35.3 (Nov2019) listening on IPV6 only. I have got > > next error in python client: > > >>> mapi_connection.connect(hostname="<server_name>", port=xxxxx, > > username="monetdb", password="monetdb", database="demo", > > language="sql", unix_socket=None, connect_timeout=-1) > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > File "/usr/local/lib/python3.7/site-packages/pymonetdb/mapi.py", line > > 143, in connect > > self.socket.connect((hostname, port)) > > socket.gaierror: [Errno -2] Name or service not known > > >>> > > My pc client is also using ipv6 to connect to MonetDB server. I also > > have jdbc connection that works fine. > > Question: > > Does it mean pymonetdb does not support ipv6 only environment? > > What would be the reason if it has nothing to do with protocol? > > > > Thanks > > Alex > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list >

2 2

Monetdb
by Kalaiarasi Gowtham 07 Jan '20

07 Jan '20

Hi, Currently, we are trying to start the database created in monetdb using the following command monetdb start <dbname> monetdbd: internal error while starting mserver 'database <dbname> did not initialise the sql scenario', please refer to the logs Getting logs as follows [root@localhost mydbfarm]# tail -f merovingian.log 2020-01-03 16:03:14 MSG merovingian[13157]: database <dbname> has shut down 2020-01-03 16:03:14 ERR control[13157]: !monetdbd: an internal error has occurred 'database <dbname> did not initialise the sql scenario' 2020-01-03 16:03:18 ERR merovingian[13157]: client error: database <dbname> did not initialise the sql scenario 2020-01-03 16:03:18 MSG merovingian[13157]: database <dbname> (13263) has exited with exit status 0 Please,help us to solve this issue. Thanks Kalai

2 1

Announcement: New Nov2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Dec '19

20 Dec '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019-SP1 bugfix release (11.35.9) Bug Fixes * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6724: Prepare confuses types when more than one argument is used * 6726: Python aggregation does not create aggr_group when aggregating over all rows * 6765: GRANT SELECT privilege on a subset of table columns results in access denied error when selecting the same columns from the table * 6790: Count distinct giving wrong results * 6791: str_to_time('11:40', '%H:%M') creates wrong time value * 6792: JSON path compiler accepts invalid input * 6793: cast(interval second value to int or decimal) is wrong (by a factor of 1000), cast(interval month value to decimal or floating point) fails * 6794: external name fits.listdir not bound (sys.listdir) Fatal error during initialization: * 6796: Incorrect crash time reported by monetdb tool after crash of mserver5 * 6798: json.text off by one error * 6804: DNS resolution of 0.0.0.0 fails on recent Ubuntus

1 0

resetting credentials from mserver5?
by Jonathan Speiser 20 Dec '19

20 Dec '19

Hello, I had tried changing the password on my monetdb server (MonetDB 5 server v11.29.3 "Mar2018") but after rebooting it seems I am locked out from mclient. Is there any way I can change the monetdb user password from mserver5? Thank you for your help Jon

2 1

starting database 'tpch'... FAILED
by DEMBELE Simon Pierre 16 Dec '19

16 Dec '19

-------- Original Message -------- Subject: starting database 'tpch'... FAILED Date: 16/12/2019 12:58 From: DEMBELE Simon Pierre <simon-pierre.dembele(a)ensma.fr> To: mailman-owner(a)monetdb.org Bonjour, Hello I need help; Why can solve this issue starting database 'tpch'... FAILED start: starting 'tpch' failed: database 'tpch' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details Thank -- Simon

2 2

MERGE TABLE - CANNOT DROP PARTITION
by Ariel Abadi 10 Dec '19

10 Dec '19

Hi! I'm still having issue dropping internal tables from the MERGE TABLE. I used to have this problem on a previous version. I have created a new DB, imported all the data... and now I need to delete a partition (NOT a table). I'm still facing the same problem, which is that I cannot remove it from the merge table. (as you can see below).. Is there anything I can do? Rgds Ariel Welcome to mclient, the MonetDB/SQL interactive terminal (Apr2019) Database: MonetDB v11.33.3 (Apr2019), 'mapi:monetdb://dt-mn:54321/cli_dg' Type \q to quit, \? for a list of available commands auto commit mode: on sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful sql>ALTER TABLE sb_traf.cub15_cubo DROP TABLE sb_traf.cub8_20190801; operation successful

2 1

CREATE DATABASE
by Ariel Abadi 09 Dec '19

09 Dec '19

Hi! Im trying to start a new fresh database .. I am creating with the following commands: monetdbd create /opt/starbi/mm/ monetdbd start /opt/starbi/mm/ monetdb create cli_dg monetdb start cli_dg When I try to start the new database... the database is NOT starting. starting database 'cli_dg'... FAILED start: starting 'cli_dg' failed: database 'cli_dg' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details *Here is the log:* 2019-12-09 15:59:07 MSG cli_dg[385]: # loading sql script: 72_fits.sql 2019-12-09 15:59:07 ERR cli_dg[385]: #main thread:!ERROR:ParseException:SQLparser:3F000!CREATE PROCEDURE: external name fits.listdir not bound (sys.listdir) 2019-12-09 15:59:07 ERR cli_dg[385]: ParseException:SQLparser:3F000!CREATE PROCEDURE: external name fits.listdir not bound (sys.listdir)Fatal error during initialization: 2019-12-09 15:59:07 ERR cli_dg[385]: ParseException:SQLparser:3F000!CREATE PROCEDURE: external name fits.listdir not bound (sys.listdir) 2019-12-09 15:59:11 MSG merovingian[370]: database 'cli_dg' (385) has exited with exit status 1 2019-12-09 15:59:11 ERR control[370]: (local): failed to fork mserver: database 'cli_dg' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details Can you please help!?

4 5

COPY INTO ... LOCKED
by Roberto Cornacchia 02 Dec '19

02 Dec '19

Hi, The documentation about the LOCKED modifier for COPY INTO mentions: *When using the LOCKED mode, it is the responsibility of the database user to make sure that there is currently only a single client connection accessing the database* I was assuming (and the syntax suggests, I think) that the LOCKED refers to the table being copied into, not to the entire database. That is, only that table is updated without the overhead of logging. But the documentation imposes the very restrictive condition that only one connection is active on the database. That seems to suggest that the entire database is being updated without logging, not only that table. Could you please clarify? Can I do COPY INTO LOCKED safely with more than one connection, as long as only 1 connection touches that table? Roberto

1 0

Issue in using Monet/Python loader
by Kalaiarasi Gowtham 29 Nov '19

29 Nov '19

Hi, While trying to use Monet/Python Loader function,we are getting the following error *pymonetdb.exceptions.OperationalError: 'pyapi.eval_loader' undefined in: pyapi.eval_loader(0x7f34f01f2c60:ptr, "{_emit.emit( { 'event_date': '2019-10-10', 'status': 1})};":str);* *Followed the below steps for installation* *Python installation steps:* We have installed python 3.6 1)yum install autoconf 2)yum install automake 3)yum install libtool 4)yum install openssl 5)yum install openssl-devel 6)yum install python-devel 7)Configured environment path settings for python export PYTHONPATH=$PATH:/usr/local/lib/python3.6/site-packages *Monet installation steps:* 1)git clone https://github.com/MonetDB/MonetDB.git 2)cd /MonetDB 3)./bootstrap 4)cd.. 5)mkdir testdir 6)cd testdir 7)../MonetDB/configure --enable-pyintegration=yes While configuring we get the following *py3integration is enabled* 8)make 9)make install 10)monetdbd create /path/to/mydbfarm 11)monetdbd start /path/to/mydbfarm 12)monet stop -a 13)monet set embedpy=true 14)monet start -a 13)Enabled numpy(pip install numpy) *Created a loader by using the below python script* import pymonetdb import sys import os connection = pymonetdb.connect(username="xyz", password="xyz123", hostname="ipaddress", database="testdb") cursor = connection.cursor() cursor.execute("CREATE LOADER myloader() LANGUAGE PYTHON {_emit.emit( { 'event_date': '2019-10-10', 'status': 1})};") #create loader cursor.execute("COPY LOADER INTO store FROM myloader();") #append the row from loader to table connection.commit() *While running the above script getting the following error* pymonetdb.exceptions.OperationalError: 'pyapi.eval_loader' undefined in: pyapi.eval_loader(0x7f34f01f2c60:ptr, "{_emit.emit( { 'event_date': '2019-10-10', 'status':1})};":str); Please, help us to fix this issue. Thanks & Regards Kalaiarasi A

2 1

Announcement: New Nov2019 Feature release of MonetDB suite
by Sjoerd Mullender 28 Nov '19

28 Nov '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019 feature release (11.35.3) MonetDB5 Server * The server "console" has been removed, as has the --daemon option. The server now doesn't read from standard input anymore. The way to stop a server is by sending it a TERM signal (on Linux/Unix) or by sending it an interrupt signal (usually control-C -- on all systems). * Implemented a function bat.diffcand to calculate difference of two candidate lists. * The mtime module was completely rewritten, the atom types date, daytime, and timestamp were changed. Upgrade code for BATs containing these types has been implemented. The old daytime type used a 32 bit integer to record milliseconds since the start of the day. The new daytime type uses a 64 bit integer to record microseconds since the start of the day. The old date type recorded days since or before the year 1. The new daytime type records the day of the month and the number of months since the year -4712 separately in a single 32 bit integer of which only 26 bits are used. Dates now use the proleptic Gregorian calendar, meaning the normal Gregorian callendar has been extended backward, and before the year 1, we have the year 0. Both the old and new timestamp types are a combination of a daytime and a date value, but since those types have changed, the timestamp type has also changed. The new date type has a smaller range than the old. The new date range is from the year -4712 to the year 170049. During conversion of date and timestamp columns, the dates are clamped to this range. * The tzone and rule atom types have been removed. They were not used by any code, and they were defined in a non-portable way. * Added "mapi_ipv6" property to monet_options to force ipv6 address binding only. This property is inherited while forking from monetdbd if it is also set there. * Removed (bat)calc.between_symmetric and changed (bat)calc.between by adding a number of extra arguments, all of type :bit: symmetric, low inclusive, high inclusive, nils false. Merovingian * Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively. * Added ipv6 property to monetdbd properties to force IPv6 addresses binding only. By default this property is false to allow IPv4 addresses as well. MonetDB Common * BATcalcbetween and all its variants now have an extra bool parameter "anti" to invert the test. * All forms of BATcalcbetween and VARcalcbetween have extra bool arguments for low inclusive, high inclusive and nils false. The latter causes the result to be false instead of nil if the value being checked is nil. SQL Frontend * Removed functions json.text(string) returns string and json.text(int) returns string. Their MAL implementation didn't exist, so they were meaningless. * There are new aggregate functions sys.median_avg and sys.quantile_avg that return the interpolated value if the median/quantile doesn't fall exactly on a particular row. These functions always return a value of type DOUBLE and only work for numeric types (various width integers and floating point). * Added sys.deltas ("schema" string, "table" string, "column" string) returns table ("values" bigint) system function which returns a single column with 6 values: a flag indicating if the column's upper table is cleared or not, the count of the RDONLY, RD_INS and RD_UPD_ID deltas of the column itself, the number of deleted values of the column's table, as well as the level of the current transaction in the transaction level tree. It should be used for debugging purposes only. * Added "VALUES row_list" statement as a top SQL projection statement. * The implementation of in-expression now follows a join-based approach instead of using iterative union/selects. This greatly improves performance for large in-value-lists. Furthermore the old approach has large in-value-lists generate large MAL plans. This is now no longer the case. * Strings are now limited to 1GB, double-quoted tokens are limited to 2kB. These sizes are bytes of (UTF-8 encoded) input data. Bug Fixes * 3533: SQL aggregate functions avg(), sum() and median() return an error when used on a column with datatype interval second * 6134: Query produces error: HEAPalloc: Insufficient space for HEAP of 1168033427456 bytes. * 6613: LATERAL crash /.../rel_bin.c:1473: rel2bin_table: Assertion `0' failed. * 6683: Bug in subselect * 6686: Bug in subselect (count function) * 6688: Bug in subselect (or condition) * 6689: Trying to improve the performance of SQL queries with a large list of members in IN clause. * 6695: timestamp transformation * 6700: Monetdb Bugs in Subselect statements: * 6722: window functions issues * 6740: while upgrading the database from verison (MonetDB-11.27.13) to (MonetDB-11.33.3) we are unable to bring up the database * 6754: in mclient a strang msg is reported after issueing command: set schema sys; * 6755: Assertion failure in rel_bin.c * 6756: Error in optimizer garbageCollector on merge tables select * 6757: Double free or corruption (out) * 6758: SIGSEGV in __strcmp_sse2_unaligned() * 6759: Python JSON loader creates invalid data type for strings * 6761: Error: Program contains errors.:(NONE).multiplex * 6762: mserver5 crashes on (re-)start * 6764: mserver5 crashes with corruption, double free, invalid size or invalid pointer * 6766: Missing bulk implementation for get_value and next_value calls * 6769: ProfilerStart is not threadsafe * 6771: R-devel * 6772: TRACE includes information from all active sessions * 6773: json.filter returns corrupted string when selecting JSON null value * 6774: PROD aggregation gives wrong result * 6775: NOT IN with an AND containing an OR gives wrong result * 6776: Creating a table with a full outer join query gives type with wrong digits on the joined key. * 6779: Using Windows Messages translation for errno error codes. * 6780: Wrong value of the rank function * 6781: Insert after index creation crash * 6783: AVG changes scale of its results * 6784: function sys.isauuid(string) should return false if string value cannot be converted to a UUID

1 0

UDF bulk implementation with some arguments as scalars
by Roberto Cornacchia 05 Nov '19

05 Nov '19

Hi, When defining a UDF, e.g. CREATE FUNCTION myfunc(a int, b int) RETURNS int we can have: - a scalar implementation command myfunc(a:int, b:int) :int - a bulk implementation command myfunc(a:bat[:int], b:bat[:int]) :bat[:int] - a mixed implementation (e.g. a is a column, b is scalar): command myfunc(a:bat[:int], b:int) :bat[:int] I'm trying to understand what the restrictions are, if any, for the third case. Is it expected to be allowed in (a combination of) the following cases? - Interleaved bulk/scalar arguments command myfunc(a:bat[:int], b:int, c:bat[:int]) :bat[:int] - Table-returning function command myfunc(a:bat[:int], b:int) (:bat[:int], :bat[:int]) - Table-returning function, used with a sub-select in input SELECT * FROM myfunc( (SELECT a, 10 FROM mytable) ); In particular I can't get the last one to work. Even though the sub-select has a constant as the second column, this constant seems to be first replicated into a full column and then given to myfunc() as a column. When the constant parameters are a few, that becomes a waste of time/space. Plus the annoyance of selecting the first value of a constant column for the actual internal implementation, which of course expects a scalar for those arguments. Am I missing something? Is there a way to make that work as I would expect? Thanks, Roberto

4 5

Report a bug
by lhy1149 04 Nov '19

04 Nov '19

Hello, I don't know if it's appropriate, but because I can't register monetdb's bugzilla, I'm going to report a bug here. This problem causes the insert statement to hang and mserver5 takes up 100% of the CPU time. Here is the code that caused the problem: Create table a(a int, b int, id bigserial); -- comment any of the next 2 lines will resolve the problem Create ordered index a_pk on a(id); Create index a_idx1 on a(a); Insert into a(a) values(1); -- this line will hang Enviroment: macOS Catalina 10.15 MonetDB v11.33.11 I hope the development team can check if there is this problem. Thank you

3 2

MERGE TABLE - CANNOT DROP PARTITION
by Ariel Abadi 28 Oct '19

28 Oct '19

Hi! Can someone help me? Im trying to REMOVE a table from a merge table, when I run the command, it states "operation successful", but the table is NOT removed. Here you can see an example... I have run it several times, but the table is still there :( [image: image.png]

4 7

Guide to installing from source on a FC30 machine with SELinux enabled.
by Arjen P. de Vries 10 Oct '19

10 Oct '19

Dear all, For what it's worth, here a report on building MonetDB from source on a Linux FC30 system with SELinux enabled. Thanks to Sjoerd for some crucial hints in getting it to work, and Roberto for the encouragement to not give up :-) First, build as usual: ./bootstrap ./configure --disable-debug --disable-developer --disable-assert --enable-optimize make -j Now, skip the usual `make install` and build rpm packages instead - this includes a single rpm that applies (almost) all the SELinux policies MonetDB needs. (Missing ones should be reported as bugs.) make rpm Install the created rpm packages in two steps, such that all directories that need policies have indeed been created (the SELinux rpm is under the noarch tree): find rpmbuild/RPMS/`uname -m` -name \*.rpm | xargs sudo dnf --disablerepo="*" install --skip-broken -y find rpmbuild/RPMS/noarch -name \*.rpm | xargs sudo dnf --disablerepo="*" install --skip-broken -y Start the server: sudo systemctl status monetdbd You can always uninstall the rpm packages with some more shell magic (replace x86_64 by `uname -m`): find rpmbuild/RPMS -name \*.rpm | sed -e 's/.*\/$MonetDB.\+fedora30$$\.x86.64\|\.noarch$\?\.rpm/\1/g' | xargs sudo dnf --disablerepo="*" -y rm This was sufficient for my, but if you are struggling with SELinux warnings/errors, it may be useful to know the following basics: You can check assigned filecontexts using ls -alRZ /var/monetdb5 These should align with the file contexts specified in monetdb.fc: grep monetdb5 /usr/share/doc/MonetDB-selinux/monetdb.fc Subsequently install those filecontexts and apply them as follows; e.g., for `.merovingian_properties` you would do: sudo semanage fcontext -a -t monetdbd_etc_t /var/monetdb5/.merovingian_properties sudo restorecon -v /var/monetdb5/.merovingian_properties Remaining issues are the following: + `make -j rpm` does not seem to work correctly. + `MonetDB-client-test` package does not want to install with the above command due to the --disablerepo flag. + SELinux complained about access to resource `cpu` - fixed by using the suggested `audit` in the error message. Good luck, hope it saves some people time! Arjen -- ==================================================================== ICIS, office M1.02.07 Radboud University Mercator 1 Faculty of Science Toernooiveld 212 arjen(a)cs.ru.nl NL-6525 EC Nijmegen, The Netherlands +31-(0)24-365 2354 ===================== http://www.informagus.nl/ ====================

1 0

trace not session-isolated?
by Roberto Cornacchia 09 Oct '19

09 Oct '19

On our production servers we keep a pre-defined number of sessions open to a database. Incoming queries are distributed among these sessions. I create an mclient connection to that database and I run a trace (any query). The trace displays all functions created by the queries belonging to the other sessions. All together concatenated (or maybe interleaved, I'm not sure) in the output of my trace. I expect this is not meant, is it?

1 0

Announcement: New Apr2019 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '19

26 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019 feature release (11.33.3) MonetDB5 Server * Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first. * Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8. * Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible. * The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter. Build Environment * We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line. * We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old). SELinux Support * There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories. Merovingian * Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it. Client Package * The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q. Mapi Library * Removed function mapi_cache_shuffle. * Removed function mapi_stream_query. Stream Library * The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.). * Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h. * Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream). * Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped. * Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'. MonetDB Common * During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts. * Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed). * Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending). * Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter). * The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'. * Function PROPdestroy now takes a BAT* parameter. * A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value. SQL Frontend * Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments. * Implemented SQL2003 natural logarithm function: ln(num_expr). * Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument). * Implemented length(blob) and comparison between blobs. * Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base. * Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm). * Removed unused global SQL variable "history". * Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong. * Allow usage of WITH clauses in insert, update, delete and merge statements. * Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] } * Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident * Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5) * Improved and extended storagemodel functionality. Changes include: * Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema. * Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns. * In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns. * System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize(). * System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya. * System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate. * System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead. * Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names. * Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table. * Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns. * Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value). * The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;). * Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);). * Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation. * Implemented X'...' style binary string literals. * Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation. * Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release. * Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending). * Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements. * The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>] * Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified. * The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated. * Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition. Bug Fixes * 2403: stream: http read support in streams * 2416: GDK: file/dir creation mask * 2478: SQL: rename columns * 2496: SQL: implement greatest / least * 3384: Auxiliary window functions * 3416: log() SQL function only supports natural logarithm * 3448: Make lbatomic_ops optional if installed * 3520: Add support for scalar function LN(num) * 3530: sigabort on window aggr function * 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename * 3743: DELETE FROM merge table not supported. * 3832: Cannot use expressions in GROUP BY clause * 3946: Expose C log2 function to SQL * 4056: mapi_timeout is not implemented * 6181: comments cause error with return in function * 6326: Eliminate unused UNION tables * 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB * 6475: Remove unused SQL global variable "history" * 6588: Table aliasing are not supported in SQL update queries * 6591: Name mangling in the SQL catalog results in ambiguity * 6608: Sqlitelogictest error message in group by with coalesce * 6609: Rename Schema in MonetDB * 6636: sys.queue() extension: allow DBA to see all running queries * 6647: Add suport to Python 3 on Windows * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6671: Error when running user function on merge table with remote part * 6674: R UDF with Date type fails to convert RAPI * 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids) * 6678: Binding NULL parameter to parametrized query results in syntax error on execution * 6680: Copy cannot open CSV file if systemctl enabled * 6684: Inserting into a replica table crashes the server * 6685: adding a view to a merge table gives unexpected behaviour * 6690: Unable to fetch empty string with ODBC driver * 6696: Re-use of MAL explain plans with complex type coercions

2 1

stethoscope not redirecting time column correctly
by Jiading Guo 17 Sep '19

17 Sep '19

Hi, When setting stethoscope dumping to disk by `stethoscope -d <db-name> -o <output-file>`, the time column is printed in the console instead of the file. In `MonetDB/clients/mapiclient/stethoscope.c` L113, it directly prints to stdout by `printf("\"%s\", ", ev->time);`. Is it on purpose? Also, I can't follow this message[1] to profile MAL because it seems that the APIs have been changed. Where can I find the newest document? Thanks, Jiading Guo [1]: https://www.monetdb.org/pipermail/users-list/2013-May/006369.html

2 2

!SQLException:sql.bind:42000!Column type mismatch
by Jiading Guo 09 Sep '19

09 Sep '19

Hi, I'm doing some tests about bind operation with v11.33.11 (Apr2019-SP1) and here's how to reproduce: ``` mclient -d test sql>create table unary(a int); sql>insert into unary(a) select value from generate_series(1, 101, 1); mclient -d test -l mal mal>sql.init(); mal>s := sql.mvc(); mal>a := sql.bind(s, "sys", "unary", "a", 0); ``` Then I got `!SQLException:sql.bind:42000!Column type mismatch`. I traced it down to `sql/backends/monet5/sql.c:mvc_bind_wrap`, `int coltype = getBatType(getArgType(mb, pci, 0));` coltype is set to 255 here, which confused me about what this coltype is used for. Also in `sql/backends/monet5/sql.mal`, the comment says the access kind (last argument) is divided into `0, base table`, `1, inserts` and `2, updates`. In what scenarios are these values used? Is `0` for read only operations? What about delete operations? Thanks for your help. Best regards, Jiading Guo

2 2

Latest release (11.33.11) still doesn't honor numactl
by Gotwisner, Dave 04 Sep '19

04 Sep '19

I have just upgraded MonetDB on an 88 core CentOS box - 2 cpu modules. I was also one of the people who originally found the numa issue. The system doesn't behave as I would expect. Here is the sequence of commands I execute to start up MonetDB (overkill, but creating a clean database for the test): 1. numactl --cpunodebind=1 --membind=1 monetdbd create /srv/zfsdata/monetdbTesting/nTC 2. numactl --cpunodebind=1 --membind=1 monetdbd set listenaddr=0.0.0.0 /srv/zfsdata/monetdbTesting/nTC 3. numactl --cpunodebind=1 --membind=1 monetdbd set discovery=false /srv/zfsdata/monetdbTesting/nTC 4. numactl --cpunodebind=1 --membind=1 monetdbd set control=false /srv/zfsdata/monetdbTesting/nTC 5. numactl --cpunodebind=1 --membind=1 monetdbd start /srv/zfsdata/monetdbTesting/nTC 6. numactl --cpunodebind=1 --membind=1 monetdb -p 50000 create nTC 7. numactl --cpunodebind=1 --membind=1 monetdb -p 50000 release nTC If I launch MonetDB using the monetdbd command as "numactl -cpunodebind=1 -membind=1", that means that we are assigning only 44 cores for use with MonetDB (mserver5). I would expect that either command 1 or 5 is the one that spawns mserver5. What I then see with numactl --show --pid=`pidof mserver5` is: policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 cpubind: 0 1 nodebind: 0 1 membind: 0 1 I would have expected that if we launch mserver5 via the monetdbd wrapper (as is what's documented in the tutorials), that the numa settings would be honored. Note also, I would expect 44 threads (plus a few extra) be set rather than 88 (plus a few extra). We are seeing 96 threads. Looking at the useful part of the /proc/`pidof mserver5`/status file gives the following: Cpus_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00fffffc,00000fff,ffc00000 Cpus_allowed_list: 22-43,66-87 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 Which implies it IS putting the program on 44 cores, but still creating 96 threads. It also shows that it isn't honoring the memory constraint? Any suggestion on how to correctly launch so it honors our numa requirements? Thanks, Dave

2 1

Possible problem with C++ based CREATE FUNCTION
by Gotwisner, Dave 04 Sep '19

04 Sep '19

I am trying to create a function that works around concurrency issues that are documented, and that we are seeing, when doing simultaneous "create table" statements and bulk insert statements (or selects) off of other tables. To do this, I am implementing a function that uses a boost::shared_mutex that we call around transactions. We are using a custom version of the boost library, so it is installed in a non-standard place: /usr/local/boost-1.54.0 for development, and /opt/xxx/ for production. >From the documentation, it appears that I can use #pragma LDFLAGS and CFLAGS to point to the non-standard directories. It appears to honor the CFLAGS option, but not the LDFLAGS option. If I use the standard /usr/lib64 libraries, calling my function works (but with the wrong library). Adding the #pragma in, I get the following link error: Failed to open shared library: libboost_thread.so.1.54.0: cannot open shared object file: No such file or directory. If I modify LD_LIBRARY_PATH before starting MonetDB, the problem goes away. Ideally, we don't want to have to add all of our relevant directories into the environment, but would expect the #pragma to work. Here is the function definition: CREATE FUNCTION rwlocker(input INT) RETURNS INT LANGUAGE CPP { #pragma LDFLAGS -L/usr/local/boost-1.54.0/lib -lboost_thread #pragma CFLAGS -I/opt/tc3/include -I/usr/local/boost-1.54.0/include #include <boost/thread/shared_mutex.hpp> // use lock() for schema changes, lock_shared() for inserts and queries static boost::shared_mutex readerWriterMutex; result->initialize(result, 1); int val = input.data[0]; switch(val) { case 1: // shared lock readerWriterMutex.lock_shared(); break; case 2: // shared unlock readerWriterMutex.unlock_shared(); break; case 3: // unique lock readerWriterMutex.lock(); break; case 4: // unique unlock readerWriterMutex.unlock(); break; default: // Error break; } result->data[0] = val; }; Invocation is easiest with: select rwlocker(12); which will just output a 12, bypassing the reader/writer locking. Any thoughts on why #pragma LDFLAGS doesn't appear to work right? We are using 11.33.3 on CentOS 7.3.1611 Thanks, Dave

2 2

Announcement: New Apr2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Sep '19

02 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019-SP1 bugfix release (11.33.11) Build Environment * Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched. MonetDB Common * We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available. SQL Frontend * Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries. * Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR. Bug Fixes * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6697: Duplicate expressions not eliminated with long CASE statement * 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed * 6703: SQL optimizer enters loop and goes into stack overflow * 6706: prepare doesn't recognize merge statement * 6712: Where clause with cast ignores sub-select * 6713: COPY INTO FROM 'file.xz' does not work * 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit) * 6715: Assertion failure in rel_bin.c for MERGE INTO command * 6716: COPY INTO does not load UTF8 encoded text * 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash) * 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation] * 6721: Possibly incorrect call to pcre_exec * 6725: Prepare statement on non-existing function crashes * 6728: SELECT DISTINCT not removing duplicates * 6729: Creating a table with duplicate column name should not be possible * 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always" * 6736: Unexpected query result with merge tables and view after upgrade * 6738: issue with window functions and hugeint type coercion * 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi * 6744: CREATE SEQUENCE seq; fails * 6748: MonetDB "forgets" columns after ALTER TABLE statements * 6749: mserver5 restart aborts/segfaults after dropping column with constraint * 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL

1 0

Crash - !mvc_init: unable to create system tables
by Nobody Noone 30 Aug '19

30 Aug '19

Hi, Would anyone have any advice on recovering. I was running a bunch of COPY SELECT * FROM x INTO '/y.z.csv' USING DELIMITERS ',' , '\n' , '"'; The log said there was already an existing file in the dir. The db was not responding so I stopped it. Then tried to start it and I get; !mvc_init: unable to create system tables #main thread:!ERROR:SQLException:SQLinit:42000!Catalogue initialization failed Fatal error during initialization: SQLException:SQLinit:42000!Catalogue initialization failed # MonetDB 5 server v11.33.3 (Apr2019) database 'y' (53342) has exited with exit status 1 ERR control[33899]: (local): failed to fork mserver: database 'y' has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details Any advice on how to recover? Thanks:)

1 0

Performance issue related to dropping (empty) tables
by Gotwisner, Dave 27 Aug '19

27 Aug '19

I am seeing a performance issue related to schema upgrades. This is from a C++ program that uses the MAPI library. We try and keep 30 days worth of data, imported at a fairly high data rate with lots of columns. We use multiple threads to ingest data (bulk insert), and the number of threads is stored in a configuration table. For 20 days worth of data, this works out to around 9000 tables (plus merge tables). We have roughly 6 sets of tables per hour, each of which has 4 tables and a merge table connecting them. Because this is a schema change, we wrap the entire upgrade in a transaction. For the purpose of the test, we had no data in the tables, but only had empty tables. If we change our setting from 4 parallel tables to 2, we end up dropping 9000 tables. About half of the "drop table" commands take 0 seconds. The rest take anywhere between 4 and 12 seconds each. Here is a plot of the time per delete, y access is seconds, x is the statement. The X access is NOT time based: [cid:image001.png@01D5568E.A5389030] Any idea on how we can reduce the time? The total time to delete 9000 tables was 480 minutes. Thanks, Dave

2 4

memory control on large results
by Daniel Zvinca 15 Aug '19

15 Aug '19

Hello, I plan to use the embedded library (custom build) for a project that will likely address very large tables. While the control over statements is handled to the end user, I need an advice how to control memory usage on fetching huge amount of data. Specifically, I already skipped the default "monetdb_result_fetch" and implemented a compressed storage (which optional can be stored on disk), but I wasn't able to see a way of releasing the associated memory for already processed columns (persistent or transient bat). Unless the bat contents is shared between different res_col's, I would like to be able to release that memory to the system before proceeding to the next column. Any suggestion? Thank you, Dan

1 0

Embedded Python Initialization Issue
by Sharma, Sreejith 14 Aug '19

14 Aug '19

All, I am using an Python based UDF and have enabled embedpy=true for DB with Python version 2.7.5. This works fine with CentOS 7, but the same doesn't wok on Red Hat Ent Linux version. Is there any dependency on this? Is it due to OS version? Am getting below error though Python path is proper in Red Hat OS version server. Embedded Python is enabled but an error was thrown during initialization. Can anyone provide some help on this? Both uses same version of MonetDB server version. MonetDB server on which function works fine [user@lnx1535 ~]$ cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" MonetDB server on Azure where function is not working [user@alcazimdbd01 ~]$ cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.5 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.5" PRETTY_NAME="Red Hat Enterprise Linux" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.5 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.5" Regards, Sreejith

2 1

Type of integers used to calculate a SUM
by Jan, Di-An 13 Aug '19

13 Aug '19

With hugeint enabled, it seems that the SUM of INT (32-bit) is done using HUGEINT (128-bit) while the SUM of BIGINT (64-bit) is done using BIGINT (64-bit). That seems weird. Is SUM supposed to use the largest type or use the same type as the input or something else? Database: MonetDB v11.33.3 (Apr2019) sql>CREATE TABLE test (bi BIGINT, i INT); sql>INSERT INTO test VALUES (0x7000000000000000, 0x70000000); sql>INSERT INTO test VALUES (0x7000000000000000, 0x70000000); sql>INSERT INTO test VALUES (1, 3); sql>INSERT INTO test VALUES (2, 4); sql>SELECT SUM(i) FROM test; -- larger than an INT -- +------------+ | L3 | +============+ | 3758096391 | +------------+ 1 tuple sql>SELECT SUM(bi) FROM test; -- larger than a BIGINT -- overflow in calculation. sql>EXPLAIN SELECT SUM(i) FROM test; +------------------------------------------------------------------------------------------------------+ | function user.s10_22():void; | | X_1:void := querylog.define("explain select sum(i) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:hge := aggr.sum(X_17:bat[:int]); | | exit X_74:bit; | | sql.resultSet("sys.L3":str, "L3":str, "hugeint":str, 128:int, 0:int, 7:int, X_18:hge); | | end user.s10_22; | +------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(bi) FROM test; +-------------------------------------------------------------------------------------------------------+ | function user.s12_22():void; | | X_1:void := querylog.define("explain select sum(bi) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:lng] := sql.bind(X_4:int, "sys":str, "test":str, "bi":str, 0:int); | | X_17:bat[:lng] := algebra.projection(C_5:bat[:oid], X_8:bat[:lng]); | | X_18:lng := aggr.sum(X_17:bat[:lng]); | | exit X_74:bit; | | sql.resultSet("sys.L3":str, "L3":str, "bigint":str, 64:int, 0:int, 7:int, X_18:lng); | | end user.s12_22; | +-------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(i AS HUGEINT)) FROM test; +-----------------------------------------------------------------------------------------------------------------------+ | function user.s20_22():void; | | X_1:void := querylog.define("explain select sum(cast(i as hugeint)) from test;":str, "default_pipe":str, 13:int); | | barrier X_75:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:bat[:hge] := batcalc.hge(X_17:bat[:int]); | | X_19:hge := aggr.sum(X_18:bat[:hge]); | | exit X_75:bit; | | sql.resultSet("sys.L4":str, "L4":str, "hugeint":str, 128:int, 0:int, 7:int, X_19:hge); | | end user.s20_22; | +-----------------------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(bi AS BIGINT)) FROM test; +-----------------------------------------------------------------------------------------------------------------------+ | function user.s8_22():void; | | X_1:void := querylog.define("explain select sum(cast(bi as bigint)) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:lng] := sql.bind(X_4:int, "sys":str, "test":str, "bi":str, 0:int); | | X_17:bat[:lng] := algebra.projection(C_5:bat[:oid], X_8:bat[:lng]); | | X_18:lng := aggr.sum(X_17:bat[:lng]); | | exit X_74:bit; | | sql.resultSet("sys.L4":str, "L4":str, "bigint":str, 64:int, 0:int, 7:int, X_18:lng); | +-----------------------------------------------------------------------------------------------------------------------+ sql>EXPLAIN SELECT SUM(CAST(i AS INT)) FROM test; +-----------------------------------------------------------------------------------------------------------+ | function user.s10_22():void; | | X_1:void := querylog.define("select sum(cast(i as int)) from test;":str, "default_pipe":str, 12:int); | | barrier X_74:bit := language.dataflow(); | | X_4:int := sql.mvc(); | | C_5:bat[:oid] := sql.tid(X_4:int, "sys":str, "test":str); | | X_8:bat[:int] := sql.bind(X_4:int, "sys":str, "test":str, "i":str, 0:int); | | X_17:bat[:int] := algebra.projection(C_5:bat[:oid], X_8:bat[:int]); | | X_18:hge := aggr.sum(X_17:bat[:int]); | | exit X_74:bit; | | sql.resultSet("sys.L4":str, "L4":str, "hugeint":str, 128:int, 0:int, 7:int, X_18:hge); | | end user.s10_22; | +-----------------------------------------------------------------------------------------------------------+ sql>SELECT SUM(CAST(i AS INT)) FROM test; +------------+ | L4 | +============+ | 3758096391 | +------------+ 1 tuple Also, the cast in SUM(CAST(bi AS BIGINT)) seems to be optimized out while SUM(CAST(i AS HUGEINT)) has one more BAT operation than SUM(i) even though they seem to do the same thing, and the cast in SUM(CAST(i AS INT)) seems to have no effect. Di-an Jan

1 0

Books about Monetdb?
by Nobody Noone 09 Aug '19

09 Aug '19

Hi, Are there any books published about Monetdb? This listserv, monetdb.org and stackoverflow.com seem to be all I can find. Thanks!

2 1

understand how monetdb choose the best query plan
by DEMBELE Simon Pierre 08 Aug '19

08 Aug '19

Hello, J'm new in the use of monetdb, i want to understand how monetdb choose the best query plan. Is the model cost is used for that or what. Please if someone can clarify me. Thank yours time. -- Simon

2 2

Questions on upgrading tables based upon the system tables.
by Gotwisner, Dave 08 Aug '19

08 Aug '19

I am writing code to migrate from one version of our schema to another (such as if we add tables or add columns). The from version is whatever is physically being used, and the new version is compiled into data structures in our code. I can successfully create or remove tables, and add and drop columns. We have multiple tables that we then bind to a merge table, so the schemas must be exact. Our tables have a primary key. For the table and column based changes, querying sys.schemas, sys.tables, and sys.columns gives us everything we need. I can also figure out how to add the primary key on the new table (via alter table commands). However, I can't figure out how to find the set of columns in the primary key, to be able to build up the primary key. Here is the table definition for our tables that go into the merge table (%04d goes from 0000 to the number of threads we send bulk data with): CREATE TABLE if not exists data_availability__%04d ( interval_start BIGINT, source INT, PRIMARY KEY(interval_start, source) ); alter table data_availability add table data_availability__%04d; Looking at system tables gives the following information (before adding this table): sql>select * from sys.tables where name = 'data_availability'; +------+-------------------+------+------+------+-------+------+-------+-------+ | id | name | sche | quer | type | syste | comm | acces | tempo | : : : ma_i : y : : m : it_a : s : rary : : : : d : : : : ctio : : : : : : : : : : n : : : +======+===================+======+======+======+=======+======+=======+=======+ | 8889 | data_availability | 8800 | null | 3 | false | 0 | 0 | 0 | +------+-------------------+------+------+------+-------+------+-------+-------+ 1 tuple sql>select * from sys.keys where table_id = 8889; +------+----------+------+----------------------------------------------+------+--------+ | id | table_id | type | name | rkey | action | +======+==========+======+==============================================+======+========+ | 8887 | 8889 | 0 | data_availability_interval_start_source_pkey | -1 | -1 | +------+----------+------+----------------------------------------------+------+--------+ 1 tuple sql>select * from sys.key_types; +-------------+---------------+ | key_type_id | key_type_name | +=============+===============+ | 0 | Primary Key | | 1 | Unique Key | | 2 | Foreign Key | +-------------+---------------+ 3 tuples sql>select * from sys.idxs where table_id = 8889; +------+----------+------+----------------------------------------------+ | id | table_id | type | name | +======+==========+======+==============================================+ | 8888 | 8889 | 0 | data_availability_interval_start_source_pkey | +------+----------+------+----------------------------------------------+ 1 tuple sql>select * from sys.index_types; +---------------+-----------------------+ | index_type_id | index_type_name | +===============+=======================+ | 0 | Hash | | 1 | Join | | 2 | Order preserving hash | | 3 | No-index | | 4 | Imprint | | 5 | Ordered | +---------------+-----------------------+ 6 tuples sql>select * from sys.columns where table_id = 8889; +------+----------------+--------+-------------+------------+----------+---------+-------+--------+---------+ | id | name | type | type_digits | type_scale | table_id | default | null | number | storage | +======+================+========+=============+============+==========+=========+=======+========+=========+ | 8885 | interval_start | bigint | 64 | 0 | 8889 | null | false | 0 | null | | 8886 | source | int | 32 | 0 | 8889 | null | false | 1 | null | +------+----------------+--------+-------------+------------+----------+---------+-------+--------+---------+ 2 tuples The above data is for the merge table that we will bind the (new) table to. I can get the name and id of the primary key (which is also an index). I could not find a table that gives a column -> key binding. Is there such a table? If not, how do I get this information from MonetDB? Thanks, Dave

2 1

Question on querying two specific columns from sys.columns
by Gotwisner, Dave 07 Aug '19

07 Aug '19

I am trying to query the default and null columns from sys.columns. Both of these are sql key words, so queries either don't work or return the wrong value. Doing a "select default from ss.columns;" returns "syntax error, unexpected DEFAULT in: "select default"" error. Doing a "select null from sys.columns;" returns null for all records, even though it is a Boolean column with true/false values. I am currently working around this by doing "select *" and pulling out the relevant fields, but, I want to protect against a future release possibly changing the column ordering or changing the number of columns in this table (without also requiring an update to our code). Is there a way to escape the column names in a way that gives me what I want? Thanks, Dave

2 1

Re: Alter question about MonetDB
by Ying Zhang 06 Aug '19

06 Aug '19

Hai, This question should have been sent to the users mailing list, so I added that e-mail address. I can confirm this problem with an Apr2019 refreshed yesterday. Please see the 1st attachment. Also, after having restarted both mclients, I can still only query “tbl5” using the mclient on the left side. Please see the 2nd attachment. Can you please fire a bug report for this? Thanks! Jennie > On 6 Aug 2019, at 08:55, 角 <3089454366(a)qq.com> wrote: > > Hi,All: > I am a MonetDB user from China. I have some troubles when I was using MonetDB. I use mclient and enter this sql : "alter table t1 rename to t2；", while this mclient is alive,I use anthor mclient to connect the same database as the former and I enter this sql : "select * from t2" then I get an error : no such table. So how can I use alter command to make all mclients take effect? > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list

1 0

Question on missing BAT file
by Sharma, Sreejith 06 Aug '19

06 Aug '19

All, Is there a way to figure out which BAT files constitutes a specific table in MonetDB? I know the location in sys.storage represents it, but I see one of the tail file is showing missing which is not available neither in sys.storage not in DB folder. We get this error during migration from older version (MonetDB 11.27.3) to MonetDB 11.33.3 version. 2019-07-31 10:43:54 MSG TSV_PROD_DB_A[161617]: !ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV/TSV_PROD_DB_A/bat/10/45/104502.tail Regards, Sreejith

2 2

Fwd: Re: Managing Access to Monetdb Schemas
by ferreira 02 Aug '19

02 Aug '19

-------- Forwarded Message -------- Subject: Re: Managing Access to Monetdb Schemas Date: Fri, 2 Aug 2019 11:06:21 +0200 From: ferreira <pedro.ferreira(a)monetdbsolutions.com> To: Nobody Noone <nobody.noone.nobody.me(a)gmail.com> Hello! I found it strange why we don't support GRANT/REVOKE privileges statements on schemas. I looked into the SQL standard, and there is no mention of it. You can use roles to limit schema access for each user. Here you have an example: CREATE ROLE my_role; CREATE SCHEMA my_schema AUTHORIZATION my_role; CREATE USER my_user with password 'p1' name 'User with role' schema "my_schema"; GRANT my_role to my_user; You can add more fine grained access to each table with table privileges as you mentioned. As for specific rights such as DDL, we don't support it as of now. You can create a feature request on Bugzilla, but as it is not paid, it will have low priority. Best regards, Pedro On 7/31/19 4:40 PM, Nobody Noone wrote: > New to Monetdb, From a DBA perspective I’m having trouble > understanding how to manage access rights to schemas and the tables > therein. My department has many senior analysts who should have admin > rights to the tables within the schema (DML - select, insert, update, > delete, trunc, etc; DDL create table, drop table, create seq, etc). > There are also some users that need select only in a given schema. How > do I manage this without having to grant access per table? Is that > where roles come into play? Say super users get privilege_code_id= 79 > (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be > assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am > not seeing those in the roles? I do see the ability when the schema is > created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I > don’t want to do is have to run a query that searches all tables in a > schema, grant admin DDL to them. > Thanks! > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list

1 0

Cast a string column with empty strings to int
by Ying Zhang 01 Aug '19

01 Aug '19

Hai, I have e.g. the following string column containing either numbers or empty strings: CREATE TABLE tst (s string); INSERT INTO tst VALUES ('1234'), (''), (‘4567'); I’d like to cast ’s' to an INT column. The following query works: sql>SELECT CAST(s AS INT) FROM (SELECT CASE WHEN s='' THEN NULL ELSE s END AS s FROM tst) AS tmp; +------+ | L4 | +======+ | 1234 | | null | | 4567 | +------+ 3 tuples But the following query doesn’t: sql>SELECT CASE WHEN s='' THEN NULL ELSE CAST(s AS INT) END FROM tst; conversion of string '' to type int failed. sql>SELECT CASE WHEN s='' THEN CAST(NULL as INT) ELSE CAST(s AS INT) END FROM tst; conversion of string '' to type int failed. This is probably because batcalc.int is applied on the whole “s” column as shown by the query plan: sql>EXPLAIN SELECT CASE WHEN s='' THEN NULL ELSE CAST(s AS INT) END FROM tst; +--------------------------------------------------------------------------------------------------------------------+ | mal | +====================================================================================================================+ … | C_7:bat[:oid] := sql.tid(X_6:int, "sys":str, "tst":str); | | X_10:bat[:str] := sql.bind(X_6:int, "sys":str, "tst":str, "s":str, 0:int); | | X_19:bat[:str] := algebra.projection(C_7:bat[:oid], X_10:bat[:str]); | | X_21:bat[:bit] := batcalc.==(X_19:bat[:str], "":str); | | X_24:bat[:bit] := batcalc.isnil(X_21:bat[:bit]); | | X_30:bat[:bit] := batcalc.ifthenelse(X_24:bat[:bit], false:bit, X_21:bat[:bit]); | | X_35:bat[:int] := batcalc.int(X_19:bat[:str]); | | X_36:bat[:int] := batcalc.ifthenelse(X_30:bat[:bit], nil:int, X_35:bat[:int]); | … Is my query wrong or the plan? Thanks, Jennie

2 2

Question with wildcarded expressions for LIKE clause
by Gotwisner, Dave 01 Aug '19

01 Aug '19

I am trying to do some constraints off the system tables to select a unique pattern of tables. We have tables like the following (I can't give the actual table names): * Config (a single record exists here, no merge tables use it) * Data_0000 (a small number of records exist in each of these) * Data_0001 * Data_0002 * Data_0003 * Data (this is a merge table over the last 4) * Test_table (a small number of records exist here, no merge tables use it) * Main__yyyy_mm_dd__hh__0000 (millions of records, yyyy, mm, dd, hh are all numbers, we have one set of tables per hour, and keep significantly over 1 weeks' worth of data) * Main__yyyy_mm_dd__hh__0001 * Main__yyyy_mm_dd__hh__0002 * Main__yyyy_mm_dd__hh__0003 * Main__yyyy_mm_dd__hh (this is a merge table over the last 4) If I do a select * from sys.tables where schema_id in (select id from schemas where name = 'frodo') and type in (select table_type_id from table_types where table_type_name = 'TABLE') I get the right thing - all non merge tables. I then want to further break it down into the tables that end in __[0-9][0-9][0-9][0-9] and those that don't (two separate queries for two separate purposes). Consider frodo the name of our schema. Adding and name like 'data\_\_____' appears to work. This should give me the 4 data_* tables. Adding and name like '%\_\_____' gives me all non-merge tables, including config and test_table. I'm assuming % is the wild card for 0 or more, and _ is the wild card for 1. This matches some documentation I have found. The \ are required because I need to explicitly look for 2 of the underscore character. Am I doing this correctly? Are these the right wildcarding characters? MS SQLServer gives a # as a match for a single digit, is there an equivalent? Or, is this just a bug in MonetDB? Thanks, Dave

3 5

Managing Access to Monetdb Schemas
by Nobody Noone 31 Jul '19

31 Jul '19

New to Monetdb, From a DBA perspective I’m having trouble understanding how to manage access rights to schemas and the tables therein. My department has many senior analysts who should have admin rights to the tables within the schema (DML - select, insert, update, delete, trunc, etc; DDL create table, drop table, create seq, etc). There are also some users that need select only in a given schema. How do I manage this without having to grant access per table? Is that where roles come into play? Say super users get privilege_code_id= 79 (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am not seeing those in the roles? I do see the ability when the schema is created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I don’t want to do is have to run a query that searches all tables in a schema, grant admin DDL to them. Thanks!

1 0

How to manage access rights
by Nobody Noone 30 Jul '19

30 Jul '19

New to Monetdb, From a DBA perspective I’m having trouble understanding how to manage access rights to schemas and the tables therein. My department has many senior analysts who should have admin rights to the tables within the schema (DML - select, insert, update, delete, trunc, etc; DDL create table, drop table, create seq, etc). There are also some users that need select only in a given schema. How do I manage this without having to grant access per table? Is that where roles come into play? Say super users get privilege_code_id= 79 (SELECT,INSERT,UPDATE,DELETE,TRUNCATE) whereas some users would be assigned privilege_code_id= 1 (SELECT). What about DDL rights? I am not seeing those in the roles? I do see the ability when the schema is created with CREATE SCHEMA "ABC" AUTHORIZATION "xyz_user". What I don’t want to do is have to run a query that searches all tables in a schema, grant admin DDL to them. Thanks!

1 0

Issue while migrating to MonetDB v11.33.3 (Apr 2019)
by Sharma, Sreejith 25 Jul '19

25 Jul '19

All, We are migrating our MonetDB database from MonetDB v11.27.13 (Jul2017-SP4) to MonetDB v11.33.3 (Apr 2019). We tried to bring up the DB instance with MonetDB v11.33.3 (Apr 2019), but getting below error though the same when we tried with previous version MonetDB v11.31.13 (Aug2018-SP2) it worked. Could you help us to identify the issue? 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: #main thread:!ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV_TEST/TSV_PROD_DB_T/bat/10/45/104502.tail 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: #main thread:!OS: No such file or directory 2019-07-24 04:00:21 ERR TSV_PROD_DB_T[88767]: !FATAL: BBPinit: BBPcheckbats failed 2019-07-24 04:00:21 MSG TSV_PROD_DB_T[88767]: !ERROR: BBPcheckbats: cannot stat file /monet_data01/DBFARM_TSV_TEST/TSV_PROD_DB_T/bat/10/45/104502.tail 2019-07-24 04:00:21 MSG TSV_PROD_DB_T[88767]: !OS: No such file or directory 2019-07-24 04:00:23 MSG merovingian[88722]: database 'TSV_PROD_DB_T' (88767) has exited with exit status 1 2019-07-24 04:00:24 ERR control[88722]: (local): failed to fork mserver: database 'TSV_PROD_DB_T' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints 2019-07-24 04:02:18 MSG control[88722]: (local): served status list Regards, Sreejith

2 1

(Без темы)
by Илья Хайбуллин 25 Jul '19

25 Jul '19

How can I write a workable join by null-safe eqauls? I have two tables: create table monet1_test ("Id" bigserial, key int, value varchar(2000)); create table monet2_test ("Id" bigserial, key int, value varchar(2000)); insert into monet1_test select value "Id", value "key", value "value" from generate_series(1,60000); insert into monet2_test select value "Id", value "key", value "value" from generate_series(1,200000); I tried select count(1) from monet2_test t1 inner join monet1_test t2 on ((t1.key = t2.key) or ((t1.key is null) and (t2.key is null))); and got errors GDK reported error: MT_mmap: CreateFileMapping(0000000000001200, &sa, 4, 22, 1508704256, NULL) failed OS: The printer is out of paper. HEAPalloc: Insufficient space for HEAP of 95997921280 bytes. MT_mmap: CreateFileMapping(00000000000011B4, &sa, 4, 22, 1508704256, NULL) failed OS: The printer is out of paper. HEAPalloc: Insufficient space for HEAP of 95997921280 bytes. I create bug-report: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6737

3 7

Indexing support on Json DataType
by Chatchawan Triperm 22 Jul '19

22 Jul '19

Hi All, Does anyone know whether there is built-in indexing on Json datatype or not. I would like to keep my dynamic columns in Json but worry about indexing. Regards, Chatchawan

3 4

Unexpected query result with merge tables and view after upgrade
by Enrico Maria Carmona 22 Jul '19

22 Jul '19

2 2

Questions on embedding R in MonetDB and client pacakge in R
by Andor 18 Jul '19

18 Jul '19

Hello everyone there, I am a beginner user of MonetDB , mainly working with database from R, new to this email list. Is there a seach function somewhere in this list? I can only see the archieve webpage. I have some general user questions, and would greatly appreciate your comments and suggestions. the embedding R in MonetDB.The very latest materials I could find on this topic on MonetDB websites, dated backed in 10/31/2014, I am wondering where I can find more embedding R in MonetDB related information? tutorial, documents etc the client package working with MonetDB from R I only see monetDB.R but not MonetDBLite in the CRAN packages index, yet in MonetDB site , both packages are introduced. any comments on the MonetDBLite, will it be there in the future? Thanks! Andor

1 0

yum upgrade doesn't allow existing db use new window functions
by Andy Allaway 09 Jul '19

09 Jul '19

Hi, I ran: yum upgrade MonetDB-SQL-server5 MonetDB-client When I create a new db I can call: select first_value(col1) over (partition by col2) from analytics; as per the blog monetdb.org/blog/extended_sql_window_functions However a db that existed before the update returns: Function first_value not found. I can see first_value in sys.functions. Any suggestions? Thanks!

2 2

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 04 Jul '19

04 Jul '19

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

4 3

Question on communications path between mserver5 and the MonetDB API (libmapi)
by Gotwisner, Dave 04 Jul '19

04 Jul '19

We are transferring a lot of data as the result of some of our queries (20-30 columns of integers and floats) for about 50,000 records, aggregating these from multiple MonetDB instances (on different machines). For this pattern of query, the largest amount of time spent on the query (from the network onwards, ignoring Mserver5 lookup time) is on the data conversion. When running "perf top" on our application, we notice we are spending a significant amount of time (the second most samples per perf top item at 14%) converting the data returned by libmapi from "const char *" to int64_t, int32_t, double, etc. We know the type of the data, and we have very few string columns in the database. Is there an option to MonetDB that allows it to pass binary data rather than textual data back and forth on the wire? Is there an API that will give the data back in the native type rather than in the on-the-wire string type? Both client and server are always running on 64-bit X86 boxes running Linux (meaning, little endian is guaranteed)? We use serialized protobufs whenever we send data on the wire when communicating between our various programs, and the data stream is significantly smaller than sending integers back and forth in ascii. If not, would switching to ODBC / JDBC give us better performance in this area? Thanks, Dave

2 1

Q: Logical plan
by Hanfeng Chen 04 Jul '19

04 Jul '19

Hello, A quick question. Is the logical plan generated by the keyword "plan" is the plan before it is sent to the SQL optimizer *OR* the output plan of the optimizer? >From the online documentation ( https://www.monetdb.org/Documentation/Manuals/SQLreference/PlanSQL) this sentence "*The relational plan used in the SQL optimizer* can be obtained by prepending the query with the keyword PLAN." is not clear to me. Thanks, Hanfeng Chen

2 1

Using the Monet API to connect to MonetDB if MonetDB comes up AFTER our application
by Gotwisner, Dave 24 Jun '19

24 Jun '19

I am currently trying to update my code that uses the Mapi API to retry connections if we start an application when MonetDB is down. Our programs do not run interactively, so detecting the failure, terminating the program, and restarting the application is problematic. There are a couple of issues here that I can use help with. What we are doing is calling mapi_connect(). Mapi_connect returns what appears to be a pointer (it's non-zero). So we call mapi_error with this return value. If we are not connected, we get an MERROR, not an MTIMEOUT, which is what I would expect. With an MERROR failure, we call mapi_error_str() to see what kind of error. We then must do a string compare (actually, a strstr() call looking for a "Connection refused" pattern inside the string. Is there a way to get an enum or #define value instead, with a different function call? Sort of like looking at errno rather than calling strerror(errno) and doing a string compare on the result. This would be way more efficient if we could do this. Yes, errors should be rare, but determining the type of error (or protecting against a string text change on an RPM update) shouldn't be expensive. We then call mapi_explain() to stderr. Is there a routine that returns the value of the output of explain() to a std::string or a "const char *"? We have a logger that does stuff with the strings before logging (timestamping, __line__ __file__, putting error codes, etc. Having the multiline output of the explain() information would be very useful for us. The above process is executed multiple times until we connect. Since the failed connects return what appears to be a pointer, do we need to do anything to free the pointers up? It doesn't appear that we have a memory leak (I ran valgrind). Note that if the connect fails, calling mapi_destroy() crashes. From the documentation, "mapi_destroy()" does "Free handle resources", so it seems reasonable to call it after a failed (and non-null) mapi_connect(). It should probably do nothing, it should not crash. Also, it appears that if I connect multiple times without disconnecting (I had a bug in my code), I can't connect more than 65 times (simultaneous connections). Is this an API issue or a server issue? The reason I ask, is that if it a client issue, we will be having a single client that potentially does queries against hundreds of servers, and aggregates the result. If it is a server issue, is there a parameter that we can tweak to increase this? We do multiple queries in parallel, each on their own connections. Thanks, Dave

2 2

Guidelines for MonetDB in production environments
by Roberto Cornacchia 20 Jun '19

20 Jun '19

Hi all, I'm struggling with optimizing resource sharing of MonetDB in production environments (see also: https://www.monetdb.org/pipermail/users-list/2018-June/010276.html) We run MonetDB instances for several projects / customers on each of our servers. Each MonetDB instance is a docker container (used mainly because of ease of deployment and environment isolation). It is not unusual to have 5-10 MonetDB containers on the same server. In principle, Docker does not even have much to do with this, but it puts everything in a realistic context. ** Memory mserver5 checks the system memory and calibrates on that. When 10 instances are running, they all assume they have the whole memory for themselves. Docker allows to set limits on the container memory. It does that by using cgroups (so Docker just makes things easier, but it's really about cgroups). However, memory limits set by cgroups are not namespaced ( https://ops.tips/blog/why-top-inside-container-wrong-memory/#memory-limits-… ). This means that each container will still see the whole memory and will simply get killed when the container limit has been reached (definitely not a solution). So far, the only mechanism I know to obtain the correect behavior is to run actual VMs for MonetDB. But this is very cumbersome and I want to avoid that as much as possible. Should we let 10 instances believe they each have the whole memory, and let them fight for it? (well, that's what's happening now, and I know for sure it's bad). Perhaps the solution can be as easy as allowing an explicit max memory setting, together with some documentation on the consequences of using low / high values. ** CPU Again, Docker allows to set quotas per container. I think cgroups CPU limits are namespaced, so perhaps this would just work well, I haven't really tried yet. ** I/O Same issue. It would be ideal to be able to set priorities, so that mserver5 instances that do background work get a lower I/O priority than instances serving online queries. Also, recommendations on swap settings would be interesting. How much swap? How to tune swappiness kernel settings? I am very aware that there is no simple answer to most of these questions. Many variables are in the picture. Still, some general thoughts from the developers would be appreciated. I think I have read pretty much everything has ever been written about MonetDB, but when it comes to resource utilization I have always bumped into the very unrealistic assumption that each MonetDB instance has a whole server for itself. As I mentioned above, things could get already much better with simple improvements, like allowing to set the maximum memory usable by each instance. But more in general, I feel there is much need for some guidelines for production environments. Or at least, to start the discussion. Best regards, Roberto

3 5

Error Upgrading to 11.33.3 (Apr2019)
by Ariel Abadi 19 Jun '19

19 Jun '19

Hi! I have a database that It started on version Dec2016-SP5. I have been upgrading from version to version without any problem. Now, Im trying to do the last upgrade (this means from Aug2018-SP2 to Apr2019. But Im facing an error. This is what I see on the log: 2019-06-06 15:17:55 ERR cli_gm[2188]: #main thread:!ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 MSG cli_gm[2188]: !ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 ERR cli_gm[2188]: !FATAL: BBPinit: BBPcheckbats failed 2019-06-06 15:17:58 MSG merovingian[2180]: database 'cli_gm' (2188) has exited with exit status 1 2019-06-06 15:17:58 ERR control[2180]: (local): failed to fork mserver: database 'cli_gm' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints Is there anything I can do to solve this problem? Thks in advance Ariel PD.If I go back to version Aug2018-SP2 the database keeps working with no problem.

2 2

Questions about using COPY INTO when using GRANTS on tables
by Gotwisner, Dave 14 Jun '19

14 Jun '19

Our application was working quite well when using COPY INTO when inserting data into our tables as the default MonetDB user "monetdb" into the system schema (sys). We have subsequently added users and granted table access to the various users. At this point, INSERT works successfully, but COPY INTO does not. The error message for the COPY INTO statement indicated that we needed to use "ON CLIENT" in our load. That makes no difference. Here are the relevant commands to the system (note the user names, passwords, schema names, and tables names were modified for this example): Program that sets up our basic database does this: START TRANSACTION; CREATE USER admin WITH UNENCRYPTED PASSWORD '***' NAME 'Admin User' SCHEMA sys; CREATE SCHEMA test AUTHORIZATION admin; ALTER USER admin SET SCHEMA test; CREATE USER update WITH UNENCRYPTED PASSWORD '***' NAME 'Update User' SCHEMA test; COMMIT; Then, the program that creates the tables does the following: START TRANSACTION; CREATE MERGE TABLE x (...); GRANT ALL ON x TO "update"; CREATE TABLE x0 (...); GRANT ALL ON x0 TO "update"; CREATE TABLE x1 (...); GRANT ALL ON x1 TO "update"; CREATE TABLE x2 (...); GRANT ALL ON x2 TO "update"; CREATE TABLE x3 (...); GRANT ALL ON x3 TO "update"; COMMIT; And we insert (or try to) (4 threads ingest into their own table, the merge table is used to combine the result set on queries): START TRANSACTION; COPY 100000 OFFSET 2 RECORDS INTO x0 FROM '/dev/shm/MonetDB/foo.csv' (columnlist) ON CLIENT DELIMETERS ',','\n'; This returns an error: !/dev/shm/MonetDB/foo.csv': cannot retrieve files. Because of company policy, I can't provide the schema, but the columns in the csv file and the schema differ, which is why I am using the (...) and (columnlist) in the example. Again, when I omit the CREATE USER, CREATE SCHEMA, and GRANT clauses, everything works correctly. Any ideas on what is needed to make it work? Right now, the ingest application and MonetDB are on the same server. In the future, that might not be true. We are running 11.33.3 on CentOS 7. Thanks, Dave

2 1

FW: Cancel query
by Zingman, Jonathan 13 Jun '19

13 Jun '19

Yes, sys.queue(), sorry. A couple of clarifications. The cookbook shows mclient, but we're using the API and we assume you can do the same thing from the API as well. Also, we use autogenerated queries that can be hundreds if not thousands of characters long. Matching in a case independent way would be very resource intensive. It's fairly clear from the documentation that MonetDB doesn't stop immediately in the middle of whatever it is doing, so that's understood. But cancelling queries has to be part of a robust database system. My model of how this might work is that the handle that’s returned when we issue a query could be used to associate with some data returned by sys.queue(), preferably tag or start time since those should be unique and fairly short. Then we stash then in some structure, and if a cancel request comes in, we do a much simpler search of the sys.queue() table that's guaranteed to work. JONATHAN ZINGMAN | Manager – TrueCall Data | DIRECT: 510.809.0423 | CELL: 510.823.5638 | www.netscout.com -----Original Message----- From: users-list <users-list-bounces+jonathan.zingman=netscout.com(a)monetdb.org> On Behalf Of Martin Kersten Sent: Tuesday, June 11, 2019 1:56 PM To: users-list(a)monetdb.org Subject: Re: Cancel query This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe. On 11/06/2019 22:38, Ying (Jennie) Zhang wrote: > > >> On 11 Jun 2019, at 00:58, Zingman, Jonathan <Jonathan.Zingman(a)netscout.com> wrote: >> >> In the cookbook, you give a good way to find out the status of queries by using the sys.query() call, > > Hai, Jonathan, > > MonetDB doesn’t have a function called sys.query(). Did you mean sys.queue() instead? > >> which returns the list of queries tagged in various ways. Our question is whether there is any way to access, e.g., the tag value. In our application, we need to occasionally cancel queries from our UI. The UI has no direct knowledge of the query. While we might be able to find the query we want to cancel by comparing the query text, our automatically generated queries can be very long, and if there’s some internal reformatting by Monetdb, we might not get an exact match. So is there anything in the handle or elsewhere that we can use to identify the query we want to cancel? > > Correct me if I’m wrong, but I’m not aware of a feature with which you can query the qtag of a query. Some code study will be needed to see if it’s feasible to add such feature, and how. > > You can probably get away with string matching, since MonetDB doesn’t do much/any reformatting next to to-lowercase. Not sure how much text the “query” field of sys.queue() holds. > > In addition, please be aware that > i) sys.queue() is originally develop for debugging purpose… > ii) the minimal cancellation entity is a function, i.e., when MonetDB is busy executing a very long running function, a cancellation will only take effect *after* that function has finished => then MonetDB won’t continue executing the next function. Small correction here, with function we mean a MAL function call. So, if the system is executing a very time consuming join, it will *not* be interrupted, but rather at the end of it the query in which is rans is terminated. > > Regards, > Jennie > >> >> JONATHAN ZINGMAN >> Manager – TrueCall Data >> >> <image001.png> >> 2855 Telegraph Ave, Suite 200 >> Berkeley, CA, 94705-1169 >> >> DIRECT: 510.809.0423 >> CELL: 510.823.5638 >> http://www.netscout.com >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma… > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma… > _______________________________________________ users-list mailing list users-list(a)monetdb.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma…

2 1

Cancel query
by Zingman, Jonathan 11 Jun '19

11 Jun '19

In the cookbook, you give a good way to find out the status of queries by using the sys.query() call, which returns the list of queries tagged in various ways. Our question is whether there is any way to access, e.g., the tag value. In our application, we need to occasionally cancel queries from our UI. The UI has no direct knowledge of the query. While we might be able to find the query we want to cancel by comparing the query text, our automatically generated queries can be very long, and if there's some internal reformatting by Monetdb, we might not get an exact match. So is there anything in the handle or elsewhere that we can use to identify the query we want to cancel? JONATHAN ZINGMAN Manager - TrueCall Data [logos] 2855 Telegraph Ave, Suite 200 Berkeley, CA, 94705-1169 DIRECT: 510.809.0423 CELL: 510.823.5638 www.netscout.com<http://www.netscout.com/>

3 2

MonetDB - Case Sensitive
by Suresh Devarakonda 11 Jun '19

11 Jun '19

Hi, I am new to the MonetDB and interested to know about its case sensitivity. We noticed that objects are being created in lower case if they are not specified in quotes so it appears to me that default case for objects created in Monet is LOWERCASE. But, we got a requirement that objects should be created in upper case even though they are not specified in quotes in DDL statements. In other words, can we make the default to UPPERCASE for all the objects when they created. Please let us know if that is possible. Your early response is very much appreciated. ~Suresh D

2 1

Installation error on AWS Linux
by Gerald Sangudi 04 Jun '19

04 Jun '19

Hello, I'm trying to install MonetDB on AWS Linux. The first command succeeds: sudo yum install https://dev.monetdb.org/downloads/epel/MonetDB-release-epel.noarch.rpm The second command fails: sudo yum install MonetDB-SQL-server5 MonetDB-client Error: *[ec2-user@ip-172-31-23-149 ~]$ sudo yum install MonetDB-SQL-server5 MonetDB-clientLoaded plugins: extras_suggestions, langpacks, priorities, update-motdhttps://dev.monetdb.org/downloads/ <https://dev.monetdb.org/downloads/>epel/2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not FoundTrying other mirror. One of the configured repositories failed (MonetDB 2 - x86_64), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this:* It seems to me there's a reference to epel/2 which is out of date. Should that be epel/7? Is it possible to get an updated rpm? Thanks, Gerald

2 2

REPLICA, CLUSTER
by Ariel Abadi 29 May '19

29 May '19

Hi! Does anybody implemented a REPLICA DB, or a CLUSTER with several nodes for monetdb? I couldnt find too much information in how to implement it. Any information you can provide, will help a lot! Thks in advance Ariel

1 0

MonetDB performance on large table with 20K columns
by Malware Research 16 May '19

16 May '19

Hi, I posted a question on StackOverflow <https://stackoverflow.com/questions/55951563/monetdb-performance-on-a-large…> regarding s specific performance issue. I’m not sure if the right people are monitoring S/O from time to time so I thought I’ll post it here too. Here it is in case you do not want to follow the link: I'm testing MonetDB as a solution for a data-science project. I have a table of 21K columns - all but three are features described as float (32bit) and 6.5M rows (which may or may not become larger, perhaps up to 20M rows). My aim is to use the integrated Python on MonetDB to achieve the ability to train without exporting the data from the DB every time. In addition, queries on specific columns are necessary so the columnar storage can be a significant advantage. I have compiled MonetDB 11.31.13 to gain the embedded Python support. OS is CentOS 7. Storage is not SSD. 48 core server with ~300GB of memory. I created an (unique) index on the table (without analyze). I noticed that when I SELECT * FROM [TABLE_NAME] SAMPLE 50; it takes a long long time to complete. I then tried: SELECT f1, f2, ..., f501 from [TABLE_NAME] SAMPLE 50; SELECT f1, f2, ..., f1001 from [TABLE_NAME] SAMPLE 50; SELECT f1, f2, ..., f2001 from [TABLE_NAME] SAMPLE 50; ... SELECT * from [TABLE_NAME] SAMPLE 50; I ran the queries locally with mclient and used time to measure the amount of time it took and I noticed two things: 1. There is a period where a single core is taking 100% CPU. The more columns the longer it takes to complete. Only when it finishes I can see all cores working, data being consumed, etc... In addition, during that time, the query does not appear in the result of select * from sys.queue(); Eventually, the time needed to get 50 rows from the table was almost 4 hours. 2. The amount of columns is doubled but between each step in the test the amount of time it takes to get a result is tripled. So my questions is: Is this behaviour expected or does it reflect something I did wrong? The data requested from the table should be around 4MB (50 * 21000 * 4Bytes), so this reflects a significant time waiting for such a small amount of data. Help is appreciated! Help is very much appreciated!

2 1

MonetDB mercurial source doesn't compile on macOS Mojave
by Ying Zhang 16 May '19

16 May '19

Dear all, When trying to compile the MonetDB mercurial source on macOS Mojave, it ends with the error: /Users/jennie/monet/Apr2019/common/stream/stream.c:819:10: error: implicit declaration of function 'fdatasync' is invalid in C99 [-Werror,-Wimplicit-function-declaration] && fdatasync(fileno(fp)) < 0 ^ 1 error generated. Does anyone know a fix or workaround for this? Thanks a lot! Jennie

1 1

BPP version problem
by Corbit, Dann 02 May '19

02 May '19

Mdblaunch.bat seems to work: C:\Program Files\MonetDB\MonetDB5>mserver5 --dbpath=dbfarm\voc\ --set mapi_port=50000 # MonetDB 5 server v11.33.3 (Apr2019) # Serving database 'voc', using 12 threads # Compiled for x86_64-pc-winnt/64bit # Found 31.927 GiB available main-memory. # Copyright (c) 1993 - July 2008 CWI. # Copyright (c) August 2008 - 2019 MonetDB B.V., all rights reserved # Visit https://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded > But neither of these batch files work. C:\Program Files\MonetDB\MonetDB5>MSQLserver.bat MonetDB/Python Disabled: Python 3.7 installation not found. !FATAL: BBPinit: incompatible BBP version: expected 061041, got 061033. This database was probably created by a too old version of MonetDB. Press any key to continue . . . C:\Program Files\MonetDB\MonetDB5>M5server.bat MonetDB/Python Disabled: Python 3.7 installation not found. !FATAL: BBPinit: incompatible BBP version: expected 061041, got 061033. This database was probably created by a too old version of MonetDB. Press any key to continue . . . They complain about the server version. I tried deleting the database folder and restarting. The database was rebuilt. But I still get the errors. How do I fix this error? I do not have real database systems. I only have the voc sample data.

3 3

Trying to link IBM Cognos and MonetDB via ODBC
by SERET Paul 25 Apr '19

25 Apr '19

Dears, I get the following error while trying to link IBM Cognos Analytics software to MonetDB via ODBC (sorry, the error message is in french, I translate below) : [cid:image001.png@01D4FB81.C8C4CA40] QE-DEF-0285 The logon failed QE-DEF-0325 The logon failed for the following reason: RQP-DEF-0068 Unable to connect to at least one database during a multi-database attach to 1 database(s) in: 'testDataSourceConnection' UDA-SQL-0031 Unable to access the 'testDataSourceConnection' database. Check that the connection parameters to the database are configured correctly. For example, ensure that the data source connection contains the signon information, such as a password, to connect to the database. UDA-SQL-0107 A general exception has occurred during the operation 'attach(error code -49)'. The specified driver has not been loaded because of system error 127 : The specified procedure cannot be found. (MonetDB ODBC Driver, C:\Program Files (x86)\MonetDB\MonetDB ODBC Driver\lib\libMonetODBC.dll). The last section of the message sounds strange to me, because the DLL seems to be correctly installed from the most recent ODBC driver installer (2019/01/04) : [cid:image002.png@01D4FB84.B9159040] It looks like there is an issue inside the DLL program. Is there a known issue with this DLL or the driver ? IBM Cognos mentions that the ODBC driver must be compliant with the 3.5 standard. I guess this is the case for MonetDB ODBC driver, isn't it ? Could you please have any clue to get this sorted out ? Thanks in advance, best regards Paul SERET

1 0

How can I get a parameterized bulk version of a scalar C UDF to work in SQL?
by aris 25 Apr '19

25 Apr '19

Hi, Say that I have a family of simple scalar transforming function f(x ,p) => y parameterized by p which are implemented as C UDF's and exposed as SQL UDF's. Is it possible to create a bulk version of this? So the scalar C signature is simplified to something like char* f(int* output, int* input, const char** func_parameter); the mal stuff module m; command f(input:int, func_parameter:str):int address f; and in sql we have CREATE FUNCTION f(input INT, func_param STRING) RETURNS INT EXTERNAL NAME m.f; Now for the bulk version I would like to use in C: char* f(bat* output, bat* input, const char** func_parameter); in MAL: module batm; command f(input:bat[:int], func_parameter:str):bat[:int] address f; in SQL: reuse the scalar SQL signature. But that doesn't seem to work. When I compile this and call the udf in a sql query on an actual column, the plan applies the scalar version on each tuple instead of calling the bulk version once. However if I remove the parametrization, i.e. removing the func_parameter from the signature, the bulk version is correctly called. Aris

1 0

Database crash issue
by Amit Ambekar 12 Apr '19

12 Apr '19

Hi, I am using MonetDB from last 2 years, but never faced such issue till now. I am not able to use database ,it is giving error as below: monetdbd: internal error while starting mserver database has crashed after starting, manual intervention needed, check monetdbd's logfile (merovingian.log) for details' Regards, Amit Ambekar Data Engineer

2 2

Get stuck when trying to copy large amounts of data
by Yinjie Lin 02 Apr '19

02 Apr '19

Hi, I'm trying to copy large amounts of data into MonetDB, but the processes would get stuck in the middle. The following is what I did. I have 6 tables, all merge tables partitioned by range of time. And there are hundreds of data files, each has almost 10 million rows and is of size about 3.5 GB. For each table, I use the following command to copy data: `cat file.txt | xargs -n1 -P1 -I {} sh -c "mclient -p 50000 -s \"copy 10000000 records into tbx from stdin best effort\" - < '{}'" ` where file.txt contains paths of files to be loaded. So 6 mclient connections are created to copy data in parallel. But mclient processes would get stuck after loading more than 200 million rows each table. Then I have to kill the processes and restart the database. The environment is CentOS 7.3, 512 GB RAM. MonetDB is compiled using the latest master branch. So does anyone know why this happened? Does MonetDB have a limit on the size of data one instance can hold? Or if there is something wrong with my practice? Thanks in advance! Yinjie Lin

2 1

Performance issue with count distinct
by AUBERT Simon 29 Mar '19

29 Mar '19

Hello all, Here a simple issue I have with performance : 1/ I create a table A and populate it with a bulk load 2/I create a table B, I populate the table with bulk load 3/I create table C with a CTAS statement. I analyze the statistics and set the primary key 4/I try a "count distinct" type query (on all the table) on the PK Field. It is pretty slow, with a high sollicitation of my hard drive for several dozens of second. the "select count" is however fast. So I have these following questions : 1/why despite statistics the count distinct query is slow? I thought the idea of statistics was to get some metrics about the table? 2/if the field I make a count distinct on is the PK (or the set of field is the PK), it's exactly the same than a count non null. So why the time are not the same? 3/if these issues are design issue or bugs, how can I ask to make it solved? For information : Monetdb 11.31.13 on Windows 10 Best regards, Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

2 1

Program contains errors.:(NONE).multiplex
by Niklas B 29 Mar '19

29 Mar '19

Hi, I'm upgrading from an old MonetDB (11.26.0 source snapshot) to the latest proper release. However, my Python aggregate functions appear to have stopped working (Program contains errors.:(NONE).multiplex): Running mserver5 with: /usr/local/bin/mserver5 --daemon=yes --set embedded_py=true --dbpath=/data/monetdb-farm/ --set gdk_nr_threads=1 --set max_clients=6 Reproduce with: CREATE FUNCTION test_udf("conum" int) RETURNS STRING LANGUAGE PYTHON3 {{ }}; Create table test("id" uuid, "date" DATE, "number" INT); select test_udf("number") from test; Gives: Program contains errors.:(NONE).multiplex I'm running it via Docker (stock Debian) with only MonetDB installed via apt-get. Not sure if this is connected: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378 <https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378> Anyone having any similar issues? Tried compiling from source (April 2019 bransch) as well, no luck Regards, Niklas

3 2

sys.storage sorted attribute changes for primary key
by Jeremy Norris 29 Mar '19

29 Mar '19

Hello, I have a large table (~700 million rows, ~150 columns). Initially after import, the database is very fast on primary key lookups, however after a few thousand creates/updates/deletes on the database, "sorted" (as reported by sys.storage), flips from true to false. This results in major performance degradation. What are the conditions under which a database can lose it's sorted-ness on the primary key column? eg: sql>select schema, table, column, sorted from sys.storage where schema = 'test1' and table = 'cmdb' and column = 'pk_id'; +---------+-------+--------+--------+ | schema | table | column | sorted | +=========+=======+========+========+ | test1 | cmdb | pk_id | false | +---------+-------+--------+--------+ Thanks, -Jeremy Norris

2 1

Fwd: Question about partition bounds of updatable merge tables
by Yinjie Lin 06 Mar '19

06 Mar '19

Hi, I found that there is a new feature called updatable merge table in the new release (Updatable merge tables with partition predicates <https://www.monetdb.org/blog/updatable-merge-tables>). I think this makes merge table similar to *partitioned table* in other databases. But I found that the upper bound of range partitioning is an inclusive bound, while the upper bound of some databases like MySQL and PostgreSQL is an exclusive bound. So my question is: why is the upper bound declaration designed like this? Will it be possible if MonetDB be consistent with other databases? Besides, currently merge table doesn't support partition pruning, so all child tables will be scanned even if the filter condition contains the partitioning key. Is it possible if this feature will be added one day? Best Regards, Yinjie Lin

1 0

#!ERROR: GDKmove: cannot rename /mnt/g/monet/wal/sql/log to /mnt/g/monet/wal/sql/log.bak
by Michael Jansen 21 Feb '19

21 Feb '19

Hello, I have a crash after about 5 minutes when running in transaction replication mode. After the crash I cannot access the db. I am running: mserver5 --trace --dbpath=/data/monetdb/dbfarm/test_db/ --set gdk_keep_persisted_log_files=1000 --set gdk_logdir=/mnt/g/monet/wal --set sql_debug=4 I am using GlusterFS under /mnt/g to manage the replication. I am running on AWS Linux 2. and I have free space on in both the data and wal directories. (I assume this is enough space): Filesystem Size Used Avail Use% Mounted on 10.0.1.68:/gv0 8.0G 7.4G 669M 92% /mnt/g bash-4.2$ the db only has one test table: sql>create table t (n float, t text) ; operation successful sql>select * from t ; +--------------------------+------+ | n | t | +==========================+======+ | 1.1 | blah | | 1.1 | blah | | 1 | blah | +--------------------------+------+ 3 tuples any help is appreciated! -mj

1 0

Error when installing on Centos 7
by Trevor Andreas 25 Jan '19

25 Jan '19

Hello, I am getting an error trying to install MonetDB following the instructions here: https://dev.monetdb.org/downloads/epel/ The error I am getting is when running: yum install -y MonetDB-SQL-server5 MonetDB-client failure: repodata/repomd.xml from monetdb: [Errno 256] No more mirrors to try. https://dev.monetdb.org/downloads/epel/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found Looks like the repodata is missing for Centos 7? It looks to be there for Centos 6 https://dev.monetdb.org/downloads/epel/6/x86_64/repodata/ Thanks! Trevor

2 1

Invalid Credentials error
by AUBERT Simon 17 Jan '19

17 Jan '19

Good morning, I have used Monetdb for a few weeks now and this evening I cannot login after a full system restart (windows+monetdb). Environment : Windows 10 (well, it's my work computer, I don't have a lot of choice) Monetdb 5 server v11.31.11 I start monetdb by using C:\Program Files\MonetDB\MonetDB5\bin\mserver5.exe Then mclient on C:\Program Files\MonetDB\MonetDB5\mclient.bat (and usually it work just fine) I also tried with mclient.exe using the -x command to debug. Error message : C:\Program Files\MonetDB\MonetDB5\bin>mclient -X -d demo user(win32):monetdb password: MAPI = monetdb@localhost:50000 ACTION= mapi_reconnect ERROR = !InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' Of course, I use the default monetdb/monetdb credentials. I have no idea of what happens and moreover I have lost an entire schema (I see that if I connect through Tableau Desktop...). I have done an upgrade on the last version (11.31.13) and same result, impossible to connect. And same result. I spend the entire evening on that a Thank in advance. Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

2 3

table-returning function with subselect in input - no mitosis?
by Roberto Cornacchia 14 Jan '19

14 Jan '19

Hi, * Background: I have a custom string tokenizer that I invoke this way: SELECT * FROM tokenize( (SELECT id, s FROM my_strings) ); The output has of course a 1-to-N rows relation with the input. The function has a bulk implementation, so it takes M rows and it outputs M rows. * The question: No matter how large the input table my_strings is, mitosis does not seem to kick in. Is it not allowed in this case because it cannot assume that my custom function allows to concatenate partial results? Do you see any chance for me to enable mitosis with this scenario? Thanks, Roberto

1 1

Announcement: New Aug2018-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Jan '19

14 Jan '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP2 bugfix release (11.31.13) SQL Frontend * Disabled function sys.getcontent(url). Bug Fixes * 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered * 6645: optimizer treats all the functions with no or constant parameters as constant * 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s * 6651: Multi-column IN clause for subquery produces wrong results * 6653: CREATE TABLE accepts empty table/column name * 6654: Incorrect handling of 'TRUE' in compound select * 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message * 6660: GRANTing a ROLE is not idempotent * 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt. * 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489 * 6665: Creation of serial types does not accept negative numbers * 6666: COPY INTO from .. LOCKED doubles input data * 6668: The SAMPLE key word doesn't work in a subquery. * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value

1 0

Re: Performance degradation
by Jeremy Norris 12 Jan '19

12 Jan '19

Here's some additional observations on this puzzling issue: The slow table is "cmdb", the normal table is "cmdb2". Both have the same number of rows according to "SELECT count(*)" (73553310). However, sys.statistics shows something a bit different: sql>SELECT x.name, count(*) AS columns_with, z.count FROM sys.tables AS x, sys.columns AS y, sys.statistics AS z WHERE x.id = y.table_id AND y.id = z.column_id GROUP BY x.name, z.count; +-------+--------------+----------+ | name | columns_with | count | +=======+==============+==========+ | cmdb | 8 | 73553310 | | cmdb | 145 | 73553311 | | cmdb2 | 153 | 73553310 | +-------+--------------+----------+ 3 tuples sql> For the slow table, "cmdb", there are 145/153 columns that have an extra "row". This row count mismatch causes it to skip parallelization optimizations in "monetdb5/optimizer/opt_mitosis.c", line 212. A few related questions: 1) In the columnar model, is is valid for conceptual tuples to exist which don't have values for each column like this? (This seems strange as now there's a difference between an explicit NULL and "not existing"). How might this situation have happened? (Only transactional full-row inserts were applied to this DB). Can this be repaired? 2) Is the conditional in "monetdb5/optimizer/opt_mitosis.c", line 212 sufficiently robust? I may be misunderstanding the full context of this line, but it seems like we could still benefit from splitting/parallelization when r < rowcnt, no? Thanks, -Jeremy Norris

1 0

Performance degradation
by Herman Knief 10 Jan '19

10 Jan '19

Greetings, Running MonetDB v11.31.11 (Aug2018-SP1), we ran into an issue where queries that have been running quite well, suddenly slowed by about 5x. Looking at the explain, it seems we hit a condition that is preventing mitosis? Copying the tables related to the query seems to restore performance to the previous level, but would be very curious to understand what happened here. What metrics or stats should we be looking at? Changes we see in the plan are as follows: Bad example: sql.tid(X_4:int, "waltest":str, "cmdb":str); Good example: sql.tid(X_4:int, "waltest":str, "cmdb2":str, 0:int, 32:int); Looking at the storage info, the sizes (heap and column), type width and counts look the same between the two tables. Any thoughts or suggestions would be appreciated. - Herman -- Please excuse typos... I can't type.

1 0

Embedded Python module is not available in recent builds of Windows version
by Kevin Chen 21 Dec '18

21 Dec '18

Hi, I upgraded the MonetDB windows version to 2018Aug and found that the embedded Python module is no longer working. Even the startup script with “embedded_py=true”, the module was not loaded. There was no message showing “MonetDB/Python2 module loaded”. All my Python functions are failed. Please help. Thanks. Kevin

2 1

!FATAL: SQL type wrd missing
by Ariel Abadi 18 Dec '18

18 Dec '18

Hi! Please I need an urgent solution (if there is one) I have upgraded a database to the latest version, and when Im trying to start the DB, there is an error: 2018-12-18 10:58:06 ERR DB[29818]: !FATAL: SQL type wrd missing Can someone help me with any clue!???? Thks in advance Ariel

2 3

Re: grafana-monetdb-datasource installation problem
by Svetlin 03 Dec '18

03 Dec '18

Hi Ariel, The current version of the grafana-monetdb-datasource plugin cannot be used with the latest grafana release. There is a missing breaking dependency that has being introduced(see #10053 <https://github.com/grafana/grafana/issues/10053>) and we didn’t have time to patch that in. Secondly, you would need a http server as a mediator to forward the queries to monetdb. The grafana monetdb datasource plugin merely creates MonetDB compatible queries and forwards responses back to grafana core for visualization. I hope this helps. Cheers, Svetlin > On Dec 3, 2018, at 11:03 AM, Ying (Jennie) Zhang <ying.zhang(a)monetdbsolutions.com> wrote: > >> aabadi(a)starconnecting.com <mailto:aabadi@starconnecting.com>

2 1

grafana-monetdb-datasource installation problem
by Ariel Abadi 01 Dec '18

01 Dec '18

Does anybody has installed the monetdb datasource plugin for Grafana? Im running Grafana v5.3.4. The plugin was downloaded from https://github.com/MonetDBSolutions/grafana-monetdb-datasource And is not working... :'( Does anybody knows how to connect MonetDB within Grafana? Any ideas will be very much appreciated!!! Thks Ariel

2 1

Re: Issues trying to create a table...
by Stefan Manegold 28 Nov '18

28 Nov '18

Hi, Internally (as in in the MonetDB code base),the impact of the problem could imho be significantly reduced (though not fully avoided)by "simply" using different separator characters (or even strings?) when creating qnames,e.g. schema:table;column insteadof schema_table_column ... A unique bijective reversible mapping from schema,table,column to qname should avoid the problem entirely , but might require to use separator chracter(s) that are not valid in SQL identifiers, if such exist ... Best,Stefan -------- Original message --------From: Ying Zhang <Y.Zhang(a)cwi.nl> Date: 28/11/2018 07:33 (GMT+01:00) To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: Re: Issues trying to create a table... > On 28 Nov 2018, at 04:18, Herman Knief <hknief(a)gmail.com> wrote: > > Hi Folks, > I'm new to MonetDB and trying to setup a schema from a MariaDB export. > I have a particular table that reports a conflict with the create statement. I've stripped this down to just a particular field that seems to be causing the issue, but unsure where to go from here. > > sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script" string); > CREATE TABLE: zboot_sc_cat_item_delivery_plan_script conflicts Hai Herman, This is because something has already been defined before, whose complete qname is “zboot_sc_cat_item_delivery_plan_script”. This often happens with (but probably not limited to) schema_name+table_name+column_name. In MonetDB test suite, we have two examples to trigger this (type of) error: sql>create table foo (id integer, bar_id integer); operation successful sql>create table foo_bar (id integer); CREATE TABLE: sys_foo_bar_id conflicts and sql>create table foo_bar (baz int); operation successful sql>create table foo (bar_baz int); CREATE TABLE: sys_foo_bar_baz conflicts This is not a bug. You need to change your table and/or column definitions, like your alternatives below. Then CREATE TABLE will succeed. Regards, Jennie > > What's interesting is that either changing the name of the table or the name of the field and these statements work. There is something about this particular combo that is problematic. > sql>CREATE TABLE "sc_cat_item_delivery_plan1" ("script" string); > operation successful > sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script1" string); > operation successful > > I have a total of 2251 tables... all were created except for this one. I've tried to destroy and recreate the schema with no luck. Logs don't seem to indicate any issue and trying to use the debug on the client didn't get me anywhere. Thoughts? Suggestions? > > -- > Please excuse typos... I can't type. > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

2 1

Issues trying to create a table...
by Herman Knief 28 Nov '18

28 Nov '18

Hi Folks, I'm new to MonetDB and trying to setup a schema from a MariaDB export. I have a particular table that reports a conflict with the create statement. I've stripped this down to just a particular field that seems to be causing the issue, but unsure where to go from here. sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script" string); CREATE TABLE: zboot_sc_cat_item_delivery_plan_script conflicts What's interesting is that either changing the name of the table or the name of the field and these statements work. There is something about this particular combo that is problematic. sql>CREATE TABLE "sc_cat_item_delivery_plan1" ("script" string); operation successful sql>CREATE TABLE "sc_cat_item_delivery_plan" ("script1" string); operation successful I have a total of 2251 tables... all were created except for this one. I've tried to destroy and recreate the schema with no luck. Logs don't seem to indicate any issue and trying to use the debug on the client didn't get me anywhere. Thoughts? Suggestions? -- Please excuse typos... I can't type.

3 2

Issue while connecting to MonetDB via Perl cpan driver
by Sharma, Sreejith 15 Nov '18

15 Nov '18

Hi, We have a Perl code via which we are trying to connect to one of the MonetDB instance. Here we are using DBD::monetdb Perl cpan driver for connectivity. when we run the snippet, it just tries to connect to DB and seems hung, looks like it's awaiting for user input for password. Also, on the server side logs we see an error (challenge error) can connection will not be established. Here is a sample snippet which isn't working. Could anyone help with an idea or code snippet which works with Perl DBI for MonetDB connectivity? [cid:image001.png@01D47D35.C23AD040] #!/usr/bin/perl # use strict; use warnings; use DBI(); print "Attempting to connect...\n"; my $dbh = DBI->connect('dbi:monetdb:host=XXXX;port=55555', 'USERNAME', 'Password', { RaiseError => 1 } ); print "Connected. Trying SQL...\n"; my $sth = $dbh->prepare('SELECT * FROM env'); $sth->execute; $sth->dump_results; print "Done\n"; exit 0; Regards, Sreejith

1 0

Error: cannot bind delete column
by Doherty, Adam 15 Nov '18

15 Nov '18

Hello all: I have run across a show-stopping error, "cannot bind delete column" which renders Monet unusable until I restart the service. A batch import of events runs every 10 minutes and has been unable to complete for the last 28 hours because the service is not accepting commands. I'm not sure where to look for the source of this error. Restarting the service correct the issue, but this is a problem as the Monet instance drives live Tableau dashboards using this imported data. Where do I need to start looking to track down the source of this error? I am currently running 11.29.3 on CentOS 7. Adam

2 1

Adding columns to tables via binary bulk loading
by Anderson, David B 12 Nov '18

12 Nov '18

All, Is it possible to add a column to an existing table via binary bulk loading? We have a table with ~1 billion rows and we would like to add columns to the table, but would prefer bulk loads for speed. Also, does anyone have a snippet of code to create the binary file format that bulk loading expects. I've tried a couple versions (simple int and double), but the server wouldn't recognize the files. Thanks, Dave

2 1

Unable to get sys.queue for all the users
by Murthy, Gautham 07 Nov '18

07 Nov '18

Hello, We are unable to get the information for all active users sessions when we use sys.queue as monetdb user. Also, the tried as user other than monetdb, it returned no access, We had provided the user select privilege on the sys.queue view. [monetadmin@lnx1487 scripts]$ /monet_binaries/MonetDB-11.27.13_PY/bin/mclient mapi:monetdb://lnx1487:50053/DB_BWS_S Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4) Database: MonetDB v11.27.13 (Jul2017-SP4), 'mapi:monetdb://lnx1487:50053/DB_BWS_S' Type \q to quit, \? for a list of available commands auto commit mode: on sql>select "user","active",count(*) as "No. Of Connections" from sys.sessions() group by "user","active"; +-------------------+--------+--------------------+ | user | active | No. Of Connections | +===================+========+====================+ | DM_POS_BWS_STG_P6 | true | 1 | | DM_POS_BWS_STG_P6 | false | 3 | | monetdb | true | 1 | +-------------------+--------+--------------------+ 3 tuples (6.027ms) sql>select * from sys.queue; +------+---------+----------------------------+----------------------------+----------+---------+--------+--------------------------+ | qtag | user | started | estimate | progress | status | tag | query | +======+=========+============================+============================+==========+=========+========+==========================+ | 3630 | monetdb | 2018-11-07 17:07:34.000000 | null | null | running | 3630@0 | select * from sys.queue; | +------+---------+----------------------------+----------------------------+----------+---------+--------+--------------------------+ 1 tuple (1.610ms) sql>\q [monetadmin@lnx1487 scripts]$ Thank You, Gautham

4 3

Cannot setup monetdb on Centos 7
by GAUQUELIN, Philippe 05 Nov '18

05 Nov '18

Hi Need help on setup. Thanks for your answer Trying to install I got this message. The GPG keys listed for the "MonetDB 7 - x86_64" repository are already installed but they are not correct for this package. More details below: []# yum reinstall https://dev.monetdb.org/downloads/epel/MonetDB-release-epel.noarch.rpm Loaded plugins: fastestmirror, langpacks MonetDB-release-epel.noarch.rpm | 6.4 kB 00:00:00 Examining /var/tmp/yum-root-x4i3eb/MonetDB-release-epel.noarch.rpm: MonetDB-release-epel-1.3-1.monetdb.noarch Resolving Dependencies --> Running transaction check ---> Package MonetDB-release-epel.noarch 0:1.3-1.monetdb will be reinstalled --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Reinstalling: MonetDB-release-epel noarch 1.3-1.monetdb /MonetDB-release-epel.noarch 4.0 k Transaction Summary ======================================================================================================================================================================= Reinstall 1 Package Total size: 4.0 k Installed size: 4.0 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : MonetDB-release-epel-1.3-1.monetdb.noarch 1/1 Verifying : MonetDB-release-epel-1.3-1.monetdb.noarch 1/1 Installed: MonetDB-release-epel.noarch 0:1.3-1.monetdb Complete! [root@franalytic02 tmp]# []# rpm --import https://www.monetdb.org/downloads/MonetDB-GPG-KEY []# yum install MonetDB-SQL-server5 MonetDB-client Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: centos.crazyfrogs.org * epel: mirrors.ircam.fr * extras: centos.crazyfrogs.org * updates: centos.mirror.fr.planethoster.net Resolving Dependencies --> Running transaction check ---> Package MonetDB-SQL-server5.x86_64 0:11.31.11-20181011.el7 will be installed --> Processing Dependency: MonetDB5-server(x86-64) = 11.31.11-20181011.el7 for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libstream.so.11()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libmonetdb5.so.25()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 --> Processing Dependency: libbat.so.17()(64bit) for package: MonetDB-SQL-server5-11.31.11-20181011.el7.x86_64 ---> Package MonetDB-client.x86_64 0:11.31.11-20181011.el7 will be installed --> Running transaction check ---> Package MonetDB.x86_64 0:11.31.11-20181011.el7 will be installed ---> Package MonetDB-stream.x86_64 0:11.31.11-20181011.el7 will be installed ---> Package MonetDB5-server.x86_64 0:11.31.11-20181011.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Installing: MonetDB-SQL-server5 x86_64 11.31.11-20181011.el7 monetdb 854 k MonetDB-client x86_64 11.31.11-20181011.el7 monetdb 275 k Installing for dependencies: MonetDB x86_64 11.31.11-20181011.el7 monetdb 1.0 M MonetDB-stream x86_64 11.31.11-20181011.el7 monetdb 197 k MonetDB5-server x86_64 11.31.11-20181011.el7 monetdb 853 k Transaction Summary ======================================================================================================================================================================= Install 2 Packages (+3 Dependent packages) Total size: 3.1 M Installed size: 9.1 M Is this ok [y/d/N]: y Downloading packages: warning: /var/cache/yum/x86_64/7/monetdb/packages/MonetDB-stream-11.31.11-20181011.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID df0e54f3: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Importing GPG key 0x0583366F: Userid : "MonetDB Database System Packager <monet(a)cwi.nl>" Fingerprint: 213e 64dc d5dd c2c0 63cc 39fd 053c 3ed4 0583 366f Package : MonetDB-release-epel-1.3-1.monetdb.noarch (installed) >From : /etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Is this ok [y/N]: y Importing GPG key 0x0583366F: Userid : "MonetDB Database System Packager <monet(a)cwi.nl>" Fingerprint: 213e 64dc d5dd c2c0 63cc 39fd 053c 3ed4 0583 366f Package : MonetDB-release-epel-1.3-1.monetdb.noarch (installed) >From : /etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Is this ok [y/N]: y Public key for MonetDB-stream-11.31.11-20181011.el7.x86_64.rpm is not installed Failing package is: MonetDB-stream-11.31.11-20181011.el7.x86_64 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MonetDB Philippe

2 1

Using capi.eval from MAL
by Renato Marroquín Mogrovejo 04 Nov '18

04 Nov '18

Hi all, I am trying to use the following patten from MAL[1] pattern eval(fptr:ptr,cpp:bit,expr:str,arg:any...):any... However, I am not sure how I should create the first parameter <fptr:ptr> from MAL. I am just trying to be able to call a C-UDF from MAL. From capi.c, it would seem that it is the client's context but I don't know how to get a handle of that. Any pointers are highly appreciated. Thanks! Renato M. [1] https://github.com/MonetDB/MonetDB/blob/2310e49a349ebcb8ff5b5c585928c8f4bd1…

2 4

Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
by Murthy, Gautham 04 Nov '18

04 Nov '18

Hello, We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data. sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$ [monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$ Thank You, Gautham

3 3

Append an array of values to a BUN
by Anton Kravchenko 02 Nov '18

02 Nov '18

Hi there, Is there a way to append an array of values to a BUN, instead one value at a time? sql_table *tbl = mvc_bind_table(m, sch, "atable"); sql_column *col = mvc_bind_column(m, tbl, "acolumn"); BAT *tmp1 = COLnew(0, TYPE_lng, (BUN)nrows, TRANSIENT); long long v_long[3] = {100000000009,100000000010,100000000011}; for (irow=0; irow<nrows; irow++) BUNappend(tmp1, &v_long[irow], FALSE); mvc_append_column(m->session->tr, col1, tmp1); mvc_commit(m, 0, tr->name); bat_destroy(tmp1); Thanks, Anton

2 5

Circular Memory Only Database
by Doherty, Adam 26 Oct '18

26 Oct '18

Hello all: I am wondering if anyone has a remote table strategy such that "hot data" is stored in a table that is backed by RAM. I have a table with approximately 1 billion events that grows by approximately 1,000 events a minute. Table is a merge table of merge tables, the parent merge table has a child merge table for every year of data under those child tables are source tables that make up data for areas. So Area1_2017, Area2_2017 -> Events_2017 Area1_2018, Area2_2018 -> Events_2018 Events_2017, Events_2018 -> Events I am finding that as I add more data the system seems to slow down. I am wanting to keep a cyclical table of say the last 6 months from the Events table in a table backed solely by RAM So a query to Recent_Events would remain very fast or at the very least constant. This should be achievable with tmpfs. I am currently running 11.29.3 on CentOS 7 on VMware 6.7 Server has 128GB of RAM and 16 cores. Adam

2 1

SQLSTATE Error not showing up in mclient Aug2018 version (Aug2018 (11.31.7))
by Sharma, Sreejith 22 Oct '18

22 Oct '18

All, Do we have a way to compile mclient along with server source code? We are not able to see SQLSTATE error code on concurrency with Aug2018 (11.31.7) server version along with mclient Aug2018. Would like to see if we need to specifically compile client too? ERROR = !COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead CODE: 40000 Regards, Sreejith

2 3

Re: users-list Digest, Vol 74, Issue 3
by Oxana Dolmatova 17 Oct '18

17 Oct '18

Hi Stefan, Thank you for the information, now I see the point. >Did you by any chance try any other DBMSs with your "non-standard" workload? >If so, how do those experiences and limitations compare to MonetDB? I did not run tests over conventional DBMSs because they have stricter limits compared to MonetDB (e.g. Postgres is limited to 1600 columns), but only over array DBMSs or math packages. Obviously, their column scalability is better, but they can not handle RA that efficiently. Best regards, Oksana On Tue, 16 Oct 2018 at 12:16, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Column limit (Oxana Dolmatova) > 2. Re: Column limit (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 15 Oct 2018 17:14:14 +0200 > From: Oxana Dolmatova <oxana.dolmatova(a)gmail.com> > To: E.Rozenberg(a)cwi.nl > Cc: users-list(a)monetdb.org > Subject: Re: Column limit > Message-ID: > <CAJJ2wAQi+4S+= > e9BfbStuJxD7izNh8z-mcBF8FYKxagjj5oRkA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Eyal, > > Sorry for the late reply. Yes, there is no problem with large number of > columns in sparse representation of arrays (i.e., dimension, dimension, > value). > However, in order to perform relational operations, this structure should > be pivoted first. When matrices are represented by tables, both algebras > can be applied to the same object, > so I'm specifically looking for the main bottleneck of processing hundreds > of thousands of columns in MonetDB. > > Best regards, > Oksana > > > On Wed, 10 Oct 2018 at 13:40, Eyal Rozenberg <E.Rozenberg(a)cwi.nl> wrote: > > > If you want to model an NxM matrix of elements of type MYTYPE in a DBMS > > (not just MonetDB), you would typically have a table with 3 columns: > > > > CREATE TABLE my_matrix ( > > "row" INTEGER NOT NULL, > > "col" INTEGER NOT NULL, > > "cell_value" MYTYPE NOT NULL, > > PRIMARY KEY ("row", "col") > > ); > > > > of length N x M. > > > > Anyways, I'm not the expert on Arrays in our group so I'll let others > > chime in. > > > > Eyal > > > > > > On 10/10/2018 13:01, Oxana Dolmatova wrote: > > > Hello, > > > > > > Thank you for all the replies and information! > > > The version is 11.23, plus I'm using the cloud platform > > with 4cpu-16ram-hpc. > > > The number of rows was just 1K for 15K attributes in a table. > > > I understand that MonetDB was made with no intention to create and > > > process tables with that many columns. But I'm interested in matrix > > > operations integration, so I would like to know what is the main > > > bottleneck to process such data (matrix-like sized) in the system. > > > > > > Best regards, > > > Oksana > > > > > > > > > On Tue, 9 Oct 2018 at 21:02, Eyal Rozenberg <eyalroz(a)technion.ac.il > > > <mailto:eyalroz@technion.ac.il>> wrote: > > > > > > I'll add two observations to what Zhang Ying said. > > > > > > 1. Internally in MonetDB, the index of a column (or a BAT) in > > MonetDB's > > > BBP (BAT Buffer Pool) is a signed 32-bit integer (on most > > platforms), so > > > there isn't a theoretical problem having more than 15,000 columns; > > but > > > MonetDB was not designed with that many tables in mind. The "long > > > dimension" is expected to be the length of columns. > > > > > > 2. When you have as many as 15,000 columns, you're not simply > finding > > > more attributes of the same items. So, for example, a person has a > > first > > > name, last name, year of birth, etc. etc. - but after several > dozens > > of > > > these you're probably adding columns such as "data relevant to this > > > person at point X", "data relevant to this person at point Y" and > so > > on. > > > > > > In your case, I'm guessing you have a lot of points and hence a lot > > of > > > columns with data for your different items. What you should > consider > > > doing instead is having an extra "coordinate" column. Thus, in the > > > example I gave, I would drop most of the columns, in favor of a > > second > > > table named "person_data". This table's columns would be > "person_id" > > (or > > > some other key columns), "sample_point" and "value". Thus instead > of > > M > > > columns and N records you would have N x M records with 3 columns. > > > > > > Eyal > > > > > > > > > > > > On 09/10/2018 16:50, Ying Zhang wrote: > > > > Hai Oksana, > > > > > > > > A.f.a.I.k., there is no explicit limit on the #columns you can > put > > > in a relation (did you mean ?table??). The reason that mserver5 > > > stopped responding is probably because it is busy with I/O. > > > > > > > > A long time ago, I once tried to create 1M tables, each with 3 > > > columns. After about a day or something, MonetDB has created ~400K > > > tables, and I could hear my HDD being constantly busy. MonetDB was > > > still working, but that was already a sufficient evidence that that > > > idea wasn?t going to work. > > > > > > > > So the question is actually what do you want to achieve? > > > > > > > > Regards, > > > > Jennie > > > > > > > > > > > >> On 9 Oct 2018, at 13:43, Oxana Dolmatova > > > <oxana.dolmatova(a)gmail.com <mailto:oxana.dolmatova@gmail.com>> > > wrote: > > > >> > > > >> Hello, > > > >> > > > >> Could you tell the limitation for number of columns in a > relation > > > and the main reason for that limitation. So far I could create > > > relations with up to 15000 columns. When the number of columns is > > > bigger mserver5 stops responding. > > > >> > > > >> Thank you and best regards, > > > >> Oksana Dolmatova > > > >> _______________________________________________ > > > >> users-list mailing list > > > >> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > >> https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > _______________________________________________ > > > > users-list mailing list > > > > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Oksana Dolmatova > > > > > > _______________________________________________ > > > users-list mailing list > > > users-list(a)monetdb.org > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > -- > Best regards, > Oksana Dolmatova >

1 0

Announcement: New Aug2018-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 17 Oct '18

17 Oct '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP1 bugfix release (11.31.11) Build Environment * On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object. Whole System * Some subtle dependencies between RPMs have been fixed. Bug Fixes * 6640: timestamp_to_str returning incorrectly adjusted results * 6641: race condition in SQL UDF with update * 6642: Hanging query * 6648: key property potentially wrong after type conversion * 6649: Projection inside within transaction gives wrong results

1 0

Column limit
by Oxana Dolmatova 15 Oct '18

15 Oct '18

Hello, Could you tell the limitation for number of columns in a relation and the main reason for that limitation. So far I could create relations with up to 15000 columns. When the number of columns is bigger mserver5 stops responding. Thank you and best regards, Oksana Dolmatova

6 7

monetdb discover not finding other db's
by Frik Lubbe 24 Sep '18

24 Sep '18

Hi all, I relatively new to MonetDB, but have many year exposure to database enignes. I have 2 x vm's, almost identical with the exception of IP's and host names. both are Ubuntu Server 18.04 LTS. I have installed MonetDB on both VM's and created a db farm and a database on each. When running 'monetdb discover' on any of the VM's it finds nothing. Any help is welcome please. Regards, Frik Outputs from 'monetdbd get all /data/monetdb/dbfarm/' *VM 1:* root@ubuntu-srv-001:~# monetdbd start /data/monetdb/dbfarm/ another monetdbd is already running root@ubuntu-srv-001:~# monetdbd get all /data/monetdb/dbfarm/ property value hostname ubuntu-srv-001 dbfarm /data/monetdb/dbfarm/ status monetdbd[1941] 1.7 (Aug2018) is serving this dbfarm mserver /usr/bin/mserver5 logfile /data/monetdb/dbfarm//merovingian.log pidfile /data/monetdb/dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 50000 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 root@ubuntu-srv-001:~# monetdb discover root@ubuntu-srv-001:~# *VM 2:* root@ubuntu-srv-002:~# monetdbd get all /data/monetdb/dbfarm/ property value hostname ubuntu-srv-002 dbfarm /data/monetdb/dbfarm/ status monetdbd[2373] 1.7 (Aug2018) is serving this dbfarm mserver /usr/bin/mserver5 logfile /data/monetdb/dbfarm//merovingian.log pidfile /data/monetdb/dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 60001 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.60001 controlsock /tmp/.s.merovingian.60001 root@ubuntu-srv-002:~#

1 0

SQLcopyinto.java
by Roberto Cornacchia 20 Sep '18

20 Sep '18

I'm trying to debug n issue I have with data disappearing after a database restart. Because part of the process for creating the table involves a mixed Mapi / JDBC approach as described in monetdb-java/example/SQLcopyinto.java, I started from this example. When I run it though, I get a final count of 0 (should be 100). Can anyone confirm? Mar2018, JDBC 2.28 Roberto

2 1

Join Order Benchmark on MonetDB
by Junxiong Wang 19 Sep '18

19 Sep '18

Hello, I am currently evaluating the performance of MonetDB on the Join Order Benchmark, proposed by Leis et al.(http://www.vldb.org/pvldb/vol9/p204-leis.pdf <http://www.vldb.org/pvldb/vol9/p204-leis.pdf>). There are a few queries where performance seems suboptmal, e.g., query 16b and 17e (https://github.com/gregrahn/join-order-benchmark <https://github.com/gregrahn/join-order-benchmark>): MonetDB takes around 400 seconds to evaluate query 17e with a single thread, more than 20 seconds with 48 threads, Postgres takes only around 9 seconds. I suspect that the query optimizer selects the wrong plan, this is the output of the plan command. | project ( | | | group by ( | | | | join ( | | | | | join ( | | | | | | join ( | | | | | | | join ( | | | | | | | | join ( | | | | | | | | | join ( | | | | | | | | | | table(sys.movie_companies) [ "movie_companies"."movie_id" NOT NULL as "mc"."movie_id", "movie_companies"."company_id" NOT NULL as "mc"."company_id" ] COUNT , | | | | | | | | | | select ( | | | | | | | | | | | table(sys.company_name) [ "company_name"."id" NOT NULL HASHCOL as "cn"."id", "company_name"."country_code" as "cn"."country_code" ] COUNT | | | | | | | | | | ) [ "cn"."country_code" = varchar(255) "[us]" ] | | | | | | | | | ) [ "mc"."company_id" NOT NULL = "cn"."id" NOT NULL HASHCOL ], | | | | | | | | | table(sys.title) [ "title"."id" NOT NULL HASHCOL as "t"."id" ] COUNT | | | | | | | | ) [ "t"."id" NOT NULL HASHCOL = "mc"."movie_id" NOT NULL ], | | | | | | | | table(sys.cast_info) [ "cast_info"."person_id" NOT NULL as "ci"."person_id", "cast_info"."movie_id" NOT NULL as "ci"."movie_id" ] COUNT | | | | | | | ) [ "ci"."movie_id" NOT NULL = "t"."id" NOT NULL HASHCOL , "ci"."movie_id" NOT NULL = "mc"."movie_id" NOT NULL ], | | | | | | | table(sys.name) [ "name"."id" NOT NULL HASHCOL as "n"."id", "name"."name" NOT NULL as "n"."name" ] COUNT | | | | | | ) [ "n"."id" NOT NULL HASHCOL = "ci"."person_id" NOT NULL ], | | | | | | table(sys.movie_keyword) [ "movie_keyword"."movie_id" NOT NULL as "mk"."movie_id", "movie_keyword"."keyword_id" NOT NULL as "mk"."keyword_id" ] COUNT | | | | | ) [ "t"."id" NOT NULL HASHCOL = "mk"."movie_id" NOT NULL, "ci"."movie_id" NOT NULL = "mk"."movie_id" NOT NULL, "mc"."movie_id" NOT NULL = "mk"."movie_id" NOT NULL ], | | | | | select ( | | | | | | table(sys.keyword) [ "keyword"."id" NOT NULL HASHCOL as "k"."id", "keyword"."keyword" NOT NULL as "k"."keyword" ] COUNT | | | | | ) [ "k"."keyword" NOT NULL = clob "character-name-in-title" ] | | | | ) [ "mk"."keyword_id" NOT NULL = "k"."id" NOT NULL HASHCOL ] | | | ) [ ] [ sys.min no nil ("n"."name" NOT NULL) NOT NULL as "L3"."L3" ] | | ) [ "L3"."L3" NOT NULL as "L4"."member_in_charnamed_movie" ] | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I was wondering whether anyone might have a hint about what the problem could be and how one could improve performance for those specific queries? Thanks a lot!

1 0

Measure the memory footprint of Monetdb from an SQL client
by Mahmoud Mohsen 12 Sep '18

12 Sep '18

Hello, Is there away to measure the physical or virtual memory consumption of Monetdb using an SQL query. Something similar to select * from storage() where table_name = name; I believe storage() helps with the disk consumption not main memory consumption, am I right? Thank you a lot in advance, Mahmoud Mohsen

2 1

COPY INTO FILE - Folder permission
by Ariel Abadi 10 Sep '18

10 Sep '18

Hi! We are running some "exports" using the COPY INTO FILE, the problem we are facing is that the file created by monetdb is Read Write ONLY by the user running the database. -rw------- 1 root monet 9477 Sep 7 15:08 exp.csv Is there any way so the created file has also RW permission to the group and NOT only the user. In this case, we are using an NFS share so we are not able to use setfacl. Does anybody has an idea how to solve this ? Thanks a lot in advance! Rgds Ariel

5 8

About Monetdb Query Plan
by Junxiong Wang 07 Sep '18

07 Sep '18

Hello, I am currently working on a project related to query optimization I have several questions, 1) Is it possible to force the query engine to use the specified join order? 2) When I change the table order in the “FROM” clause, the query time varies a lot (many orders of magnitude)? Does the table order in the “FROM” clause affect the query plan? 3) I want to know the details of MonetDB query plan. When I use “explain” command, I just get the MAL code. Are there any tools to visualize it, just like PostgreSQL “analyze" command? Thanks a lot.

2 1

About Monetdb Query Plan
by Junxiong Wang 06 Sep '18

06 Sep '18

Hello, I am currently working on a project related to query optimization I have several questions, 1) Is it possible to force the query engine to use the specified join order? 2) When I change the table order in the “FROM” clause, the query time varies a lot (many orders of magnitude)? Does the table order in the “FROM” clause affect the query plan? 3) I want to know the details of MonetDB query plan? But when I use “explain” command, I just get the MAL code. Are there any tools to visualize it, just like PostgreSQL “analyze" command? Thanks a lot.

1 0

How to enable SQLSTATE error code in mclient
by Murthy, Gautham 06 Sep '18

06 Sep '18

Hello, How do we enable SQLSTATE error code on mclient ? Thank You, Gautham

1 0

Announcement: New Aug2018 Feature release of MonetDB suite
by Sjoerd Mullender 31 Aug '18

31 Aug '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018 feature release (11.31.7) MonetDB5 Server * The lsst module was moved to a separate repository (https://dev.monetdb.org/hg/MonetDB-lsst/) Build Environment * Build the MonetDB-cfitsio RPM and libmonetdb5-server-cfitsio Debian/Ubuntu package. * On Windows, the separate MonetDB5-Geom installer has been incorporated into the main MonetDB5-SQL installer and is therefore no longer available as a separate download. Merovingian * Added a "logrotate" configuration file. See /etc/logrotate.d/monetdbd. * Changed the monetdb profilerstart command to be more robust. If the server or stethoscope crashed before, the pid file is still there, so the next time we try to start stethoscope, it will fail. Now the profilerstart command will check if a stethoscope process with the recorded pid is running. If not, we start stethoscope, assuming that something went wrong before. * Changed the monetdb stop command to try to stop stethoscope before stoping the server. The error conditions that can arrise from attempting to stop stethoscope are: + The database is not running. + The profilerlogpath is not set. + The profiler.pid file does not exist or is inaccessible. + The contents of the profiler.pid are not valid. + Shutdown of stethoscope did not succeed. + Removing the profiler.pid file failed. In all the cases, the attempt to stop the server can continue normally, so we actually ignore any errors that rise from the attempt to stop stethoscope. Client Package * ODBC: Implemented SQL_ATTR_QUERY_TIMEOUT parameter in SQLSetStmtAttr. * ODBC SQLGetInfo now returns a positive numeric value for InfoTypes: SQL_MAX_COLUMN_NAME_LEN, SQL_MAX_DRIVER_CONNECTIONS, SQL_MAX_IDENTIFIER_LEN, SQL_MAX_PROCEDURE_NAME_LEN, SQL_MAX_SCHEMA_NAME_LEN, SQL_MAX_TABLE_NAME_LEN and SQL_MAX_USER_NAME_LEN. * Added a '-f rowcount' option in mclient to repress printing the actual data of a resultset, but only print the number of returned tuples Stream Library * Added support for lz4 compressed files in the stream library MonetDB Common * Hash indexes are now persistent across server restarts. * The macros bunfastapp and tfastins and variants no longer set the dirty flag of the heap they write to. This now needs to be done separately (and preferably outside of the inner loop). * Removed batDirty flag from BAT record. Its function is completely superseded by batDirtydesc and the dirty flags on the various heaps. * Removed "masksize" argument of function BAThash. * A whole bunch of functions that took an int argument that was used as a Boolean (true/false) value now take a value of type bool. The functions BATkeyed, BATordered and BATordered_rev now return a bool instead of an int. * Removed the tdense property: it's function is completely replaced by whether or not tseqbase is equal to oid_nil. Testing Environment * Removed helper programs Mtimeout and MkillUsers: they have long been superseded by timeout handling by Mtest.py itself. SQL Frontend * Removed deprecated table producing system functions: sys.dependencies_columns_on_functions() sys.dependencies_columns_on_indexes() sys.dependencies_columns_on_keys() sys.dependencies_columns_on_triggers() sys.dependencies_columns_on_views() sys.dependencies_functions_on_functions() sys.dependencies_functions_on_triggers() sys.dependencies_keys_on_foreignkeys() sys.dependencies_owners_on_schemas() sys.dependencies_schemas_on_users() sys.dependencies_tables_on_foreignkeys() sys.dependencies_tables_on_functions() sys.dependencies_tables_on_indexes() sys.dependencies_tables_on_triggers() sys.dependencies_tables_on_views() sys.dependencies_views_on_functions() sys.dependencies_views_on_triggers() They are replaced by new system dependency_* views: sys.dependency_args_on_types sys.dependency_columns_on_functions sys.dependency_columns_on_indexes sys.dependency_columns_on_keys sys.dependency_columns_on_procedures sys.dependency_columns_on_triggers sys.dependency_columns_on_types sys.dependency_columns_on_views sys.dependency_functions_on_functions sys.dependency_functions_on_procedures sys.dependency_functions_on_triggers sys.dependency_functions_on_types sys.dependency_functions_on_views sys.dependency_keys_on_foreignkeys sys.dependency_owners_on_schemas sys.dependency_schemas_on_users sys.dependency_tables_on_foreignkeys sys.dependency_tables_on_functions sys.dependency_tables_on_indexes sys.dependency_tables_on_procedures sys.dependency_tables_on_triggers sys.dependency_tables_on_views sys.dependency_views_on_functions sys.dependency_views_on_procedures sys.dependency_views_on_views * Implemented group_concat(X,Y) aggregate function which also concatenates a column of strings X, but using a supplied string Y as the separator. This function is also a SQL extension. * Implemented group_concat(X) aggregate function which concatenates a column of strings using a comma as a separator. This function is not featured in the SQL standard. Bug Fixes * 4020: Importing timestamp with zone from copy into * 6506: Improper performance counters * 6556: Sqlitelogictest division by zero on COALESCE call * 6564: Changes to the Remote Table definition * 6575: Sqlitelogictest crash on groupby query with coalesce call * 6579: Sqlitelogic test infinite loop while compiling SQL query * 6586: Sqlitelogictest crash on complex aggregation query * 6593: Poor performance with like operator and escape clause * 6596: Multicolumn aggregation very slow after ANALYZE when persistent hashes are enabled * 6605: Sqlitelogictest set queries with wrong results * 6606: Misleading parameter name in generate_series function * 6610: Sqlitelogictest algebra.rangejoin undefined * 6611: Cannot compile with GCC 8.1 and --enable-debug=no * 6612: Implement BLOB handling in python UDFs * 6618: dependency column on sequence violated by DROP SEQUENCE * 6621: SELECT FROM REMOTE TABLE WHERE <> returns wrong results * 6624: "Cannot use non GROUP BY column in query results without an aggregate function" when using aggregate function in both HAVING and ORDER BY clauses. * 6625: OR in subselect causes the server to crash with segmentation fault * 6627: stddev_pop inconsistent behaviour * 6628: User cannot insert into own local temporary table * 6629: CREATE TABLE IF NOT EXISTS returns 42000! * 6630: Sqlitelogictest cast NULL to integer failing * 6632: Dataflow causes crash when THRnew fails * 6633: ILIKE clauses don't work on certain characters * 6635: monetdbd exits due to "Too many open files" error * 6637: Within a transaction, \d after an error causes mclient to exit * 6638: (sequences of) mkey.bulk_rotate_xor_hash() can generate NIL from non-NIL making multi-col joins return wrong results * 6639: COMMENT ON TABLE abc IS NULL invalidly sets the remark column to null where remark column is defined as NOT NULLable

1 0

Re: #!ERROR: THRnew: too many threads
by Stefan Manegold 14 Aug '18

14 Aug '18

Hi, The usual questions:Which version of MonetDBAndWhat kind of workload triggers this? Best,Stefan -------- Original message --------From: "Murthy, Gautham" <Gautham.Murthy(a)harman.com> Date: 13/08/2018 18:06 (GMT+01:00) To: users-list(a)monetdb.org Subject: #!ERROR: THRnew: too many threads Hello, We had a DB crash reported with the below messages, Do we need alter any system level parameters or changes to avoid the below issue – Server has 128 CPU and 1TB memory. 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads /etc/security/limits.conf monetadmin soft nproc 102400 monetadmin hard nproc 102400 monetadmin soft nofile 500000 monetadmin hard nofile 500000 monetadmin soft stack 10240 monetadmin soft memlock unlimited monetadmin hard memlock unlimited [monetadmin@lnx1535 DBFARM_TSV_P_B]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb -p 50010 get all TSV_PROD_DB_B name prop source value TSV_PROD_DB_B name - TSV_PROD_DB_B TSV_PROD_DB_B type default database TSV_PROD_DB_B shared default yes TSV_PROD_DB_B nthreads default 128 TSV_PROD_DB_B optpipe local sequential_pipe TSV_PROD_DB_B readonly local yes TSV_PROD_DB_B embedr default no TSV_PROD_DB_B embedpy local yes TSV_PROD_DB_B embedpy3 default no TSV_PROD_DB_B nclients local 2048 TSV_PROD_DB_B dbextra default <unknown> [monetadmin@lnx1535 DBFARM_TSV_P_B]$ 2018-08-13 08:46:30 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:30 MSG merovingian[36790]: proxying client 10.106.5.250:29929 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:46:30 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:33 MSG merovingian[36790]: proxying client 10.106.5.250:29957 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:46:33 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:36 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:40 MSG merovingian[36790]: proxying client 10.106.5.250:64830 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:46:40 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:42 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:47:47 MSG discovery[36790]: new neighbour lnx1536.ch3.prod.i.com (lnx1536.ch3.prod.i.com) 2018-08-13 08:47:48 MSG discovery[36790]: new database mapi:monetdb://lnx1536.ch3.prod.i.com:50010/TSV_PROD_DB_B (ttl=660s) 2018-08-13 08:48:55 MSG merovingian[36790]: database 'TSV_PROD_DB_B' (57408) has crashed (dumped core) 2018-08-13 08:49:12 MSG merovingian[36790]: database 'TSV_PROD_DB_B' has crashed after start on 2018-08-13 05:35:17, attempting restart, up min/avg/max: 2h/5d/3w, crash average: 1.00 0.70 0.43 (23-10=13) 2018-08-13 08:49:12 MSG TSV_PROD_DB_B[72455]: arguments: /monet_binaries/MonetDB-11.27.13_PY/bin/mserver5 --dbpath=/monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B --set merovingian_uri=mapi:monetdb://lnx1535.ch3.prod.i.com:50010/TSV_PROD_DB_B --set mapi_open=false --set mapi_port=0 --set 2018-08-13 08:49:12 MSG TSV_PROD_DB_B[72455]: mapi_usock=/monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock --set monet_vault_key=/monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.vaultkey --set gdk_nr_threads=128 --set max_clients=2048 --set sql_optimizer=sequential_pipe --set embedded_py=true --readonly --set monet_daemon=yes 2018-08-13 08:49:12 MSG TSV_PROD_DB_B[72455]: 2018-08-13 08:49:12 MSG merovingian[36790]: proxying client 10.106.5.250:16467 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:49:12 MSG merovingian[36790]: proxying client 10.106.5.250:45439 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:49:12 MSG merovingian[36790]: starting a proxy failed: cannot connect: Connection refused 2018-08-13 08:49:12 ERR control[36790]: !monetdbd: an internal error has occurred 'cannot connect: Connection refused' 2018-08-13 08:49:12 ERR merovingian[36790]: client error: cannot connect: Connection refused 2018-08-13 08:49:12 MSG merovingian[36790]: proxying client 10.106.5.250:45447 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:49:12 ERR control[36790]: !monetdbd: an internal error has occurred 'cannot connect: Connection refused' 2018-08-13 08:49:12 MSG merovingian[36790]: starting a proxy failed: cannot connect: Connection refused 2018-08-13 08:49:13 ERR merovingian[36790]: client error: cannot connect: Connection refused 2018-08-13 08:49:13 MSG merovingian[36790]: proxying client 10.106.5.250:36605 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B /etc/security/limits.conf monetadmin soft nproc 102400 monetadmin hard nproc 102400 monetadmin soft nofile 500000 monetadmin hard nofile 500000 monetadmin soft stack 10240 monetadmin soft memlock unlimited monetadmin hard memlock unlimited [monetadmin@lnx1535 DBFARM_TSV_P_B]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb -p 50010 get all TSV_PROD_DB_B name prop source value TSV_PROD_DB_B name - TSV_PROD_DB_B TSV_PROD_DB_B type default database TSV_PROD_DB_B shared default yes TSV_PROD_DB_B nthreads default 128 TSV_PROD_DB_B optpipe local sequential_pipe TSV_PROD_DB_B readonly local yes TSV_PROD_DB_B embedr default no TSV_PROD_DB_B embedpy local yes TSV_PROD_DB_B embedpy3 default no TSV_PROD_DB_B nclients local 2048 TSV_PROD_DB_B dbextra default <unknown> [monetadmin@lnx1535 DBFARM_TSV_P_B]$

2 2

#!ERROR: THRnew: too many threads
by Murthy, Gautham 13 Aug '18

13 Aug '18

Hello, We had a DB crash reported with the below messages, Do we need alter any system level parameters or changes to avoid the below issue - Server has 128 CPU and 1TB memory. 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads /etc/security/limits.conf monetadmin soft nproc 102400 monetadmin hard nproc 102400 monetadmin soft nofile 500000 monetadmin hard nofile 500000 monetadmin soft stack 10240 monetadmin soft memlock unlimited monetadmin hard memlock unlimited [monetadmin@lnx1535 DBFARM_TSV_P_B]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb -p 50010 get all TSV_PROD_DB_B name prop source value TSV_PROD_DB_B name - TSV_PROD_DB_B TSV_PROD_DB_B type default database TSV_PROD_DB_B shared default yes TSV_PROD_DB_B nthreads default 128 TSV_PROD_DB_B optpipe local sequential_pipe TSV_PROD_DB_B readonly local yes TSV_PROD_DB_B embedr default no TSV_PROD_DB_B embedpy local yes TSV_PROD_DB_B embedpy3 default no TSV_PROD_DB_B nclients local 2048 TSV_PROD_DB_B dbextra default <unknown> [monetadmin@lnx1535 DBFARM_TSV_P_B]$ 2018-08-13 08:46:30 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:30 MSG merovingian[36790]: proxying client 10.106.5.250:29929 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:46:30 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:31 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:33 MSG merovingian[36790]: proxying client 10.106.5.250:29957 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:46:33 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:36 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:46:40 MSG merovingian[36790]: proxying client 10.106.5.250:64830 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:46:40 MSG merovingian[36790]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-08-13 08:46:42 MSG TSV_PROD_DB_B[57408]: #!ERROR: THRnew: too many threads 2018-08-13 08:47:47 MSG discovery[36790]: new neighbour lnx1536.ch3.prod.i.com (lnx1536.ch3.prod.i.com) 2018-08-13 08:47:48 MSG discovery[36790]: new database mapi:monetdb://lnx1536.ch3.prod.i.com:50010/TSV_PROD_DB_B (ttl=660s) 2018-08-13 08:48:55 MSG merovingian[36790]: database 'TSV_PROD_DB_B' (57408) has crashed (dumped core) 2018-08-13 08:49:12 MSG merovingian[36790]: database 'TSV_PROD_DB_B' has crashed after start on 2018-08-13 05:35:17, attempting restart, up min/avg/max: 2h/5d/3w, crash average: 1.00 0.70 0.43 (23-10=13) 2018-08-13 08:49:12 MSG TSV_PROD_DB_B[72455]: arguments: /monet_binaries/MonetDB-11.27.13_PY/bin/mserver5 --dbpath=/monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B --set merovingian_uri=mapi:monetdb://lnx1535.ch3.prod.i.com:50010/TSV_PROD_DB_B --set mapi_open=false --set mapi_port=0 --set 2018-08-13 08:49:12 MSG TSV_PROD_DB_B[72455]: mapi_usock=/monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock --set monet_vault_key=/monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.vaultkey --set gdk_nr_threads=128 --set max_clients=2048 --set sql_optimizer=sequential_pipe --set embedded_py=true --readonly --set monet_daemon=yes 2018-08-13 08:49:12 MSG TSV_PROD_DB_B[72455]: 2018-08-13 08:49:12 MSG merovingian[36790]: proxying client 10.106.5.250:16467 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:49:12 MSG merovingian[36790]: proxying client 10.106.5.250:45439 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:49:12 MSG merovingian[36790]: starting a proxy failed: cannot connect: Connection refused 2018-08-13 08:49:12 ERR control[36790]: !monetdbd: an internal error has occurred 'cannot connect: Connection refused' 2018-08-13 08:49:12 ERR merovingian[36790]: client error: cannot connect: Connection refused 2018-08-13 08:49:12 MSG merovingian[36790]: proxying client 10.106.5.250:45447 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B 2018-08-13 08:49:12 ERR control[36790]: !monetdbd: an internal error has occurred 'cannot connect: Connection refused' 2018-08-13 08:49:12 MSG merovingian[36790]: starting a proxy failed: cannot connect: Connection refused 2018-08-13 08:49:13 ERR merovingian[36790]: client error: cannot connect: Connection refused 2018-08-13 08:49:13 MSG merovingian[36790]: proxying client 10.106.5.250:36605 for database 'TSV_PROD_DB_B' to mapi:monetdb:///monet_data02/DBFARM_TSV_P_B/TSV_PROD_DB_B/.mapi.sock?database=TSV_PROD_DB_B /etc/security/limits.conf monetadmin soft nproc 102400 monetadmin hard nproc 102400 monetadmin soft nofile 500000 monetadmin hard nofile 500000 monetadmin soft stack 10240 monetadmin soft memlock unlimited monetadmin hard memlock unlimited [monetadmin@lnx1535 DBFARM_TSV_P_B]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb -p 50010 get all TSV_PROD_DB_B name prop source value TSV_PROD_DB_B name - TSV_PROD_DB_B TSV_PROD_DB_B type default database TSV_PROD_DB_B shared default yes TSV_PROD_DB_B nthreads default 128 TSV_PROD_DB_B optpipe local sequential_pipe TSV_PROD_DB_B readonly local yes TSV_PROD_DB_B embedr default no TSV_PROD_DB_B embedpy local yes TSV_PROD_DB_B embedpy3 default no TSV_PROD_DB_B nclients local 2048 TSV_PROD_DB_B dbextra default <unknown> [monetadmin@lnx1535 DBFARM_TSV_P_B]$

1 0

SQL Statements are skipped when run via scripts concurrently on different tables
by Murthy, Gautham 06 Aug '18

06 Aug '18

Hello, Application Teams are experiencing a behavior where few statements are skipped when executed via scripts, They are running multiple scripts concurrently on different tables. Please let us know if there are any know issues or a fix for the reported behavior. Database: MonetDB v11.27.13 (Jul2017-SP4) ü DROP, CREATE, INSERT - New SQL sequence, however looks like SQL statement are not executing as in order in database as per system catalog table log time entries. We need to check on this. ü First DROP, INSERT started and then after couple of statements, CREATE SQL started executing. DROP TABLE IF EXISTS "KEYFOOD_IT_S_22_43321_C";^M CREATE TABLE "KEYFOOD_IT_S_22_43321_C"(ATTR_NAME VARCHAR(500) NOT NULL, COL_NAME VARCHAR(30) NOT NULL, AVP_KEY INTEGER NOT NULL, ATTR_VALUE VARCHAR(3000), ATTR_LONG_VALUE VARCHAR(3000), ATTR_SHORT_VALUE VARCHAR(100), ATTR_MEDIUM_VALUE VARCHAR(100), PARENT_AVP_KEY INTEGER, SORT_ORDER INTEGER); INSERT INTO "KEYFOOD_IT_S_22_43321_C" (ATTR_NAME, COL_NAME, AVP_KEY, ATTR_VALUE, ATTR_LONG_VALUE, ATTR_SHORT_VALUE, ATTR_MEDIUM_VALUE, PARENT_AVP_KEY, SORT_ORDER) SELECT ATTR_NAME, COL_NAME, AVP_KEY, ATTR_VALUE, ATTR_LONG_VALUE, ATTR_SHORT_VALUE, ATTR_MEDIUM_VALUE, PARENT_AVP_KEY, SORT_ORDER FROM "IT_DSJ_KEYFOOD_43321" WHERE COL_NAME='S_22_KEY' AND SORT_TYPE='CATG'; We were able to statements being skipped in sys.querylog_catalog and sys.querylog_calls. Thank You, Gautham

2 1

incorrect data being fetched with WHERE clause
by Murthy, Gautham 06 Aug '18

06 Aug '18

Hello, Data is retrieved which are not available in "where" clause. Application teams have seen some cases where this works fine after some hours. After data load we are seeing incorrect data being fetched. Are there any know issues with the below versions ? Database: MonetDB v11.27.13 (Jul2017-SP4) sql>select * FROM "IT_DSJ_NCKLDN_45121" WHERE COL_NAME='M_59647_KEY' AND SORT_TYPE='ORTH'; +-------------------+--------------+------------+-------------------+-------------------+--------------------+--------------------+------+-------+------+ | attr_name | col_name | avp_key | attr_value | attr_long_value | attr_short_value | attr_medium_value | pare | sort_ | sort | : : : : : : : : nt_a : order : _typ : : : : : : : : : vp_k : : e : : : : : : : : : ey : : : +===================+==============+============+===================+===================+====================+====================+======+=======+======+ | ABRASIVE INFO | S_22_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | CATG | | ABRASIVE INFO | S_22_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | ORTH | | ABRASIVE INFO | S_22_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | ORTH | | ABSORBENCY LEVEL | S_23_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | CATG | | ABSORBENCY LEVEL | S_23_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | ORTH | | ABSORBENCY LEVEL | S_23_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | ORTH | | ACTIVE INGREDIENT | S_25_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | CATG | | ACTIVE INGREDIENT | S_25_KEY | 30000 | N/A | N/A | N/A | N/A | null | 30000 | ORTH | Thank You, Gautham

2 1

Do "upserts" supports in MonetDB?
by Sharma, Sreejith 06 Aug '18

06 Aug '18

Hello, Is there any roadmap for implementing "upserts" or "merge" alike in oracle for MonetDB? Or is it already available? Regards, Sreejith

1 0

Django + MonetDB
by ALEX OSSIPOV 05 Aug '18

05 Aug '18

Hi All I was looking to use MonetDB with Django framework but couldn't find MonetSQLdb package and all references point to 10 years old git code. Is it possible to use current versions of MonetDB with Django and where can I find required packages with configuration instructions? Thanks Alex

1 0

outstanding updates
by Roberto Cornacchia 03 Aug '18

03 Aug '18

Hi, We are having a problem, for a while, that is difficult to isolate. Our ETL includes the following steps: 1) Open JDBC connection, add data to a table, commit, close connection 2) Open JDBC connection, some ALTER TALBLE statements, commit, close connection Occasionally, we get the following during step 2) : java.sql.SQLException: ALTER TABLE: set READ or INSERT ONLY not possible with outstanding updates (wait until updates are flushed) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2732) What puzzles me is that step one is completed when this happens, the commit is done and the connection is closed (we triple-checked that is actually is closed). Notice that inserting a sleep between the to steps makes it work correctly. As I said, this is hard to isolate and reproduce. Still, can anyone guess what is exactly happening? Can it be that a background process is still flushing updates from step 1) ? Even if the JDBC connection is closed ? Can we force a blocking flush, so that it doesn't return until it's safe? If not, how can we know when updates are flushed? Thanks, Roberto

2 5

Mercurial source compilation fails on Mac OS 10.13.5
by Ying Zhang 19 Jul '18

19 Jul '18

Dear all, Not sure how long this has been going on, but I was just trying to refresh my MonetDB installation (compiled from Mercurial source code), but it failed with the following errors: In file included from /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/pyapi3.c:13: /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/../pyapi/pyapi.c:968:18: error: assigning to 'char *' from 'const char *' discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] retnames[i] = PyUnicode_AsUTF8(colname); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~ /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/../pyapi/pyapi.c:1382:19: error: assigning to 'char *' from 'const char *' discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] py_error_string = PyString_AS_STRING(error); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~ Mac OS 10.13.5 Darwin Kernel Version 17.6.0 Happens with both Mar2018 and default branch Does anyone know how to fix these errors and get MonetDB compiled? Thanks in advance! Regards, Jennie

2 2

Mar2018 production compilation fails on Fedora 28
by Roberto Cornacchia 19 Jul '18

19 Jul '18

Compiling from Mar2018 branch, today's hg tip, with production settings: --disable-debug --disable-developer --disable-assert --enable-optimize Fails with this on Fedora 28: In file included from /usr/include/string.h:494, from /home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk.h:316, from /home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk_calc.c:10: In function ‘memcpy’, inlined from ‘addstr_loop’ at /home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk_calc.c:3415:3: /usr/include/bits/string_fortified.h:34:10: error: ‘__builtin_memcpy’ forming offset [3, 65535] is out of the bounds [0, 2] of object ‘str_nil’ with type ‘const char[2]’ [-Werror=array-bounds] return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (same error in several files) Compiling with debug settings succeeds: --enable-debug --enable-developer --enable-assert --disable-optimize Is this reproducible or has it something to do with my machine?

3 2

MonetDB Mar2018 Release - Connection terminates abruptly with no error codes/messages
by Murthy, Gautham 18 Jul '18

18 Jul '18

Hello, MonetDB release - MonetDB Mar2018 We are facing issues when trying to run queries concurrently on the newer version of MonetDB. When 2 or more sessions (create/drop tables) are executed concurrently, sessions get terminated without any error messages. Do we need to explicitly enable error feedback? Does hitting concurrency abruptly terminates the session/connection ? [monetadmin@lnx1490 ~]$ /monet_binaries/MonetDB-11.27.13_PY/bin/mclient mapi:monetdb://lnx1490.ch3.prod.i.com:50105/TSV_PROD_DB_A_C -u DM_POS_TSV_MRT_P6A password: Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4) Database: MonetDB v11.29.7 (Mar2018-SP1), 'mapi:monetdb://lnx1490.ch3.prod.i.com:50105/TSV_PROD_DB_A_C' Type \q to quit, \? for a list of available commands auto commit mode: on sql>\</factorydata/prddata02/prd/crosscut_output/create_desc_it_dsj_campbell_33423.sql [monetadmin@lnx1490 ~]$ Thank You, Gautham

1 0

MonetDB Crashed - Version Jul2017-SP4 - GDKmmap(327680) fails, try to free up space memory in use=48376596752, virtual memory in use=102304729360
by Murthy, Gautham 12 Jul '18

12 Jul '18

Hello All, We had a crash reported on our MonetDB database, Below are the error messages. Can you share any insights on the below reported issue. There were no memory/cpu crunch during the crash, System had ample resource available. MonetDB Version - Jul2017-SP4 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48376596752,virtual memory in use=102304729360<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48368938424,virtual memory in use=102295694776<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48367612864,virtual memory in use=102294369216<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48374211792,virtual memory in use=102300968144<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:47 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(524288) fails, try to free up space memory in use=48312652864,virtual memory in use=103295521856<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:34:33 MSG merovingian315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: database 'TSV_PROD_DB_A' has crashed after start on 2018-07-02 04:52:12, attempting restart, up min/avg/max: 2h/6d/4w, crash average: 1.00 0.60 0.20 (14-8=6) 2018-07-03 06:34:38 ERR control315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: !monetdbd: an internal error has occurred 'unknown or impossible state: 4' 2018-07-03 06:34:40 ERR merovingian315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: client error: unknown or impossible state: 4 2018-07-03 06:34:40 ERR control315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: !monetdbd: an internal error has occurred 'unknown or impossible state: 4' 2018-07-03 06:34:41 ERR merovingian315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: client error: unknown or impossible state: 4 2018-07-03 06:34:41 ERR control315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: !monetdbd: an internal error has occurred 'unknown or impossible state: 4' What was running during the crash ? : Users were dropping, recreating tables and loading data. monetadmin@lnx1535 ~<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/monetadmin%40ln…>$ cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) monetadmin@lnx1535 ~<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/monetadmin%40ln…>$ free -g total used free shared buff/cache available Mem: 1007 199 419 1 388 804 Swap: 31 0 31 monetadmin@lnx1535 ~<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/monetadmin%40ln…>$ cat /proc/cpuinfo|grep -i processor|wc -l 128 Thank You, Gautham

1 0

Memory management, 8 years later
by Roberto Cornacchia 28 Jun '18

28 Jun '18

Hi all, 8 yeas after this interesting thread about using ulimit to limit the data segment size ( https://www.monetdb.org/pipermail/users-list/2010-March/004007.html) a few things have changed and I find myself a bit lost again about memory management. At some point (can't remember when) mserver5 stopped respecting ulimit. No matter how I change the limits, it always sees all available physical ram. Does any other mechanism exist? The issue I'm facing all the time, and for which I find very little documentation, is: - multiple dbfarms, possibly tens of mserver5 instances, running on the same (virtual) machine, plus other applications - Not necessarily, but most times each dmfarm is served by a Docker container - CPU and memory resources need to be share fairly (ideally, controlling priorities) - No mserver5 should ever be killed because of memory issues Now, especially when using Docker, there are means to control the maximum memory size allowed to each process. However, the resolution in case of overusage is to kill the process. That is of course not a solution. What I'm looking for is: can I set a cap to resources *at process level*, in a way that mserver5 will just *see* those, make the best out of them, and never try to use more? (yes, one way would be a VM, but far from ideal for me) Does anyone have experience with / tips about multi-dbfarm environments? Thanks! Roberto

2 2

Troublesome Query
by Doherty, Adam 23 Jun '18

23 Jun '18

Hi all: I'm trying to figure how to calculate the time delta between two events. My events have a timestamp, event source, event type, and an event state and event id. So an event looks like - event_date_time event_source event_type event_state event_id 2018-06-22 09:05:07.000000 MODULE123 TYPEABC ACK 24567890 2018-06-22 11:32:14.000000 MODULE123 TYPEABC UNACK 24812266 This makes up a unique "event", start to finish. However I have hundreds or thousands of entries between and I want be able to group them together in their unique pairings as above. I'm not sure how to wrap my head around this one. Adam Doherty

2 1

Performance issues when streaming records from table.
by Manuel Pagliai 20 Jun '18

20 Jun '18

We are interested in adding support to import records from Monetdb tables in the ETL we are developing. We have performance issues when executing queries like: select col0, col1, ... coln from <table> on tables with a large number of records. What I am concerned about is the time spent upfront, before the first record is retrieved: once records become available we can start processing them. For example, on my machine, executing a executing a select * from <table> query on a 68 million records table takes around 24 seconds for the query to execute, before it is actually possible to start retrieving records from the server response. Queries are submitted via the JDBC driver. Is there a more efficient way of streaming records from a table ? Thanks, Best regards, *Manuel Pagliai* Visokio Ltd www.visokio.com +44 20 7351 4646

1 0

Announcement: New Mar2018-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 20 Jun '18

20 Jun '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Mar2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Mar 2018-SP1 bugfix release (11.29.7) Build Environment * On Linux and Unix, try not to link to libraries multiple times: that only causes the code (and worse, the variables in those libraries) to be included multiple times. On Windows, we do need to link to libraries multiple times (once for each DLL we create if the DLL references anything from the library) but that doesn't cause the linked library to be loaded multiple times. SQL Frontend * Corrected the definition of view: sys.ids. Bug Fixes * 6562: Sqlitelogictest crash on group by query with not in operator * 6565: Sqlitelogictest crash on complex select query with coalesce call * 6566: Sqlitelogictest unavailable calc.- MAL operations * 6568: Sqlitelogictest crash on complex case query * 6569: Sqlitelogictest select query with not between cause with wrong results * 6570: Sqlitelogictest select coalesce undefined calc * 6572: ordered index Error in optimizer garbageCollector * 6573: Sqlitelogictest crash on complex select query with case statement * 6574: server crashed could not find ordered index while creating a table * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6577: creating temp table kills performance of the original query * 6578: One two-tuple insert gives different results than two single inserts * 6581: Join condition errors. * 6583: Fixed size string concatenation with integer results in fixed size string of size 0 * 6584: SELECT FROM REMOTE TABLE WHERE IS NOT NULL produces wrong result * 6585: Nested Merge tables cause an infinite loop in rel_optimizer * 6587: Sqlitelogictest crash on complex case statement * 6589: Sqlitelogictest crash on complex on complex expression * 6594: Sqlitelogictest crash on complex case statement * 6595: Remote decimal division triggers assertion / returns wrong answer * 6598: Python 3.4 not supported (due to usage of Py_DecodeLocale) * 6600: Sqlitelogictest queries fail to execute * 6601: "where is null" clause on remote table causes problem with next query * 6602: Sqlitelogictest wrong results in IN query * 6603: Sqlitelogictest: Aggregation query with distinct clause produces duplicated rows

1 0

pymonetdb or Python UDF
by Doherty, Adam 17 Jun '18

17 Jun '18

Hello all: This is more of curiosity but I have data in my events table currently trapped in a nvarchar field. I am wondering if I can build a table and do execute a select into with a python UDF pulling data from the existing fields plus the values of my nvarchar field. My string data has the following format - "key1=value, key2=val, key3=value"... I want to break out the data first by splitting on the , and then the = Keys would be the columns in the table. I can do this with a python script and pymonetdb but a UDF would be faster. Adam Doherty Process Control Systems Administrator Kearl Technical, Imperial 505 Quarry Park Blvd SE W3B.224 Calgary, AB T2C 5N1 adam.doherty(a)esso.ca<mailto:adam.doherty@esso.ca> P 587.476.4640<tel:+15874764640> M 403.869.1636<tel:+14038691636> imperialoil.ca<http://www.imperialoil.ca/Canada-English/default.aspx> | Twitter<http://www.twitter.com/imperialoil> | YouTube<http://www.youtube.com/imperialoil>

3 3

joinable generate_series with arguments from a table
by Stefan de Konink 14 Jun '18

14 Jun '18

Goodmorning, At this moment there is no functional compatibility between the PostgreSQL and MonetDB with respect to the generate_series function. The range in PostgreSQL is inclusive, in MonetDB exclusive. More importantly generate_series can't be used with argument that is joined from a different table, MonetDB just seems to support a subquery, hence joining that back without a primary is impossible. PostgreSQL; select generate_series(a, b), a from (select 1 as a, 2 as b) as a; generate_series | a -----------------+--- 1 | 1 2 | 1 (2 rows) MonetDB; select * from generate_series((select 1 as a, 2 as b)); +-------+ | value | +=======+ | 1 | +-------+ If I would like to functionally achieve the following; I have list that states the speeds over partial segments, identified by linear referencing. From 10 meter to 12 meter, the distance was 2 meter and speed was 2s. 10 2 2 I would like to expand these values to segments of one meter, hence end up with a table that would state: 10 2 2 11 2 2 12 2 2 Now I am aware of the Python bindings and I could achieve what I wanted by the function below, but I would find generate_series still very practical. DROP FUNCTION expand_per_meter; CREATE FUNCTION expand_per_meter(trip_hash BIGINT, userstopcode STRING, distancesincelastuserstop INTEGER, s INTEGER, t INTEGER) RETURNS TABLE(trip_hash BIGINT, userstopcode STRING, distancesincelastuserstop INTEGER, s INTEGER, t INTEGER) LANGUAGE PYTHON { result = dict() result['trip_hash'] = [] result['userstopcode'] = [] result['distancesincelastuserstop'] = [] result['s'] = [] result['t'] = [] for i in range(0, len(trip_hash)): expanded = range(distancesincelastuserstop[i], distancesincelastuserstop[i] + s[i] + 1) expand = len(expanded) result['trip_hash'] += [trip_hash[i]] * expand result['userstopcode'] += [userstopcode[i]] * expand result['distancesincelastuserstop'] += expanded result['s'] += [s[i]] * expand result['t'] += [t[i]] * expand return result }; -- Stefan

4 4

Re: Combine a table-returning function with the source table
by Mark Clements 13 Jun '18

13 Jun '18

Stefan, This is a good question. For the specific example, you are quite correct. However, the motivation for the question came from using the MonetDB-rmath extension and embedded R: often it is more efficient to calculate a number of values (e.g. the confidence intervals and p-values) together and then return all of the values as a table. This probably also reflects that I am a statistician and an old R user. The recent article by Raasveldt suggested that analysis objects from Python could be stored natively rather than being serialised BLOBs. This could be further extended to storing R objects natively. Members of the objects could then be extracted as they are needed. Admittedly, this would lose many relational advantages. My suggestion is halfway, where R and Python can be used for analytics, with simple relational storage of the results. Actually, is there any way to optimise a lateral call using BATs? -- Mark On 13 Jun 2018 17:43, Stefan Manegold <Stefan.Manegold(a)cwi.nl> wrote: Hi Mark, just out of curiosity, (in the given case/example) what's wrong with a simple select *, mean-1.96*sd as lci, mean+1.96*sd as uci from test; or create or replace function ci(se double) returns double begin return 1.96*se; end; select *, mean-ci(sd) as lci, mean+ci(sd) as uci from test; in case you really want/need to avoid the "redundant" occurrence or "1.96" ... Thanks! Best, Stefan ----- On Jun 13, 2018, at 5:30 PM, Mark Clements mark.clements(a)ki.se wrote: > Jennie, > > Thank you for replying to this inquiry - it's appreciated. > > Your solution gives the right result - but it requires changing the > function call and will not generalise easily to other column types or > other number of covariates. If we had another table such as: > > drop table test; > create table test as select 1 as id, 2 as sex, cast(3 as double) as > mean, cast(4.0 as double) as sd; > -- and repeating the function > create or replace function cis(mu double, se double) returns table(lci > double, uci double) > begin return select mu-1.96*se, mu+1.96*se; > end; > > we can get all of the test table combined with the confidence interval by: > > -- row_number() > select * from > (select *, row_number() over() as rid from test) as t1 > natural join > (select *, row_number() over() as rid from cis((select mean,sd from > test))) as t2; > -- or using lateral > select * from test, lateral cis(test.mean,test.sd); > > The row_number solution is fast, and the lateral solution is simple. I > was hoping for: > > select *, cis(mean,test).* from test; > > Again, thank you for your help. > > -- Mark > > On 06/08/2018 01:53 PM, Ying Zhang wrote: >> Hai Mark, >> >> I’m just wonder if this gives you wat you want: >> >> create function cis(mu double, se double) >> returns table(mu_out double, lci double, uci double) >> begin >> return select mu, mu-1.96*se, mu+1.96*se; >> end; >> >> Regards, >> Jennie > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/<http://www.CWI.nl/~manegold/> | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

1 0

Combine a table-returning function with the source table
by Mark Clements 13 Jun '18

13 Jun '18

For using the MonetDB-rmath extension, I was interested in using table-returning functions to return multiple values. Is there a fast and elegant way to combine the results with the source table in SQL? To make this concrete, consider some test data and a function to calculate a 95% confidence interval from a mean and standard error: drop table test; create table test as select cast(value as double) as value from generate_series(0,5); drop function cis(double,double); create function cis(mu double, se double) returns table(lci double, uci double) begin return select mu-1.96*se, mu+1.96*se; end; We can then call the table-returning function: select * from cis((select value, 1.0 from test)); If we want to re-join the results with the source table, we could use row_number() or use lateral: -- Using row_number: select * from (select *, row_number() over() as rid from test) as t1 natural join (select *, row_number() over() as rid from cis((select value,1.0 from test))) as t2; -- Using lateral: select * from test, lateral cis(test.value, cast(1.0 as double)) as t2; I believe that the latter will lead to a slow loop, even when BAT functions are available. The table-returning function can be used in the select statement only for a scalar expression: select cis(0,1).*; -- ok select cis(0,1).* from test; -- Error: no such operator 'cis' select cis(value,1).* from test; -- Error: identifier 'value' unknown Again, is there a fast and elegant way to combine the source table with a table-returning function in SQL? Kindly, Mark.

3 4

Temporal Function Issues
by Doherty, Adam 13 Jun '18

13 Jun '18

Hello all: I am trying to find the most efficient way to filter results for those matching today only. My events table has 550 million rows and I want a view as follows- CREATE VIEW events_today AS SELECT * FROM events WHERE str_to_date(timestamp_to_str(event_timestamp, '%Y-%m-%d'), '%Y-%m-%d') = CURRENT_DATE which should give me all events with today's date. Running the SELECT by itself also never returns. However, the query just hangs without ever returning. I'm running 11.29.3 (Mar2018) on CentOS 7 with 128GB and 16 Cores. Adam

2 2

High Memory Usage - Sever Not Releasing Resources?
by Doherty, Adam 08 Jun '18

08 Jun '18

Hi all: I ran into an issue yesterday when I ran a simple group count query - The query took 7 minutes, 13 seconds to run. Query was as follows - SELECT zone, count(*) FROM chronicle.events WHERE event_type = 'mb' GROUP BY zone This table, which a merge table currently has 546 million events. The merge table is made up of currently two merge table each representing a year (events_2018, and events_2017). Those merge tables are each made up of four tables, events_year_zone_(1-4) I have a scheduled task which inserts events into the relevant tables every 5 minutes from 5 Microsoft SQL Servers. Roughly 12,000 events broken across 5 inserts (one for each server). The import process is handled with a C# Windows service that utilizes ODBC to talk to MonetDB. The server is currently MonetDB v11.27.11 (July2017-SP3) running on VMware 6.5 in a CentOS 7 VM with 64GB of RAM, and 16 CPU cores. When I ran into this issue yesterday, the mserver5 process was utilizing 90% of available RAM. This was making my insertion task take longer and longer each time, and simple queries like the one above took minutes. Logged into the server via SSH, the input lag as a result of the almost total resource usage caused me to reboot the server. I restarted the VM, and the same query took 20 seconds. I'd like to sort where this issue might be coming from, but I'm not sure where to look. This server is currently part of a proof-of-concept project that will soon be going to production. Any guidance is greatly appreciated. Adam

2 2

ODBC Error on Insert
by Doherty, Adam 06 Jun '18

06 Jun '18

Hi All: I'm having some issues using ODBC with C#. I am trying to insert the contents of a DataTable into Monet using a foreach loop with MonetDB ODBC client under Windows. This is the error I receive... System.Data.Odbc.OdbcException (0x80131937): ERROR [42000] [MonetDB][ODBC Driver 11.27.11]syntax error, unexpected ',' in: "execute 5 (timestamp '2018-06-04 21:22:25',8651,'EVENT','','SYSTEM','1810_INFRA_" This is my loop which is wrapped in a transaction - foreach (DataRow row in events.Rows) { cmd.CommandText = String.Format("INSERT INTO chronicle.events_{0}_zone_{1} VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", year, zone); cmd.Parameters.AddWithValue("@date_time", row["Date_Time"]); cmd.Parameters.AddWithValue("@fracsec", row["FracSec"]); cmd.Parameters.AddWithValue("@event_type", row["Event_Type"]); Truncated for brevity. I'm not sure where the error in my insert statement is because the error message itself appear to be truncated. Very frustrating as it hangs on the first row. Adam

2 1

new MonetDB SQL overview documentation
by martin van dinther 31 May '18

31 May '18

1 0

JIT implementation of the rmath extension
by Mark Clements 30 May '18

30 May '18

The JIT functions by Mark Raasveldt provide a powerful alternative approach to bring C and C++ code into MonetDB. I have provided an alternative implementation of the MonetDB-rmath extension using JIT functions. The JIT implementation is considerably shorter and simpler, and slightly more general - where the code can deal with function arguments that are any combination of scalars and BATs. However, there are indications that the functions are considerably slower. As a minor point, I needed to rename the functions (e.g. using r_ as a prefix) -- otherwise the function names would collide with the function signatures in Rmath.h. To use the JIT-based rmath extension: git clone https://github.com/mclements/MonetDB-rmath.git git checkout jit make sudo make install monetdb create test monetdb release test monetdb set embedc=yes test mclient -d test -s "select r_pnorm(1.96,0,1);" As a challenge for JIT-based extensions, new C functions (e.g. new functions that call Rmath.h, and for which we want to call the new functions in MonetDB) will need a header file that is available to MonetDB. A fragile (non-portable?) solution to find the MonetDB include folder is to use: MONETDBINCLUDE = $(patsubst -I%,%,$(firstword $(shell pkg-config --cflags-only-I monetdb5))) in the Makefile. This assumes that the MonetDB include folder is the first include folder in CFLAGS. The extension-specific header for any new functions can then be copied to that folder: cp rmath_ext.h $(MONETDBINCLUDE) I hope that this is useful. Sincerely, Mark.

1 0

Error: Failed to import - Leftover Data
by Doherty, Adam 23 May '18

23 May '18

Hi all: I am receiving an error "Failed to import table ... Leftover data ..." I am using a simple BASH script and the mclient tool to import my data from hundreds of CSV files. The fields are pipe | separated, text values are surrounded with double-quotes, and rows end with LF (\n). This is the command I am using to process the files - for csvfile in `ls | grep -i 2018.*\.csv$`; do mclient -d mydatabase -s "COPY INTO table FROM '/path/to/csv/files/$csvfile' USING DELIMITERS '|', '\n', '\"'"; done I have a .monetdb file setup in my home directory so as not to need the username and password. Thanks kindly, Adam Doherty

3 5

MonetDB Memory usage and release cached memory
by Murthy, Gautham 17 May '18

17 May '18

Hello, How do we make MonetDB release memory after usage. We are seeing high cached memory on the DB Servers and We also see queries failing with the below error - Memory Allocation Failure : Total Memory - 1123348 Physical Memory - 1123348 Virtual Memory - 2667052 How do we overcome this issue? Thank You, Gautham

1 0

Problem/bug with remote tables
by Bernát Marcell 10 May '18

10 May '18

Hello everyone, I believe I've encountered a bug with the remote table functionality. Here are the steps to replicate it. Tested with MonetDB 11.29.3, both on Ubuntu 14.04.5 LTS and Ubuntu 16.04.4 LTS. The problem persists even after restarting the DB farms and the MonetDB daemons. On node 1 Create a normal table, insert some data and check its contents: create table testtable(id int not null, name text, valid boolean); insert into testtable (id, name) values (1, '1'); insert into testtable (id, name) values (2, '2'); insert into testtable (id) values (3); select * from testtable; +------+------+-------+ | id | name | valid | +======+======+=======+ | 1 | 1 | null | | 2 | 2 | null | | 3 | null | null | +------+------+-------+ 3 tuples (6.446ms) On node 2 Create a remote table and check its contents to make sure it works: create remote table testtable(id int not null, name text, valid boolean) on 'mapi:monetdb://node1:50000/dbfarm'; select * from testtable; +------+------+-------+ | id | name | valid | +======+======+=======+ | 1 | 1 | null | | 2 | 2 | null | | 3 | null | null | +------+------+-------+ 3 tuples (6.446ms) Then try to select anything with a WHERE clause: select * from testtable where name is null; +------+------+-------+ | id | name | valid | +======+======+=======+ | 3 | null | null | +------+------+-------+ 1 tuple (5.186ms) It seemingly works fine, but now when you try to select anything else, you get the same result: select * from testtable; +------+------+-------+ | id | name | valid | +======+======+=======+ | 3 | null | null | +------+------+-------+ 1 tuple (4.466ms) If you log out from mclient and then back in, select * works fine again, so there's no data loss. However, even if you try the opposite of the previous query (is not null instead of is null), you get the same result: select * from testtable where name is not null; +------+------+-------+ | id | name | valid | +======+======+=======+ | 3 | null | null | +------+------+-------+ 1 tuple (5.358ms) Can someone else please try this to confirm it's a bug and not something on my end? Thanks in advance. Best regards, Marcell

3 4

Announcement: MonetDB-rmath
by Mark Clements 09 May '18

09 May '18

Dear all, A first version for MonetDB-rmath is now available. It includes approximately 90% of the functions in Rmath.h, including [rpdq](norm|unif|gamma|beta|lnorm|chisq|nchisq|f|t|binom|cauchy|exp|geom|hyper| nbinom|pois|weibull|logis|wilcox|signrank) and another 40 functions. The current implementation uses m4 for code generation and requires the r-mathlib package under Debian. One can clone the GitHub repository and install using: git clone https://github.com/mclements/MonetDB-rmath.git cd MonetDB-rmath make sudo make install You will then need to use a fresh database. An example script is: -- calculate the quantile and cumulative distribution function for a unit normal select qnorm(0.025,0,1), pnorm(-1.96,0,1); -- calculate E(Z^2) for Z~Normal(0,1) select sum(R_pow(value,2)*0.01*dnorm(value,0,1)) from sys.generate_series(-5.0,5.0,0.01); -- random numbers select set_seed(1,2); select rpois(value*0+100) from sys.generate_series(1,10); -- ok select rpois(100) from sys.generate_series(1,10); -- repeats the values 10 times:-< All of the functions are currently scalars. Next steps include adding BAT functions, including some of the constants at the beginning of Rmath.h, and adding some statistical tests. As an example, I have included poissonci(y,boundary) for an exact Poisson confidence interval for count y for a specific boundary (1=left, 2=right) with default confidence level of 95%: select poissonci(10,1), poissonci(10,2); The equivalent code in R would be poisson.test(10). For confidence intervals, it would be nice to have a type that is a pair of doubles. Sincerely, Mark.

2 2

Buggy attempt at MonetDB-rmath
by Mark Clements 03 May '18

03 May '18

I am trying to adapt MonetDB-gsl to use the rmath library. My code is at: https://github.com/mclements/MonetDB-rmath Under Ubuntu 16.04 LTS, I used: sudo apt install r-mathlib git clone https://github.com/mclements/MonetDB-rmath cd MonetDB-rmath make sudo make install // start up MonetDB server with a test database mclient -d test -s "select sys.pchisq(20.0,5.0);" This compiles but when I try "select sys.pchisq(20.0,5.0);" I get the error: SELECT: no such binary operator 'pchisq(decimal,decimal)' Any guidance would be appreciated. (You may ask why one would want to do this when one could use embedded R - I wanted to see how fast and easy it was to use the rmath library.) Kindly, Mark.

2 3

Copy a set database files and register on another database
by Murthy, Gautham 24 Apr '18

24 Apr '18

Hello, On MonetDB, Can we list the files for a schema and copy the files and register to another database ? We would want to know if we can - List the files for a specific table or a schema(all tables under a schema)? Copy the files to target database and register them on the other database Thank You,

1 0

MonetDB Read-Only without restarting the database
by Murthy, Gautham 24 Apr '18

24 Apr '18

Hello, This is regarding read-only mode in MonetDB. Can we mark the database read-only without restarting the database ? If Yes, How do we do it ? What are the options available to copy a database without restarting to another node or DB Farm? Msqldump the export and import is serialized, How do I increase the performance of exp/imp? Thank You, Gautham

1 0

mclient backward compatibility
by Roberto Cornacchia 18 Apr '18

18 Apr '18

mclient Mar2018 is not really compatible with mserver5 Jul2017. Not a real issue for me, but I'm just curious: is mclient supposed to be compatible with an earlier release of mserver5 (at least the previous release)? $ mclient x -u monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Mar2018) Database: MonetDB v11.27.14 (unreleased), 'mapi:monetdb:// photon.hq.spinque.com:55500/x' Type \q to quit, \? for a list of available commands auto commit mode: on sql>-- \d silently fails (nothing dumped) sql>\d sql> sql>-- \d on a table doesn't find table comments (introduced in Mar2018) sql>\d tables SELECT: no such table 'comments' Shouldn't queries that rely on the system catalog be defined server-side?

2 1

double precision IS NULL does not work ?
by Nicolas Paris 16 Apr '18

16 Apr '18

Hello I have those DOUBLE PRECISION fields, sometimes containing null values: sql>select count(1), range_low from omop.measurement group by range_low; +-----------+--------------------------+ | L2 | range_low | +===========+==========================+ | 304875924 | null | | 9448250 | 50 | | 16519843 | 0 | | 13198932 | 10 | | 14364002 | 20 | | 3964808 | 30 | | 2998528 | 3 | | 855829 | 2 | +-----------+--------------------------+ 8 tuples However, I am not able to filter based on is null: sql>select count(1) from omop.measurement where range_low is null; +------+ | L3 | +======+ | 0 | +------+ 1 tuple I am not able to understand if this behavior is normal regarding the documentation: "The types REAL, FLOAT and DOUBLE map to the underlying implementation system. No special attention is given to the value NaN." Thanks

2 1

mserver5 process consuming resources while not busy
by Nicolas Paris 15 Apr '18

15 Apr '18

Hello Sorry, I gave up digging in the mailing list because didn't found search functionality there ( https://www.monetdb.org/pipermail/users-list/ ) Then I wonder why I get that result when running "top" command on my server: > PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM. > 3297 monetdb 20 0 53,007g 9,028g 8,396g S 100,5 28,8 117:20.95 mserver5 There is no query running AFAIK (select * from sys.queue is empty). I loaded my database several hours before from scratch, and then ran some sql. How can I figure out what appends in the background ? Is it indexing based on queries I made ? Apparently the disk is stable, by mean there is no creation of new indexes, or files under the hood. Any insight welcome, Thanks,

1 0

MonetDB Settings and Open Files
by Murthy, Gautham 14 Apr '18

14 Apr '18

Hello, We have MonetDB open files usage reaching 65k and beyond. How do we minimize the openfiles, We see most of them are mapped to memory. What is the storage(); function? What does the storage() function do when we execute it ? The usage of open files drastically reduced when the select *from storage() was executed on the database. What are the effects of running the select *from storage(); when users are accessing the database? What is the maximum size of database MonetDB can support ? Does running the storage() function when there is a large set of open files crash the database ? Do we need to set any system level parameters ? shmmax, file-max etc., Do we have any standard settings as prerequisites that needs to be considered if we are hosting databases ranging 1TB to 2TB? What should be the open files setting if we have 6-10 databases with size ranging from 100GB to 2TB ? Do we have options to export specific schemas rather that complete database or tables? Request you to please review and share the above information. Open files Test Case - DB size 630Gb Max Open files set to - 65536 [oracle@lnx1386 scripts]$ sh 10min_ofc.sh lsof -u oracle|wc -l Timestamp: 04-14-18 01:56:34 Open Files: 6559 lsof -u oracle|grep -i tcp|wc -l Timestamp: 04-14-18 01:56:34 Open Files Of TCP: 21 lsof -u oracle | grep -i /data/MYDBFARM_DIR/SV11DMP_DBFARM/TSV_PROD_DB | wc -l Timestamp: 04-14-18 01:56:35 Open Files - DB Files : 5864 lsof -u oracle | grep -i protocol: TCP | wc -l Timestamp: 04-14-18 01:56:35 Open Files Of - protocol: TCP : 2 lsof -u oracle | grep -i CLOSE_WAIT | wc -l Timestamp: 04-14-18 01:56:35 Open Files of TCP - CLOSE_WAIT: 0 lsof -u oracle | grep -i ESTABLISHED | wc -l Timestamp: 04-14-18 01:56:36 Open Files Of TCP - ESTABLISHED: 15 netstat -an |grep tcp|wc -l Timestamp: 04-14-18 01:56:36 netstat -an |grep tcp : 62 [oracle@lnx1386 scripts]$ sh /home/oracle/scripts/purge_op_fl.sh 50005 TSV_PROD_DB 1>/home/oracle/scripts/logs/purge_op_fl.stout 2>/home/oracle/scripts/logs/purge_op_fl.sterr [oracle@lnx1386 scripts]$ cat /home/oracle/scripts/purge_op_fl.sh /usr/bin/mclient -p ${1} -d ${2} < /home/oracle/scripts/purge_op_fl.sql [oracle@lnx1386 scripts]$ cat /home/oracle/scripts/purge_op_fl.sql select * from storage(); [oracle@lnx1386 scripts]$ [oracle@lnx1386 scripts]$ sh 10min_ofc.sh lsof -u oracle|wc -l Timestamp: 04-14-18 02:09:49 Open Files: 862 lsof -u oracle|grep -i tcp|wc -l Timestamp: 04-14-18 02:09:49 Open Files Of TCP: 25 lsof -u oracle | grep -i /data/MYDBFARM_DIR/SV11DMP_DBFARM/TSV_PROD_DB | wc -l Timestamp: 04-14-18 02:09:50 Open Files - DB Files : 163 lsof -u oracle | grep -i protocol: TCP | wc -l Timestamp: 04-14-18 02:09:50 Open Files Of - protocol: TCP : 2 lsof -u oracle | grep -i CLOSE_WAIT | wc -l Timestamp: 04-14-18 02:09:50 Open Files of TCP - CLOSE_WAIT: 0 lsof -u oracle | grep -i ESTABLISHED | wc -l Timestamp: 04-14-18 02:09:50 Open Files Of TCP - ESTABLISHED: 19 netstat -an |grep tcp|wc -l Timestamp: 04-14-18 02:09:50 netstat -an |grep tcp : 66 [oracle@lnx1386 scripts]$ Thank You, Gautham

2 1

Window Functions
by Doherty, Adam 13 Apr '18

13 Apr '18

Hello: I am trying to build an event tree from a series timestamps, however for each event I want to be able to return the preceding and succeeding events +/- 2 seconds to capture a complete series of events. I believe this is called lead/lag in other databases. I'm not sure how to proceed in building a CTE, short of pulling all the events out an running a filter function on them in Python. Thanks kindly, Adam Doherty

1 0

statistical MODE function
by Sharma, Sreejith 13 Apr '18

13 Apr '18

All, Is there any statistical MODE function available in MonetDB? If not, has anyone tried it for custom function? Need help here. Regards, Sreejith

2 1

INSERT using multiple cores (inspired by COPY from CSV in LOCKED mode)
by Anton Kravchenko 12 Apr '18

12 Apr '18

Hi there, 1) It looks like INSERTing into tables is bounded by a single CPU. And it kills performance of a query that normally uses multiple CPUs... CREATE TEMP TABLE t1 AS (some 'select query' that uses multiple cores when run standalone) ON COMMIT PRESERVE ROWS; or INSERT into t1 (some 'select query' that uses multiple cores when run standalone); 2) Do you think you could implement INSERT with LOCKED option? (like it was done for COPY from CSV in LOCKED mode) Please let me know whether you think it's doable. Thank you, Anton

1 1

Server Parameter Settings
by Murthy, Gautham 11 Apr '18

11 Apr '18

Hello, Need inputs on the initial prerequisites om server parameter settings - Database Size ranges from 300Gb to 1TB. Optimal sysctl parameters to be set for the server with below configuration - [oracle@lnx1386 ~]$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E7-4850 v4 @ 2.10GHz Stepping: 1 CPU MHz: 1199.953 BogoMIPS: 4197.77 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 40960K NUMA node0 CPU(s): 0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124 NUMA node1 CPU(s): 1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97,101,105,109,113,117,121,125 NUMA node2 CPU(s): 2,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,66,70,74,78,82,86,90,94,98,102,106,110,114,118,122,126 NUMA node3 CPU(s): 3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63,67,71,75,79,83,87,91,95,99,103,107,111,115,119,123,127 [oracle@lnx1386 ~]$ [oracle@lnx1386 ~]$ free -g total used free shared buff/cache available Mem: 2015 17 1945 0 52 1996 Swap: 63 0 63 [oracle@lnx1386 ~]$ I/O - Storage : 10TB SSD Disks Thank You,

1 0

MonetDB - Issue with "too many open files"
by Sharma, Sreejith 11 Apr '18

11 Apr '18

Hi All, We encountered an issue with "too many open files", however not able to trace the root cause . Any information or suggestions here will be very helpful. We have a DB which has around 10K tables with a volume of 300GB approx sizing. The default "File Descriptor Limit" of 16K was set initially and when we encountered "too many open files" issue with parallel loads. Later, it's increased to 65K and upsized the ulimit assuming this should go through. However, at some point the issue reoccurred now with concurrent select or too many concurrent connections. Is there any optimal "file open" size MonetDB recommends? Or is there an option to clean up these? We noticed too many socket connection also opened which is not released. [cid:image001.png@01D3D1F0.58291540] Regards, Sreejith

1 0

Streaming queries
by Jeff Burnett 10 Apr '18

10 Apr '18

Hello, I am searching a CEP to do streaming queries that is embedded and written in C/C++. I also want it to work on linux. Except for the language requirement, Nesper/Esper would be a perfect fit for me. I have wondered if an extension of MonetDbLite would do the job. For me all the data arrives as a time series, like the price of a stock. This can either happen in real time or the data might be replayed from stored history to test other parts of the system. The goal is to build arbitrary queries that fire events when their conditions are met. For example, a listener may be attached to a query that fires if the value of the input stream has changed more than x% over the last n observations. Other events might fire every day at 4pm, indicating the end of the day. As I said, Nesper/Esper do this out of the box, but Nesper is in .net and esper is in Java. My code base is in c++ and JNI is unappealing. Does MonetdbLite handle these things already? Where might I run into difficulty in trying to adapt monet for this project? Any help is appreciated. Please let me know if I didn’t explain the task very well. Jeff

1 0

MonetDB - Transaction aborted issue
by Sharma, Sreejith 10 Apr '18

10 Apr '18

All, We are getting below error while creating a table. In the current session no other bulk load has happened. CREATE TABLE IF NOT EXISTS "SCHEMA_NAME"."TABLE_NAME" AS SELECT * FROM " SCHEMA_NAME "."TABLE_NAME" WHERE 1=0; INSERT INTO "SCHEMA_NAME"."TABLE_NAME" SELECT * FROM "SCHEMA_NAME"."TABLE_NAME2"; ERROR = !COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead ~ Also, is there a way to update multiple tables at the same time without above error? Is this due to concurrent insert? Any help here is much appreciated! Regards, Sreejith

3 3

ordered index Error in optimizer garbageCollector
by Anton Kravchenko 06 Apr '18

06 Apr '18

Here is a reproducible example. create table t1(v0 int, v1 char(1)); insert into t1 values (1,'a'),(2,'b'),(3,'c'); create ordered index index_t1_v1 on t1(v1); create table t2(v1 char(1)); insert into t2 values ('a'); 1) gives "Error in optimizer garbageCollector" create temp table t3 as (select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1) on commit preserve rows; 2) works as expected create temp table t3 as (select t1.v0 from t1,t2 where t2.v1=t1.v1) on commit preserve rows; 3) works as expected select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1; 4) works as expected drop index index_t1_v1; create temp table t3 as (select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1) on commit preserve rows; I am using the latest MonetDB version: MonetDB 5 server v11.29.3 "Mar2018" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 31.3GiB available memory, 4 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.2k 26 Jan 2017 (compiled with ) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: mockbuild@ (x86_64-redhat-linux-gnu) Compilation: gcc -std=gnu99 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic Linking : /usr/bin/ld -m elf_x86_64 -Wl,-z,relro -Wl,-Bsymbolic-functions Thanks, Anton

2 1

Announcement: New Mar2018 Feature release of MonetDB suite
by Sjoerd Mullender 29 Mar '18

29 Mar '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Mar2018 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Some notable changes: - Code base requires a C99 compiler. - The internal representation of REAL and DOUBLE NULL values has changed. Before they were represented as the smallest C float/double finite value (-FLT_MAX and -DBL_MAX). Now they are represented by a NaN (not-a-number) value. This requires a database upgrade which is done automatically at startup. Be advised that it is recommended to make a backup before upgrading. - Various extensions to SQL have been implemented. For more details, see below. Mar 2018 feature release (11.29.3) MonetDB5 Server * Implemented function pcre.replace_first which is like pcre.replace, except it only replaces the first match. * The EXPLAIN command now shows all the MAL type resolutions, because in general users may not be aware of their signatures. It also simplifies programs to analyze such plans. * Implemented versions of group.(sub)group(done) that don't return a histogram. * Removed MAL functions streams.socketRead, streams.socketReadBytes, streams.socketWrite, and streams.socketWriteBytes. * Removed MAL functions streams.openRead(s:streams):streams and streams.openWrite(s:streams):streams. * Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard). Build Environment * Added the .pdb files needed for debug symbols to the Windows installer for MonetDB/SQL. . * mclient's execution time profiling options and output format have been changed. Instead of implicitly via the "--interactive"/"-i" option (or when using an interactive mclient console), execution time profiling is now controlled via an explicit "--timer=timermode"/"-t timermode" command-line option, or a "\t timermode" command in the interactive mclient console. The default (also in the interactive mclient console) is now timermode "none", i.e., no timing information is given. Timermode "clock" activates client-side wall-clock timing ("clk") in "human-friendly" format much like the interactive mode did before. Timermode "performance" also provides detailed server-side timings: "sql" is the time to parse the SQL query, optimize the logical relational plan and create the initial physical (MAL) plan; "opt" is the time to optimize the physical (MAL) plan; "run" is the time to execute the physical (MAL) plan. With timermode "performance" all server-side timings and the client-side wall-clock time are given in milliseconds (ms). Note that the client-measured wall-clock time "clk" is reported per query only when options "--interactive" or "--echo" are used, because only then does mclient send individual lines (statements) of the SQL script to the server. Otherwise, mclient sends the SQL script in large(r) batch(es) to the server, and, thus, only the total wall-clock time per batch is measured and reported. The server-measured detailed performance timings "sql", "opt", "run" are always measured and reported per query. Also, all timing information is now given on a separate line and sent to stderr rather than stdout. * Some types and constants were moved from configure (and hence monetdb_config.h) to gdk.h. In particular, the types "lng" and "ulng" have been moved and can therefore no longer be used by code that doesn't (ultimately) include gdk.h. Just use int64_t instead. A bunch of format defines have been removed: SZFMT, SSZFMT, PTRFMT, PDFMT. Just use the C standard codes for those (%zu, %zd, %p, %td). The define for printing a lng (LLFMT) was also moved. Use PRId64 for printing int64_t in code not using gdk.h. Removed all references to __int64 and long long (use int64_t instead). Merovingian * Add daemon commands for starting/stopping collection of profiler (stethoscope) logs. Client Package * ODBC: The driver function SQLProcedureColumns was implemented. * ODBC: Changed table types as used by SQLTables from "LOCAL TEMPORARY" and "GLOBAL TEMPORARY" to "LOCAL TEMPORARY TABLE" and "GLOBAL TEMPORARY TABLE" respectively. * ODBC: Initial support for the HUGEINT type in SQL was added. Any value with type HUGEINT can be retrieved in a C variable with type SQL_C_CHAR or SQL_C_WCHAR. A value of type HUGEINT can be retrieved in other C types as long as they fit, the largest C type supported being a 64 bit integer (equivalent to BIGINT). * The functions in the mapi library that require 64 bit integers now use the standard type int64_t instead of the non-standard mapi_int64. This requires a compilation environment that has the stdint.h include file (standardized in C99). Compilation of the library also requires the inttypes.h include file (also standardized in C99). * Add a new pretty printing option to stethoscope Running stethoscope with the flag -j will produce not pretty printed output (one json object per line). Running with the -y flag will produce pretty printed output. Running with neither will produce the legacy, line oriented format mapilib * The three mapi_explain* functions and mapi_trace don't return any useful information, so they now return void. * The functions mapi_error_str and mapi_result_error now return const char * instead of plain char * to indicate that the returned data belongs to the library and should not be changed or freed by the application. * New function const char *mapi_result_errorcode(MapiHdl) which returns the SQLSTATE code if available when an error has occurred. stream * The interface of mnstr_fgetpos and mnstr_fsetpos was changed to look more like the standard C functions fsetpos and fgetpos: they both have a second argument "pointer to fpos_t". * Removed function wbstream. * Removed functions udp_rastream and udp_wastream. * Removed functions socket_rstream and socket_wstream. * Removed functions append_wstream and append_wastream. * Removed functions mnstr_rstream and mnstr_wstream. utils * The function mcrypt_getHashAlgorithms now returns a static, constant string, so the result should not be modified or freed. MonetDB Common * Changed return type of function void_replace_bat from BUN to gdk_return: it now only returns whether the operation succeeded or not. * Changed the return type of BATroles from void to gdk_return: it can fail due to malloc failure. * Removed functions ALIGNsetH, ALIGNsetT, and CREATEview_ (mind the underscore). The first can easily be replace by using BAThseqbase (that's all it did), the second was only used once, and the third can be replace by VIEWcreate. * Removed unused functions BATmemsize and BATvmsize. * Removed the tnodense property: it was maintained but never actually used, not even stored. * The NIL representation of the internal flt and dbl types was changed from the smallest representable finite value to NaN (not-a-number). * Changed the interface of ATOMformat and VALformat: they now return a pointer to the allocated string. * The length "method" for atoms now returns a size_t, the "len" field of a ValRecord is now a size_t, the "size" field of the atomDesc structure is now unsigned short. * Removed the "align" field from the ATOM descriptor (atomDesc) structure. * The atomtostr and atomfromstr "methods" for atoms now return ssize_t and require a pointer to size_t for the size of the buffer. * The atom tostr and fromstr "methods" now always return -1 on error. A return value greater than 0 is normal, a return value of 0 is not normal, but technically not an error. Testing Environment * Added a --data_path option to Mtest.py that defines an external data repository. See the commit message of c484932c7fd8 for more info. SQL * Extended support to use CREATE ORDERED INDEX on columns of type: char, varchar, clob, blob, url, json, inet and uuid. * Added new system view: sys.ids which contains all database objects ids which can be used in sys.dependencies table. * Added new system view: sys.dependencies_vw which shows all data of sys.dependencies including names on objects, object types and dependency types. * Added 25 new system views for finding out dependencies between database objects. These new dependency views improve, extend and replace the 17 sys.dependencies_X_on_Y() functions as previously defined in 21_dependency_functions.sql. Those sys.dependencies_X_on_Y() functions are now marked as deprecated. * Added new system view: sys.roles which contains all defined roles. * Added new system view: sys.var_values which shows the values for system variables. * Added support for COMMENT ON statements using SQL syntax: COMMENT ON { SCHEMA | TABLE | VIEW | COLUMN | INDEX | SEQUENCE | FUNCTION | PROCEDURE | AGGREGATE | FILTER FUNCTION | LOADER } qname IS { 'my description text' | NULL | '' } ; For COLUMN the qname can be "table_name"."column_name" or fully qualified as in: "schema_name"."table_name"."column_name". For FUNCTION, PROCEDURE, AGGREGATE, FILTER FUNCTION and LOADER the qname may need to include the signature (argument types) to be able to differentiate between multiple overloaded functions which have the same name and schema. By specifying IS NULL or IS '' you remove the comment for the database object. If a database object is dropped, the associated comment is also removed. Note: it is not allowed or possible to add comments for temporary tables or objects in schema "tmp". The sql catalog has been extended with system table: sys.comments. The keyword 'COMMENT' has now become a reserved keyword. * Removed system function sys.environment(). It was a duplicate of system function sys.env(). * Implemented behavior for DROP SCHEMA my_schema RESTRICT command. Previously the RESTRICT keyword was accepted but not obeyed. It would always do a CASCADE operation. Also the default behavior of DROP SCHEMA my_schema command is now changed into RESTRICT behavior (was CASCADE). * The internal NULL representation of the REAL (FLOAT) and DOUBLE types was changed from the smallest representable finite value to NaN (not-a-number). * A column default value can be used in a UPDATE statement: UPDATE tname SET cname = DEFAULT, and INSERT statements: INSERT INTO tname VALUES (..., DEFAULT, ...) * Added support for TRUNCATE statements conforming to the SQL:2008 standard: TRUNCATE [ TABLE ] qname [ CONTINUE IDENTITY | RESTART IDENTITY ] [ RESTRICT | CASCADE ] In a TRUNCATE statement a 'CONTINUE IDENTITY' or 'RESTART IDENTITY' clause can be passed to restart or not, being the former the default one. The 'CASCADE' option instructs to truncate referencing table(s) also if the referencing table(s) have foreign key references to this table. The default behavior is 'RESTRICT'. Note: it is possible to use TRUNCATE statements in a transaction and thus to rollback the effects of a truncate. The keywords 'TRUNCATE' and 'CONTINUE' have now become reserved keywords. * Added possibility to GRANT or REVOKE a TRUNCATE privilege to a user or role. * Added possibility to define a TRIGGER on a TRUNCATE event. * Added possibility to specify 'OR REPLACE' in following CREATE commands: CREATE [ OR REPLACE ] VIEW qname ... CREATE [ OR REPLACE ] TRIGGER qname ... * Added possibility to specify 'IF EXIST' in following DROP commands: DROP AGGREGATE [ IF EXISTS ] qname ... DROP FUNCTION [ IF EXISTS ] qname ... DROP FILTER FUNCTION [ IF EXISTS ] qname ... DROP LOADER [ IF EXISTS ] qname ... DROP PROCEDURE [ IF EXISTS ] qname ... DROP TRIGGER [ IF EXISTS ] qname ... * Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard). * Added support for extracting the quarter (number between 1 and 4) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( QUARTER FROM my_date_expr ). Added support for extracting the week (number between 1 and 53) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( WEEK FROM my_date_expr ). Added support for scalar functions: quarter(date_expr), quarter(timestamp_expr) and quarter(timestamptz_expr). Bug Fixes * 3574: Add support for: create OR REPLACE view ... * 3824: Created table not visible from ODBC * 3831: Extend date part extraction and date formating functions to support more formats like "quarter", "iso year/week" * 6244: Add support for: TRUNCATE TABLE * 6249: DEFAULT in row-values missing (sqlsmith) * 6346: BATsort returns GDK_SUCCEED when **sorted bat is null * 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT; * 6507: Column Header coming with trailing spaces in compiled version of MonetDB 11.27.9/11 * 6513: Sqlitelogictest: Wrong MAL plan generation for column product * 6526: Crash using aggregate function inside a case statement in having clause * 6529: Sqlitelogictest crash in select query with IN operator and cast * 6530: Sqlitelogictest: select query with NOT IN giving wrong results * 6532: copy into ignore null as directive if first column doesn't come from file * 6534: [Mar2018]: mclient -f tab / --format=tab complains about "unsupported formatter" * 6535: [Mar2018]: mclient -t / --timer does not work as documented * 6536: [Mar2018]: timing output of mclient -t / --timer= should go to stderr rather than stdout * 6537: [Mar2018]: mclient's default timing mode should (again) be "none" rather than "clock" * 6541: [Mar2018]: mclient reports incorrect wall-clock time * 6542: assertion failure when querying: select count(*) from sys.commented_function_signatures; * 6543: Mar2018: truncate on SQL system tables should NOT be allowed * 6545: Sqlitelogictest crash in IN query * 6546: Sqlitelogictest crash in IN query with division * 6547: OS-dependent behaviour for ilike * 6548: Select from remote table leaves session open * 6549: Add log messages when listen fails * 6550: Sqlitelogictest crash on complex CASE statement * 6551: Sqlitelogictest wrong NULL value cast * 6552: Sqlitelogictest crash on complex case statement * 6553: Sqlitelogictest crash on aggregation with having statement * 6554: Sqlitelogictest crash on complex case statement * 6555: Sqlitelogictest wrong result set generated from complex case statement * 6556: Sqlitelogictest division by zero on COALESCE call * 6557: Sqlitelogictest crash on aggregation query with not in * 6559: rows in sys.statistics are not removed when a temporary table is dropped. * 6560: Sqlitelogictest crash on group by query with having in * 6561: Sqlitelogictest crash on group by query with having not in clause

1 0

RE: Adding sum() to a query appears to disable multithreading/parallel execution
by Anderson, David B 28 Mar '18

28 Mar '18

But couldn't the exact summation operate in parallel on each of the sub-groups coming from the group by statement? I guess I was thinking that part of the computation would be split up as well. Would it be difficult to add a psum() function that would drop bits (so understood to be an approximation), but still operate in parallel? Thanks, Dave -----Original Message----- From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Sjoerd Mullender Sent: Tuesday, March 27, 2018 10:36 AM To: users-list(a)monetdb.org Subject: Re: Adding sum() to a query appears to disable multithreading/parallel execution The reason for the lack of parallelization in case of sum() is that the column is of type float. When summing floating point numbers we try to give the correct result, even if there are large differences between individual values (e.g. a column that has 1e37, 1e-37, and -1e37, the result should be 1e-37, not 0). For this reason, we use an algorithm that remembers all bits during summing, but we then cannot parallelize the sum since the bits that didn't fit in a float (or double) for the partial sums get lost. On 27/03/18 15:42, Anderson, David B wrote: > All, > > Not entirely sure how to submit a bug report for this, but it appears that adding a sum() term to a query completely disables parallel execution of the query. > > When I run this query on my server (11.27.13 built from source), linux RH6, 24 core, 190GB memory: > > select s.product, d.fctrdt, d.delinq, count(*), count(d.realized) from > dyn d, stt s where s.loanseqnum = d.loanseqnum group by s.product, > d.fctrdt, d.delinq order by s.product, d.fctrdt, d.delinq; > > It executes in ~20 seconds (the table stt contains ~20 million > records, and the table dyn contains ~550 million records), and when I > run top I can see the process using all 24 cores, > > but when I change the term count(d.realized) to sum(d.realized), the query now takes 5-6 minutes to run, and when I run top, the mserver process appears to use a single core for most of the query. > > select s.product, d.fctrdt, d.delinq, count(*), sum(d.realized) from > dyn d, stt s where s.loanseqnum = d.loanseqnum group by s.product, > d.fctrdt, d.delinq order by s.product, d.fctrdt, d.delinq; > > Is this expected behavior? Am I doing something wrong? Queries are both run on a hot server. > > Note: the explain from the query containing sum(d.realized) only contains 103 tuples, but the initial query with count(d.realized) contains 1052 tuples. > > Thanks, > Dave > > > Here are the definitions of the stt/dyn tables: > > create TABLE stt ( > product char(12) NULL, > prefix char(2) NULL, > coupon float NULL, > productionyear smallint NULL, > cusip char(9) NULL, > loanseqnum char(12) NULL, > loanpurpose char(1) NULL, > tpoflag char(1) NULL, > proptype char(2) NULL, > occstatus char(1) NULL, > numunits smallint NULL, > state char(2) NULL, > cs smallint NULL, > origterm smallint NULL, > origltv float NULL, > orignoterate float NULL, > origloanamt float NULL, > origwala smallint NULL, > origserv char(8) NULL, > govtagy char(1) NULL); > > create TABLE dyn ( > fctrdt DATE NULL, > loanseqnum char(12) NULL, > scheduled float NULL, > realized float NULL, > smm float NULL, > delinq int NULL, > paidahead int NULL); > > And here is the Merovingian log: > > $ more merovingian.log > 2018-03-26 09:17:16 MSG merovingian[26418]: Merovingian 1.7 > (Jul2017-SP4) starting > 2018-03-26 09:17:16 MSG merovingian[26418]: monitoring dbfarm > /opt/flash1/test/DB > 2018-03-26 09:17:16 MSG merovingian[26418]: Merovingian 1.7 stopped > 2018-03-26 09:17:16 ERR merovingian[26418]: binding to stream socket > port 50000 failed: Address already in use > 2018-03-26 09:17:16 ERR merovingian[26418]: fatal startup condition > encountered, aborting startup > 2018-03-26 09:18:15 MSG merovingian[28537]: Merovingian 1.7 > (Jul2017-SP4) starting > 2018-03-26 09:18:15 MSG merovingian[28537]: monitoring dbfarm > /opt/flash1/test/DB > 2018-03-26 09:18:15 MSG merovingian[28537]: accepting connections on > TCP socket localhost:50000 > 2018-03-26 09:18:15 MSG merovingian[28537]: accepting connections on > UNIX domain socket /tmp/.s.monetdb.50000 > 2018-03-26 09:18:15 MSG discovery[28537]: listening for UDP messages > on localhost:50000 > 2018-03-26 09:18:15 MSG control[28537]: accepting connections on UNIX > domain socket /tmp/.s.merovingian.50000 > 2018-03-26 09:18:15 MSG control[28537]: (local): created database 'loan' > 2018-03-26 09:18:15 MSG control[28537]: (local): served status list > 2018-03-26 09:18:15 MSG control[28537]: (local): released database 'loan' > 2018-03-26 09:18:15 MSG merovingian[28537]: starting database 'loan', > up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) > 2018-03-26 09:18:15 MSG merovingian[28537]: proxying client (local) > for database 'loan' to > mapi:monetdb:///opt/flash1/test/DB/loan/.mapi.sock?database=loan > 2018-03-26 09:18:15 MSG merovingian[28537]: target connection is on > local UNIX domain socket, passing on filedescriptor instead of > proxying > 2018-03-26 09:18:15 MSG loan[28550]: arguments: > /opt/tools/root/MonetDB-11.27.13/bin/mserver5 > --dbpath=/opt/flash1/test/DB/loan --set > merovingian_uri=mapi:monetdb://malxcs1p:50000/loan --set > mapi_open=false --set mapi_port=0 - -set > mapi_usock=/opt/flash1/test/DB/loan/.mapi.sock --set > monet_vault_key=/opt/flash1/test/DB/loan/.vaultkey --set > gdk_nr_threads=24 --set max_clients=64 --set > sql_optimizer=default_pipe --set monet_daemon=yes > 2018-03-26 09:18:15 MSG loan[28550]: # MonetDB 5 server v11.27.13 "Jul2017-SP4" > 2018-03-26 09:18:15 MSG loan[28550]: # Serving database 'loan', using > 24 threads > 2018-03-26 09:18:15 MSG loan[28550]: # Compiled for > x86_64-pc-linux-gnu/64bit with 128bit integers > 2018-03-26 09:18:15 MSG loan[28550]: # Found 189.145 GiB available main-memory. > 2018-03-26 09:18:15 MSG loan[28550]: # Copyright (c) 1993 - July 2008 CWI. > 2018-03-26 09:18:15 MSG loan[28550]: # Copyright (c) August 2008 - > 2018 MonetDB B.V., all rights reserved > 2018-03-26 09:18:15 MSG loan[28550]: # Visit https://www.monetdb.org/ > for further information > 2018-03-26 09:18:15 MSG loan[28550]: # Listening for UNIX domain > connection requests on > mapi:monetdb:///opt/flash1/test/DB/loan/.mapi.sock > 2018-03-26 09:18:15 MSG loan[28550]: # MonetDB/SQL module loaded > 2018-03-26 09:18:16 MSG loan[28550]: # SQL catalog created, loading > sql scripts once > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 09_like.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 10_math.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 11_times.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 12_url.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 13_date.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 14_inet.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 15_querylog.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 16_tracelog.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 17_temporal.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 18_index.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 20_vacuum.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 21_dependency_functions.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 22_clients.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 23_skyserver.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 25_debug.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 26_sysmon.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 27_rejects.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 39_analytics.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 39_analytics_hge.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 40_json.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 40_json_hge.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 41_md5sum.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 45_uuid.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 46_profiler.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 51_sys_schema_extension.sql > 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: > 75_storagemodel.sql > > > > > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > -- Sjoerd Mullender

3 3

DB Crashing Post Data Load
by Sharma, Sreejith 28 Mar '18

28 Mar '18

All, We have a scenario where we load the data via the shell scripts from .dat file using ControlM process. We notice the DB crashes intermittently post the load. Is there any limitation or settings on multiple session loads? Or what happens if load is parallel? 2018-03-27 00:21:54 MSG merovingian[45726]: database 'TEST_MDP' has crashed after start on 2018-03-13 05:04:41, attempting restart, up min/avg/max: 2m/4d/2w, crash average: 1.00 1.00 0.80 (35-9=26 MonetDB : 11.27.13 OS : CentOS 6 Regards, Sreejith

2 2

Adding sum() to a query appears to disable multithreading/parallel execution
by Anderson, David B 27 Mar '18

27 Mar '18

All, Not entirely sure how to submit a bug report for this, but it appears that adding a sum() term to a query completely disables parallel execution of the query. When I run this query on my server (11.27.13 built from source), linux RH6, 24 core, 190GB memory: select s.product, d.fctrdt, d.delinq, count(*), count(d.realized) from dyn d, stt s where s.loanseqnum = d.loanseqnum group by s.product, d.fctrdt, d.delinq order by s.product, d.fctrdt, d.delinq; It executes in ~20 seconds (the table stt contains ~20 million records, and the table dyn contains ~550 million records), and when I run top I can see the process using all 24 cores, but when I change the term count(d.realized) to sum(d.realized), the query now takes 5-6 minutes to run, and when I run top, the mserver process appears to use a single core for most of the query. select s.product, d.fctrdt, d.delinq, count(*), sum(d.realized) from dyn d, stt s where s.loanseqnum = d.loanseqnum group by s.product, d.fctrdt, d.delinq order by s.product, d.fctrdt, d.delinq; Is this expected behavior? Am I doing something wrong? Queries are both run on a hot server. Note: the explain from the query containing sum(d.realized) only contains 103 tuples, but the initial query with count(d.realized) contains 1052 tuples. Thanks, Dave Here are the definitions of the stt/dyn tables: create TABLE stt ( product char(12) NULL, prefix char(2) NULL, coupon float NULL, productionyear smallint NULL, cusip char(9) NULL, loanseqnum char(12) NULL, loanpurpose char(1) NULL, tpoflag char(1) NULL, proptype char(2) NULL, occstatus char(1) NULL, numunits smallint NULL, state char(2) NULL, cs smallint NULL, origterm smallint NULL, origltv float NULL, orignoterate float NULL, origloanamt float NULL, origwala smallint NULL, origserv char(8) NULL, govtagy char(1) NULL); create TABLE dyn ( fctrdt DATE NULL, loanseqnum char(12) NULL, scheduled float NULL, realized float NULL, smm float NULL, delinq int NULL, paidahead int NULL); And here is the Merovingian log: $ more merovingian.log 2018-03-26 09:17:16 MSG merovingian[26418]: Merovingian 1.7 (Jul2017-SP4) starting 2018-03-26 09:17:16 MSG merovingian[26418]: monitoring dbfarm /opt/flash1/test/DB 2018-03-26 09:17:16 MSG merovingian[26418]: Merovingian 1.7 stopped 2018-03-26 09:17:16 ERR merovingian[26418]: binding to stream socket port 50000 failed: Address already in use 2018-03-26 09:17:16 ERR merovingian[26418]: fatal startup condition encountered, aborting startup 2018-03-26 09:18:15 MSG merovingian[28537]: Merovingian 1.7 (Jul2017-SP4) starting 2018-03-26 09:18:15 MSG merovingian[28537]: monitoring dbfarm /opt/flash1/test/DB 2018-03-26 09:18:15 MSG merovingian[28537]: accepting connections on TCP socket localhost:50000 2018-03-26 09:18:15 MSG merovingian[28537]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000 2018-03-26 09:18:15 MSG discovery[28537]: listening for UDP messages on localhost:50000 2018-03-26 09:18:15 MSG control[28537]: accepting connections on UNIX domain socket /tmp/.s.merovingian.50000 2018-03-26 09:18:15 MSG control[28537]: (local): created database 'loan' 2018-03-26 09:18:15 MSG control[28537]: (local): served status list 2018-03-26 09:18:15 MSG control[28537]: (local): released database 'loan' 2018-03-26 09:18:15 MSG merovingian[28537]: starting database 'loan', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2018-03-26 09:18:15 MSG merovingian[28537]: proxying client (local) for database 'loan' to mapi:monetdb:///opt/flash1/test/DB/loan/.mapi.sock?database=loan 2018-03-26 09:18:15 MSG merovingian[28537]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2018-03-26 09:18:15 MSG loan[28550]: arguments: /opt/tools/root/MonetDB-11.27.13/bin/mserver5 --dbpath=/opt/flash1/test/DB/loan --set merovingian_uri=mapi:monetdb://malxcs1p:50000/loan --set mapi_open=false --set mapi_port=0 - -set mapi_usock=/opt/flash1/test/DB/loan/.mapi.sock --set monet_vault_key=/opt/flash1/test/DB/loan/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2018-03-26 09:18:15 MSG loan[28550]: # MonetDB 5 server v11.27.13 "Jul2017-SP4" 2018-03-26 09:18:15 MSG loan[28550]: # Serving database 'loan', using 24 threads 2018-03-26 09:18:15 MSG loan[28550]: # Compiled for x86_64-pc-linux-gnu/64bit with 128bit integers 2018-03-26 09:18:15 MSG loan[28550]: # Found 189.145 GiB available main-memory. 2018-03-26 09:18:15 MSG loan[28550]: # Copyright (c) 1993 - July 2008 CWI. 2018-03-26 09:18:15 MSG loan[28550]: # Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved 2018-03-26 09:18:15 MSG loan[28550]: # Visit https://www.monetdb.org/ for further information 2018-03-26 09:18:15 MSG loan[28550]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/flash1/test/DB/loan/.mapi.sock 2018-03-26 09:18:15 MSG loan[28550]: # MonetDB/SQL module loaded 2018-03-26 09:18:16 MSG loan[28550]: # SQL catalog created, loading sql scripts once 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 09_like.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 10_math.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 11_times.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 12_url.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 13_date.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 14_inet.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 15_querylog.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 16_tracelog.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 17_temporal.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 18_index.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 20_vacuum.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 21_dependency_functions.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 22_clients.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 23_skyserver.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 25_debug.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 26_sysmon.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 27_rejects.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 39_analytics.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 39_analytics_hge.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 40_json.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 40_json_hge.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 41_md5sum.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 45_uuid.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 46_profiler.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 51_sys_schema_extension.sql 2018-03-26 09:18:16 MSG loan[28550]: # loading sql script: 75_storagemodel.sql

2 1

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 26 Mar '18

26 Mar '18

total used free shared buff/cache available Mem: 17825792 59904 17765888 0 0 17765888 Swap: 33554432 0 33554432

1 0

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 26 Mar '18

26 Mar '18

I'm not sure that you can get the accessory(script). Here I put the text. expe.sh: #!/bin/bash BASEDIR=$(cd `dirname $0`;pwd -P) DAT_DIR=$BASEDIR/tpc-ds_test_data10G #drop tables if they are existed for i in $(ls $DAT_DIR/*.dat) do filename=$(echo $i | awk -F '/' '{print $NF}' | awk -F '.' '{print $1}') echo "start dropping table $filename" #./drop_tables.sh $filename mclient -s "drop table $filename;" echo "end dropping table $filename" done #create tables mclient /home/monetdb/v2.6.0/tools/tpcds.sql #load data for i in $(ls $DAT_DIR/*.dat) do filename=$(echo $i | awk -F '/' '{print $NF}' | awk -F '.' '{print $1}') #./autoinsert.sh $filename echo "start loading data to table $filename" mclient -s "copy into $filename from '$DAT_DIR/$filename.dat' using delimiters '|' null as '';" echo "end loading data to table $filename" done the query is showed in previous email.

1 0

Less verbose logs?
by Roberto Cornacchia 22 Mar '18

22 Mar '18

merovingian.log files on our production servers are dominated by these two lines: 2018-03-21 16:55:38 MSG merovingian[23237]: proxying client localhost.localdomain:51502 for database 'test' to mapi:monetdb://path/test/.mapi.sock?database=test 2018-03-21 16:55:38 MSG merovingian[23237]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying They account for 90% of the log. Is there a way to reduce verbosity?

1 1

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 22 Mar '18

22 Mar '18

The free disk space is 800G. And the query also fail when I run query right after a restart of the MonetDB server. And I'm not very sure what the mserver5 size means. When I run the query, the log file provide some information below, is that your mentioned? #GDKmremap(2465464320) fails, try to free up space [memory in use=149750256,virtual memory in use=4497801712] Also, I'm sorry that the information(query statement and result) I put in is incomplete by my negligence. I copy the query statement and result and error message again as followed: result and error message: GDK reported error. MT_mremap: mremap(0x7ffc80000000,2054553600,2465464320) failed !OS: Cannot allocate memory HEAPextend: failed to extend to 2465464320 for 55/5577.tail: GDKmremap() failed +-------------+--------------+-------+ | c_last_name | c_first_name | sales | +=============+==============+=======+ +-------------+--------------+-------+ 0 tuples query statement: with frequent_ss_items as ( select substr(i_item_desc,1,30) itemdesc ,i_item_sk item_sk ,d_date solddate ,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999+1,1999+2,1999+3) group by itemdesc ,i_item_sk ,d_date having count(*) >4 ), max_store_sales as ( select max(csales) tpcds_cmax from ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk ) x ), best_ss_customer as ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * ( select * from max_store_sales ) ) select sum(sales) from ( select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) y limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999 + 1,1999 + 2,1999 + 3) group by itemdesc,i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk) x), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) x order by c_last_name,c_first_name,sales limit 100; Thanks! Regards, Rancho

2 1

M5server.bat error
by Amit Ambekar 22 Mar '18

22 Mar '18

Hi, i had installed monetdb on Microsoft server 12, but now when i want to start M5server , error occured. And i had tried this on various machine of Windows Server 2012, still same issue occurred. Is monetdb is not compatible with Windows 2012 server?

2 1

How to use MonetDB cluster?
by Li Japin 20 Mar '18

20 Mar '18

Hi, Recently, I read the MonetDB cluster management document, but I'm not sure which situation will use the MonetDB cluster and how to use it. I guess the cluster will combine with REMOTE table, is that right? But how can I monitor the remote table is online? using `monetdb discover` command? Best regards, Japin

1 0

merge tables: how to reveal member tabels?
by Stefan Manegold 16 Mar '18

16 Mar '18

Hi, given a merge table, e.g., sql>create table t1(a int, b real); operation successful (9.315ms) sql>create table t2(a int, b real); operation successful (8.497ms) sql>create merge table mt(a int, b real); operation successful (8.218ms) sql>alter table mt add table t1; operation successful (7.174ms) sql>alter table mt add table t2; operation successful (6.178ms) How can I see which tables are members of a merge tables, i.e., in this case, how can I see that t1 & t2 are members of mt? a plain select (obviously) only shows the content of mt, i.e., the combined content of t1 & t2 ... sql>select * from mt; +---+---+ | a | b | +===+===+ +---+---+ 0 tuples (4.690ms) Is there a query to get this directly from mt, or where can I otherwise find that info in the SQL catalog? Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 2

Can I avoid "n affected rows" messages?
by Eyal Rozenberg 15 Mar '18

15 Mar '18

When you perform an insert or update using mclient, you get a line printed saying "n rows affected". Now, I can avoid this line if I use -f trash on the command line - but I don't want to give up on actual output. I can also pass the output through a grep -Ev "^[0-9]+ rows affected$" as a filter, but I was wondering if I could just turn that message off somehow. I couldn't find a command-line switch which would have this effect. Eyal

2 1

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 15 Mar '18

15 Mar '18

Hello all, I did a tpc-ds testing on MonetDB, and when it ran some queries, the error message was appeared: GDK reported error. MT_mremap: mremap(0x7ffdc0000000,2054553600,2465464320) failed !OS: Cannot allocate memory HEAPextend: failed to extend to 2465464320 for 44/4415.tail: GDKmremap() failed It seems that the memory has been ran out. But I have no idea what is the cause of this error. The version of Monetdb is 11.27.13, and OS is Ubunbtu-16.04 with memory is 8G, and with 20G testing data.(It is worth mentioning, a table named store_sales is 7.5G in the query.) The query statement is: with frequent_ss_items as ( select substr(i_item_desc,1,30) itemdesc ,i_item_sk item_sk ,d_date solddate ,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999+1,1999+2,1999+3) group by itemdesc ,i_item_sk ,d_date having count(*) >4 ), max_store_sales as ( select max(csales) tpcds_cmax from ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk ) x ), best_ss_customer as ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * ( select * from max_store_sales ) ) select sum(sales) from ( select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) y limit 100; Please give me some advice if you have any idea, either of the direct question or my expression. Thanks a lot! Regards, Rancho

2 2

MonetDB - table without column- NOT able to delete it or use it
by Sharma, Sreejith 12 Mar '18

12 Mar '18

All, We are finding a strange issue where below table has been created via a shell script. Strangely it doesn't show up any columns and when we try to drop it's saying table doesn't exists. Is there any way below can happen? Is this because it's corrupted? How can we remove this? CREATE TABLE "DM_MON_TSV_STG_T1"."AMA_US_IRI_8135_2001" ( ); Regards, Sreejith

4 3

Backup and restore in Monetdb
by Amit Ambekar 10 Mar '18

10 Mar '18

Hi, How i can take back of monetdb database , and how to restore it?

2 1

current_time() precision
by Roberto Cornacchia 28 Feb '18

28 Feb '18

sql>select current_time(); +-----------------------+ | L2 | +=======================+ | 17:05:10.000000+01:00 | +-----------------------+ 1 tuple (0.796ms) Is it expected that the result type's precision of current_time() is limited to seconds? Any way to get milliseconds?

1 0

Monetdb hanging on query
by Manuel Pagliai 27 Feb '18

27 Feb '18

Hello, I am using Monetdb as the query engine of an application currently in development. I am connecting to monetdb via a jdbc connection. I can easily reproduce a scenario where monetdb gets stuck on a query, and never complete execution. Although reproducible, the issue is non deterministic, and it can occur while executing different queries, or not occur at all. When it happens I can see the thread stuck at I extracted the queries submitted to monetdb, and execute them via a standalone client (both the native client and a jdbc-based one) rather than our application, but I have been unable to reproduce the issue that way. I guess timing is important. The problem still occur even when queries are submitted sequentially and setting gdk_nr_threads=1 and sql_optimizer=sequential_pipe. The issue happens in Jul 2017-SP3 bugfix release (11.27.11), running on a Windows 7 machine. I have not tried 11.27.13, but by looking at the bug fixes I can't see anything relevant to this issue. Is there something that I can try, or a way to gather more information about what is happening ? I can share both the data, and the complete list of queries used during this scenario. Thanks, Best regards, *Manuel Pagliai* Visokio Ltd www.visokio.com +44 20 7351 4646 P.S. This is the stack-trace of the thread submitting the query when this happens: at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000000cae0b730> (a java.io.BufferedInputStream) at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream._read(Unknown Source) at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.readBlock(Unknown Source) at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(Unknown Source) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) - locked <0x00000000cae13810> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) - locked <0x00000000cae13810> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:389) at nl.cwi.monetdb.mcl.io.BufferedMCLReader.readLine(Unknown Source) at nl.cwi.monetdb.mcl.io.BufferedMCLReader.waitForPrompt(Unknown Source) - locked <0x00000000cae137d8> (a nl.cwi.monetdb.mcl.io.BufferedMCLReader) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(Unknown Source) - locked <0x00000000cae0b430> (a nl.cwi.monetdb.mcl.net.MapiSocket) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(Unknown Source) at nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(Unknown Source) at nl.cwi.monetdb.jdbc.MonetStatement.execute(Unknown Source) at nl.cwi.monetdb.jdbc.MonetStatement.executeQuery(Unknown Source)

2 1

MonetDB :: Alter statement to change datatype
by Sharma, Sreejith 22 Feb '18

22 Feb '18

All, Is it possible to change the datatype of a column in an existing table? With or without data? I saw the only way to change is to create a temporary column and swap if data is present. Is it true? Regards, Sreejith

2 1

Re: Order indexes: used or not?
by Stefan Manegold 20 Feb '18

20 Feb '18

Hi Torsten, To start mserver5 manually on a database created by monetdb/monetdb, you need to use the same command line (-options) the monedtd use to start mserver5, in particular the valley option. You find the details in the merovingian.log in your dbfarm Best,Stefan -------- Original message --------From: Torsten Grust <torsten.grust(a)gmail.com> Date: 2/20/18 17:08 (GMT+01:00) To: Lefteris <lsidir(a)gmail.com> Cc: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: Re: Order indexes: used or not? Hi Lefteris, that's very interesting. Thanks! I was about to experiment with this here, but I stumbled. When you start a database via mserver5 directly (instead of using monetdbd which, as I understand, is the recommend method), does the method of authentication change? I fail to login to my scratch database: $  mserver5 --dbpath=<path>/scratch # MonetDB 5 server v11.27.13 "Jul2017-SP4" # Serving database 'scratch', using 4 threads # [...] # MonetDB/SQL module loaded > [switch to other terminal] $ mclient -d scratch -u monetdb password: <supply password "monetdb"> InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' Whereas the login works just fine below: $ monetdbd start <path> $ mclient -d scratch -u monetdb password: <supply password "monetdb"> Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4) Database: MonetDB v11.27.13 (Jul2017-SP4), 'mapi:monetdb://closure:50000/scratch' [...] sql> I must be doing something stupid. Cheers, —Torsten On Tue, Feb 20, 2018 at 3:40 PM Lefteris <lsidir(a)gmail.com> wrote: Hi Torsten, the reason why you don't see anything in the EXPLAIN statement is that the use of an index is decided at runtime by the respective operator. The algebra.sort code will have an if statement that determines at query evaluation time if there is a usable ordered index and use it. The best way (for now) to see if the index is used or not is to start the mserver5 with the --algorithms flag. This flag will produce alot of output on the runtime decissions. For your example, I get the following output: #BATproject(l=tmp_523#3-sorted-key,r=tmp_433#3[int]-revsorted-key) #BATproject(l=tmp_523,r=tmp_433)=tmp_323#3-revsorted-key #BATcheckorderidx: reusing persisted orderidx 283 #BATproject(l=tmp_322#3-key,r=tmp_323#3[int]-revsorted-key) #BATproject(l=tmp_322,r=tmp_323)=tmp_417#3-key 11us #BATgroup(b=tmp_417#3[int],s=NULL#0,g=NULL#0,e=NULL#0,h=NULL#0,subsorted=1): trivial case: 1 element per group #BATprojectchain with 3 (2) BATs, size 3, type int #BATproject(l=tmp_322#3-key,r=tmp_323#3[int]-revsorted-key) #BATproject(l=tmp_322,r=tmp_323)=tmp_523#3-key 9us And the third line says "BATcheckorderidx: reusing persisted orderidx 283" which is an indication that MonetDB has found the index and is using it. As a side note, I think in this case it has also figured out that x is revsorted-key (aka unique reverse sorted) but I dont know which one wins over the other:) So in short, if you start mserver5 with --algorithms, redirect the console output in a file and search for BATcheckorderidx you will see if the ordered index is used. Please let us know if there are cases that ordered index should be used but it is not. lefteris On Fri, Feb 16, 2018 at 10:28 AM, Martin Kersten <martin.kersten(a)cwi.nl> wrote: > Hi > Same holds for ordered indices > > On 16/02/2018 10:27, Torsten Grust wrote: >> >> Hi Martin, >> >> thanks for the super-quick reply. You are talking of /imprints/ here but >> I assume that the same is true for /order indexes/ (these are separate >> concepts, right?). >> >> Best wishes, >> —Torsten >> >> >> On Fri, Feb 16, 2018 at 9:55 AM Martin Kersten <martin.kersten(a)cwi.nl >> <mailto:martin.kersten@cwi.nl>> wrote: >> >> Dear Torsten >> >> The use of an imprint is only part of the algebra.select >> implementation >> and depends on actual several properties. >> >> In this particular query, the SQL compiler when encountering an order >> by >> issues a sort statement, even if this is strictly not needed. That >> will be detected >> by the sort implementation. >> >> regards, Martin >> On 15/02/2018 23:31, Torsten Grust wrote: >> > Dear all, >> > >> > given a table definition and (explained) SQL query as follows: >> > >> > |CREATE TABLE foo (x INT,y INT); INSERT INTO foo VALUES (3,1), >> (2,2), (1,3); ALTER TABLE foo SET READ ONLY; CREATE ORDERED INDEX foo_x ON >> foo(x); EXPLAIN SELECT * FROM foo ORDER BY x; | >> > >> > In the |EXPLAIN| output, am I supposed to see that the order index >> is indeed used? I currently do see an |algebra.sort| operation, of course, >> but do not see any reference to the |foo_x| index. I had expected to find an >> |bat.getorderidx| operation in the >> > MAL program. >> > >> > Can anyone shed light on this? I'd be grateful. >> > >> > Cheers, >> > >> > —Torsten >> > >> > -- >> > | Prof. Dr. Torsten Grust >> > | Database Systems — Universität Tübingen (Germany) >> > | ✉︎ torsten.grust(a)uni-tuebingen.de >> <mailto:torsten.grust@uni-tuebingen.de> >> <mailto:torsten.grust@uni-tuebingen.de >> <mailto:torsten.grust@uni-tuebingen.de>> >> > | db.inf.uni-tuebingen.de <http://db.inf.uni-tuebingen.de> >> <http://db.inf.uni-tuebingen.de> >> > >> > >> > _______________________________________________ >> > users-list mailing list >> > users-list(a)monetdb.org <mailto:users-list@monetdb.org> >> > https://www.monetdb.org/mailman/listinfo/users-list >> > >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org <mailto:users-list@monetdb.org> >> https://www.monetdb.org/mailman/listinfo/users-list >> >> -- >> | Prof. Dr. Torsten Grust >> | Database Systems — Universität Tübingen (Germany) >> | ✉︎ torsten.grust(a)uni-tuebingen.de >> <mailto:torsten.grust@uni-tuebingen.de> >> | db.inf.uni-tuebingen.de <http://db.inf.uni-tuebingen.de> >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list -- | Prof. Dr. Torsten Grust | Database Systems — Universität Tübingen (Germany)| ✉︎ torsten.grust(a)uni-tuebingen.de| db.inf.uni-tuebingen.de

2 1

Order indexes: used or not?
by Torsten Grust 20 Feb '18

20 Feb '18

Dear all, given a table definition and (explained) SQL query as follows: CREATE TABLE foo (x INT,y INT); INSERT INTO foo VALUES (3,1), (2,2), (1,3); ALTER TABLE foo SET READ ONLY; CREATE ORDERED INDEX foo_x ON foo(x); EXPLAIN SELECT * FROM foo ORDER BY x; In the EXPLAIN output, am I supposed to see that the order index is indeed used? I currently do see an algebra.sort operation, of course, but do not see any reference to the foo_x index. I had expected to find an bat.getorderidx operation in the MAL program. Can anyone shed light on this? I'd be grateful. Cheers, —Torsten -- | Prof. Dr. Torsten Grust | Database Systems — Universität Tübingen (Germany) | ✉︎ torsten.grust(a)uni-tuebingen.de | db.inf.uni-tuebingen.de

3 5

matrix manipulation
by Jiří Fejfar 20 Feb '18

20 Feb '18

Dear MonetDB users! I would like to do some simple calculations (matrix multiplication and inversion) with large matrices (cca 40k by 40k or more). We are using PostgreSQL & R but we are facing problems when matrix doesn't fit memory. My initial idea was to use SciQL, but probably it is not merged into MonetDB yet [1]? I am sending also short presentation [2] illustrating nature of our data and calculation (slide 6). Thank you for your suggestions! Jiří. [1] https://www.monetdb.org/pipermail/users-list/2014-May/007319.html [2] https://drive.google.com/open?id=1P4Nuo_YuqhNyxoBqQHeIw6WiXe28I8kE

1 0

Select from remote table leaves session open
by Николай Чернов 19 Feb '18

19 Feb '18

I have two databases (v11.27.13 both) on two Debian8 machines. One with corrupted table (in my case it got corrupted due to bug in Jul2017-SP1) and another with remote table to the corrupted table. When I run the SELECT query from the remote table I get an error. And the session, opened by this query to the first DB did not close. I see it by querying 'sys.sessions' table. And every time I run this SELECT I get additional record in 'sys.sessions'. This happens only if en error occurs, on successful SELECT the session immediately disappears. How can I force this sessions to close or is it a bug?

1 0

Converting a timestamp to a UNIX timestamp
by Bernát Marcell 15 Feb '18

15 Feb '18

Dear MonetDB users, I'm looking for a way to convert a timestamp (or timestamp string) like "2018-02-14 23:59:59" to a UNIX timestamp using MonetDB's functions. In MySQL this can be done using the function UNIX_TIMESTAMP(), so I'm looking for the MonetDB equivalent of that, if it exists. I found sys.epoch(), but that does the opposite of what I need. Thanks in advance. Best regards, Marcell

1 1

Announcement: New Jul2017-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 12 Feb '18

12 Feb '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP4 bugfix release (11.27.13) Build Environment * Added the .lib and .h files needed for building extensions to the Windows installer. Bug Fixes * 3470: Support setClob without length restrictions * 6468: JDBC 2.27 fails with year < 1000 * 6482: Query failures on order by on union * 6483: Monetdb crashes, on query * 6487: UNION of NULLs with several tables fails * 6488: Semijoin returns duplicate values from a column with unique values * 6489: Sqlitelogictest - Wrong result set of complex conditional query * 6490: Sqlitelogictest - Select query with an IN clause parse error * 6491: SELECT IN returns NULL instead of false when not found * 6492: Persistent hashes stored and then ignored. Storage info not in sync with actual indices. * 6493: Sqlitelogictest - Aggregation query on empty table with DISTINCT clause * 6494: Sqlitelogictest - Algebra operators priority in select query * 6495: Sqlitelogictest - Omitting AS in a result set column alias name * 6496: Sqlitelogictest - Select interval comparisons between floating-points and NULL * 6497: Sqlitelogictest - Select not between query producing wrong results * 6498: large virtual memory spike on BLOB column COUNT * 6499: Crash when trying to replace a function defined in sys from a different schema * 6502: Query with multiple limit clauses does not return anything * 6508: Segmentation fault in mserver5 on Python2 UDF with TIMESTAMP column input that has NULL values (conversion.c:438, PyNullMask_FromBAT) * 6510: Sqlitelogictest: Wrong output in aggregation query * 6512: Monetdb crashes on query with limit after sort with case * 6514: Sqlitelogictest: Range query between NULL values not possible * 6515: Insert null second interval value results in 0 * 6516: Sqlitelogictest unknown bat append operation * 6517: Sqlitelogictest overflow in conversion during MAL plan execution * 6518: Sqlitelogictest: count aggregation with not in operator * 6519: Sqlitelogictest: algebra join between lng and int BATs undefined * 6520: UPDATE with correlated subquery causes assertion (or segfault) * 6522: Sqlitelogictest: IN operator return a single column * 6523: Sqlitelogictest: Case statement subquery missing * 6524: Sqlitelogictest: Crash in aggregation query with IN operator * 6527: Crash using order by alias in subquery

1 0

MonetDB Connection drops issue on concurrency
by Sharma, Sreejith 12 Feb '18

12 Feb '18

Hi All, We have a situation where we are trying to emulate a scenario where we have multiple clients hitting into MonetDB instance in parallel, I mean with concurrency of for say 12 users. Queries are going through fine, however we notice connection drops in some cases and queries get cancelled. We assume, this happens when port is idle and when trying to reconnect there is a SQLConnect failure issues. If it's queries are firing uninterruptedly no issues spotted, if there is any idle situation the issue occurs. Can the "forward" - "redirect" option help here? If not, what other options? Regards, Sreejith

1 0

Diacritic removal
by Roberto Cornacchia 09 Feb '18

09 Feb '18

Just wondering, before I do it myself, whether "normalizing" strings by removing diacritics is a feature that you guys would consider interesting to implement. (I am aware that results of this "normalization" can be unsatisfactory in some cases - I'm ok with most obvious cases covered) Basically, something like what iconv does here: $ echo "Ã é ç" | iconv -f UTF-8 -t ASCII//TRANSLIT A e c I can imagine that these above would be pretty easy to solve using the same hash lookup approach used for case conversion. These, however, are example that produce more than one symbol in output. So that would be a little more complicated. $ echo "æ ß" | iconv -f UTF-8 -t ASCII//TRANSLIT ae ss So I have two questions: 1) Is this something you guys (MonetDB developers) would be interested in implementing? 2) If not - in that case I'll give it a go myself, would you advise an approach similar to the case conversion or to use the iconv lib? Thanks for your input. Rboerto

2 2

Install on Linux with no access to Internet
by ALEX OSSIPOV 08 Feb '18

08 Feb '18

I am trying to install MonetDB with client on the host without access to Internet. Are there any instructions to follow? Thanks Alex

3 5

MonetDB Evaluation - Excessive slowness in a particular query
by Alfio Bettini 06 Feb '18

06 Feb '18

Hi all, we're testing MonetDB performances in order to understand if it could be possible to use it as data mart in our BI software. We're considering a set of queries executed in our test environment (CentOS 7, 32 GB RAM, 4 cpu). All the executed queries have quite good performances (between 10 and 45 seconds) except the following one that is taking several minutes in order to be executed: SELECT id, count(distinct value) FROM table GROUP BY id; The table contains 100 millions of data. It has about 130 columns (varchar, int, bigint, smallint, tinyint, timestamp, datetime). We have no BLOB/CLOB/Text types. The fields used in the query have these data types: id: int value: bigint Can you suggest us any optimizations in order to have this query run faster? Do you know some possible reasons why it's so slow? Kind Regards, Alfio Bettini The information contained in this message, together with any attachments, may be privileged and confidential and is intended for the use of the addressee(s) only. Any review, copying, use or distribution of this email by others (including any of its attachments) is strictly prohibited. If you are not the intended recipient, please destroy this email and its attachments and contact the sender immediately. This email was sent by EidosMedia SpA<http://www.eidosmedia.com>

2 3

SQL Connect issues - ODBC Driver failure when idle situation
by Sharma, Sreejith 06 Feb '18

06 Feb '18

Hi All, Have a quick question - I have a situation where we are trying to emulate a scenario where we have multiple clients hitting into MonetDB instance in parallel, I mean with concurrency of for say 12 users. Queries are going through fine, however we notice some cases when a port/machine is idle and trying to reconnect we are finding SQLConnect failure issues. If it's queries are firing continuously it's fine, here if when some idle situation it may occur. Can the "forward" - redirect option help here? Regards, Sreejith

1 0

SQL Connect issues - ODBC Driver failure when idle situation
by Sharma, Sreejith 01 Feb '18

01 Feb '18

Hi All, Have a quick question - I have a situation where we are trying to emulate a scenario where we have multiple clients hitting into MonetDB instance in parallel, I mean with concurrency of for say 12 users. Queries are going through fine, however we notice some cases when a port/machine is idle and trying to reconnect we are finding SQLConnect failure issues. If it's queries are firing continuously it's fine, here if when some idle situation it may occur. Can the "forward" - redirect option help here? Regards, Sreejith

1 0

Crash using aggregate function inside a case statement in having clause
by Trevor Andreas 30 Jan '18

30 Jan '18

Hello, Database: MonetDB v11.27.11 (Jul2017-SP3) I am experiencing a crash when attempting to use an aggregate function inside a case statement in a having clause. Below is an example. One use case for this is that I am computing a formula on aggregated data, and want to ensure no divide by zero occurs. Are case statements in having clauses supported? Thanks! create table "t1" ( "id" int, "c1" varchar(100), "d1" int, "d2" int ); insert into "t1" values (1, 'A', 50, 80), (2, 'A', 200, 350), (3, 'A', 89, 125), (4, 'B', 4845, 13), (5, 'B', 194, 597), (6, 'C', 5636, 5802), (7, 'C', 375, 3405), (7, 'D', 365, 0), (7, 'D', 87, 0); -- Works select "c1", sum("d1") as "d1", sum("d2") as "d2" from "t1" group by "c1" having sum("d1") < case when 5 > 10 then 500 else 400 end; -- Works select "c1", sum("d1")as "d1", sum("d2") as "d2", 1.0 * sum("d1") / (1.0 * case when sum("d2") > 0 then sum("d2") else null end) as "formula" from "t1" group by "c1"; -- Crashes select "c1", sum("d1")as "d1", sum("d2") as "d2", 1.0 * sum("d1") / (1.0 * case when sum("d2") > 0 then sum("d2") else null end) as "formula" from "t1" group by "c1" having (1.0 * sum("d1") / (1.0 * case when sum("d2") > 0 then sum("d2") else null end)) > 1; Looking in merovingian.log: 2018-01-29 14:47:15 ERR test[2879]: *** Error in `/usr/bin/mserver5': double free or corruption (out): 0x00007f57200ea4b0 *** 2018-01-29 14:47:18 MSG merovingian[504]: database 'test' (2879) has crashed (dumped core) 2018-01-29 14:49:35 MSG merovingian[504]: database 'test' has crashed after start on 2018-01-29 14:42:36, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.20 0.07 (2-0=2) 2

2 1

Help with performance issue
by Roberto Cornacchia 26 Jan '18

26 Jan '18

Hi there, Would someone like to pick up this wonderful opportunity to tell me that I'm doing something silly? Please do. In this loop, the cleanup take 4/5 of the total time. Is there a better way of doing this? loop(...) { BAT *gn, *en, *hn; BAT *idxn, *tn, *pn; // tokenize, group, transform, append // this takes 1/5 of the total loop time if (BATutf8_tokenize(&tokens, s, delims, min_tok_len) != GDK_SUCCEED) goto fail; if (BATgroup(&gn, &en, &hn, tokens, NULL, NULL, NULL, NULL) != GDK_SUCCEED) goto fail; idxn = BATconstant(0, TYPE_int, &idx, BATcount(en), TRANSIENT); tn = BATproject(en,tokens); pn = BATconvert(hn, NULL, TYPE_dbl, TRUE); if (BATappend(br1,idxn,NULL,FALSE) != GDK_SUCCEED) goto fail; if (BATappend(br2,tn,NULL,FALSE) != GDK_SUCCEED) goto fail; if (BATappend(br3,pn,NULL,FALSE) != GDK_SUCCEED) goto fail; // this takes 4/5 of the total loop time BBPreclaim(idxn); BBPreclaim(tn); BBPreclaim(pn); BBPreclaim(hn); BBPreclaim(gn); BBPreclaim(en); BBPreclaim(tokens); }

3 6

"brew install monetdb" failed due to "libxml2 library not found"
by Ying Zhang 25 Jan '18

25 Jan '18

Dear all, I try to install MonetDB with homebrew, but it failed with the error: configure: error: libxml2 library not found but required for MonetDB5 But as far as I can tell, libxml2 is already installed, e.g. $ brew list libxml2 /usr/local/Cellar/libxml2/2.9.7/bin/xml2-config /usr/local/Cellar/libxml2/2.9.7/bin/xmlcatalog /usr/local/Cellar/libxml2/2.9.7/bin/xmllint /usr/local/Cellar/libxml2/2.9.7/include/libxml2/ (47 files) /usr/local/Cellar/libxml2/2.9.7/lib/libxml2.2.dylib /usr/local/Cellar/libxml2/2.9.7/lib/cmake/libxml2/libxml2-config.cmake /usr/local/Cellar/libxml2/2.9.7/lib/pkgconfig/libxml-2.0.pc /usr/local/Cellar/libxml2/2.9.7/lib/python2.7/ (6 files) /usr/local/Cellar/libxml2/2.9.7/lib/ (3 other files) /usr/local/Cellar/libxml2/2.9.7/share/aclocal/libxml.m4 /usr/local/Cellar/libxml2/2.9.7/share/doc/ (153 files) /usr/local/Cellar/libxml2/2.9.7/share/gtk-doc/ (55 files) /usr/local/Cellar/libxml2/2.9.7/share/man/ (4 files) $ brew install libxml2 Warning: libxml2-2.9.7 already installed Is there anything I can do to fix this? Thanks a lot in advance! Jennie =============== Complete output of “brew install monetdb” =============== $ brew install monetdb Warning: You are using macOS 10.13. We do not provide support for this pre-release version. You may encounter build failures or other breakages. Please create pull-requests instead of filing issues. ==> Using the sandbox ==> Downloading https://www.monetdb.org/downloads/sources/Jul2017-SP3/MonetDB-11.27.11.tar.… Already downloaded: /Users/jennie/Library/Caches/Homebrew/monetdb-11.27.11.tar.xz ==> ./configure --prefix=/usr/local/Cellar/monetdb/11.27.11 --enable-debug=no --enable-assert=no --enable-optimi Last 15 lines from /Users/jennie/Library/Logs/Homebrew/monetdb/01.configure: checking how to link with libiconv... -liconv checking for iconv declaration... extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft); checking for library containing uuid_generate... none required checking for CC_MD5_Update... yes checking for CC_RIPEMD160_Update... no checking for CC_SHA1_Update... yes checking for CC_SHA224_Update... yes checking for CC_SHA256_Update... yes checking for CC_SHA384_Update... yes checking for CC_SHA512_Update... yes checking for pcre... yes checking whether pcre comes with UTF-8 support... yes checking for libxml2... no configure: error: libxml2 library not found but required for MonetDB5 READ THIS: https://git.io/brew-troubleshooting If reporting this issue please do so at (not Homebrew/brew): https://github.com/Homebrew/homebrew-core/issues Warning: You are using macOS 10.13. We do not provide support for this pre-release version. You may encounter build failures or other breakages. Please create pull-requests instead of filing issues.

2 2

Re: MonetDB - question on rename table
by David Jankoski 25 Jan '18

25 Jan '18

hey sreejith, maybe this SO answer will be helfpul to you: https://stackoverflow.com/questions/19661538/how-to-rename-a-table-in-monet… hth. david On 25 January 2018 at 08:56, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. MonetDB-extend clang error: ld: unknown option: -soname > (Ying Zhang) > 2. Re: MonetDB-extend clang error: ld: unknown option: -soname > (Sjoerd Mullender) > 3. "brew install monetdb" failed due to "libxml2 library not > found" (Ying Zhang) > 4. MonetDB - question on rename table (Sharma, Sreejith) > 5. Re: MonetDB - question on rename table (Stefan Manegold) > 6. RE: MonetDB - question on rename table (Sharma, Sreejith) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 24 Jan 2018 15:56:19 +0100 > From: Ying Zhang <Y.Zhang(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: MonetDB-extend clang error: ld: unknown option: -soname > Message-ID: <1DD5C9B8-D559-4CD2-89E0-35CA729C9B67(a)cwi.nl> > Content-Type: text/plain; charset=utf-8 > > Dear all, > > I was trying to compile the regexp example in the MonetDB-extend repo on a Mac OS 10.13, but got an ?unknown option? error: > > $ make > cc -fPIC -DPIC -o lib_regexp.so -shared regexp.o -L/usr/local/opt/openssl/lib -L/Users/jennie/scratch/monet-install/Jul2017/debug/lib -L/usr/local/Cellar/pcre/8.41/lib -lmonetdb5 -lbat -lpcre -Wl,-soname -Wl,lib_regexp.so > ld: unknown option: -soname > clang: error: linker command failed with exit code 1 (use -v to see invocation) > make: *** [lib_regexp.so] Error 1 > > I looked around a bit. There are many discussions about clang error ld unknown option, but nothing specific for the ?-soname? option. Simply remove this option is not an option. > > Does anyone know how to fix/workaround this? > > Thanks a lot! > > Jennie > > ------------------------------ > > Message: 2 > Date: Wed, 24 Jan 2018 16:34:38 +0100 > From: Sjoerd Mullender <sjoerd(a)acm.org> > To: users-list(a)monetdb.org > Subject: Re: MonetDB-extend clang error: ld: unknown option: -soname > Message-ID: <78539081-3ab0-3b89-d22d-dbd12af4a6fa(a)acm.org> > Content-Type: text/plain; charset="utf-8" > > On 24/01/18 15:56, Ying Zhang wrote: >> Dear all, >> >> I was trying to compile the regexp example in the MonetDB-extend repo on a Mac OS 10.13, but got an ?unknown option? error: >> >> $ make >> cc -fPIC -DPIC -o lib_regexp.so -shared regexp.o -L/usr/local/opt/openssl/lib -L/Users/jennie/scratch/monet-install/Jul2017/debug/lib -L/usr/local/Cellar/pcre/8.41/lib -lmonetdb5 -lbat -lpcre -Wl,-soname -Wl,lib_regexp.so >> ld: unknown option: -soname >> clang: error: linker command failed with exit code 1 (use -v to see invocation) >> make: *** [lib_regexp.so] Error 1 >> >> I looked around a bit. There are many discussions about clang error ld unknown option, but nothing specific for the ?-soname? option. Simply remove this option is not an option. >> >> Does anyone know how to fix/workaround this? >> >> Thanks a lot! >> >> Jennie > > The question really is, how does one create a dynamically loaded module > on MacOS? The file will be loaded dynamically by mserver, not linked > into a program. > > > -- > Sjoerd Mullender > >

1 0

Re: MonetDB - question on rename table
by Stefan Manegold 25 Jan '18

25 Jan '18

Yep: that feature is not implemented. -------- Original message --------From: "Sharma, Sreejith" <Sreejith.Sharma(a)harman.com> Date: 1/25/18 07:45 (GMT+01:00) To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: MonetDB - question on rename table All, Any reason why we can’t do RENAME of table in MonetDB? Regards, Sreejith

2 1

MonetDB - question on rename table
by Sharma, Sreejith 25 Jan '18

25 Jan '18

All, Any reason why we can't do RENAME of table in MonetDB? Regards, Sreejith

1 0

MonetDB-extend clang error: ld: unknown option: -soname
by Ying Zhang 24 Jan '18

24 Jan '18

Dear all, I was trying to compile the regexp example in the MonetDB-extend repo on a Mac OS 10.13, but got an “unknown option” error: $ make cc -fPIC -DPIC -o lib_regexp.so -shared regexp.o -L/usr/local/opt/openssl/lib -L/Users/jennie/scratch/monet-install/Jul2017/debug/lib -L/usr/local/Cellar/pcre/8.41/lib -lmonetdb5 -lbat -lpcre -Wl,-soname -Wl,lib_regexp.so ld: unknown option: -soname clang: error: linker command failed with exit code 1 (use -v to see invocation) make: *** [lib_regexp.so] Error 1 I looked around a bit. There are many discussions about clang error ld unknown option, but nothing specific for the “-soname” option. Simply remove this option is not an option. Does anyone know how to fix/workaround this? Thanks a lot! Jennie

2 1

Question on MonetDB Source Code Compilation
by Sharma, Sreejith 24 Jan '18

24 Jan '18

All, Here is a quick question. I have compiled "MonetDB v11.27.9 (Jul2017-SP2)" version of source code and while running below sample SQL am finding an issue. Could anyone help me to trace issue? Is this due to the compiler version? Issue Details: While using SUM() or any function without an alias the header datatype returned is showing as HUGEINT for base INTEGER column, where as it should have been BIGINT. This is causing a failure in my BI application as it's having issues in identifying HUGEINT. Below are the details. Source Code Version : MonetDB v11.27.9 (Jul2017-SP2) Linux OS Version : CentOS release 6.3 (Final) C Compile Version : gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) Also, I see below note in MonetDB documentation. Which version of GCC should be appropriate for compilation with __128 bit integer. Note: HUGEINT is only available since the Jul2015 release and on platforms with a C-compiler that supports the __int128 or __int128_t data type (e.g., recent gcc, clang, & icc on Linux or MacOS X) Sample Table DDL : CREATE TABLE INT_VALID (COL_INT INTEGER, COL_BIGINT BIGINT, COL_HUGEINT HUGEINT, COL_DOUBLE DOUBLE, COL_NUMERIC NUMERIC(10)); Compiled Version sql>SELECT SUM(col_int) FROM INT_VALID; mapi_query_part:36:SELECT SUM(col_int) FROM INT_VALID; fetch next block: start at:10093 got next block: length:105 text:&1 4 1 1 1 2452 % DM_POS_TSV_MRT_P6A.L4 # table_name % L3 # name % hugeint # type % 5 # length [ 11110 ] Binary Version sql>SELECT SUM(col_int) FROM INT_VALID; mapi_query_part:36:SELECT SUM(col_int) FROM INT_VALID; fetch next block: start at:1168 got next block: length:105 text:&1 4 1 1 1 13998 % DM_POS_TSV_MRT_P6A.L4 # table_name % L3 # name % bigint # type % 5 # length [ 11110 ] +---------+------------+--------------------------+-------------+ | col_int | col_bigint | col_double | col_numeric | +=========+============+==========================+=============+ | 1111 | 1234567 | 1234.44 | 1111 | | 2222 | 2345678 | 34355.66 | 2222 | | 3333 | 3456789 | 66423.66 | 3333 | | 4444 | 45678912 | 8765.66 | 4444 | +---------+------------+--------------------------+-------------+ Regards, Sreejith

2 3

Memory leak in MonetDB
by Sergii Kliuchnyk 23 Jan '18

23 Jan '18

Hello, my name Sergii. I hope this is right place for this message. In our company we using MonetDB near 1 year and we have only one problem with it - memory leak. We found only one dependence - if we making queries then memory starts growing. I read "Memory footprint" article that MonetDB excessively uses main memory for processing, but I thought it should stop allocate memory after sometime but it's not. And looks like this allocated memory is not gives some performance. What we do now is restarting monet server every month. I found on mserver5 man-page parameter "gdk_vmtrim" but I'm not 100% sure is it what we need and what correct way to apply this parameter. Maybe MonetDB SQL has some function to unload no needed memory? Maybe it's php/nodejs clients (from github.com/MonetDB repo) somehow cause this memory leaks? I will appreciate any help

1 1

BUG: on CREATE TABLE (like)
by Ariel Abadi 18 Jan '18

18 Jan '18

Hi! It seems to be, that there is a BUG on the statement CREATE TABLE.. It is not being consider the *DEFAULT property.* Take a look on the following example: sql>CREATE TABLE test (updated TIMESTAMP(0) DEFAULT current_timestamp()); operation successful (1.433ms) sql>\d test CREATE TABLE "sb_traf"."test" ( "updated" TIMESTAMP(0) DEFAULT current_timestamp() ); sql>CREATE TABLE test2 (LIKE test); operation successful (1.102ms) sql>\d test2 CREATE TABLE "sb_traf"."test2" ( "updated" TIMESTAMP(0) ); May be Im missing something, if thats the case, please let me know! Rgds Ariel

2 2

Embedded Python/NumPy
by Ariel Abadi 18 Jan '18

18 Jan '18

Hi! Someone can help please!?!?!!! I have created a function in python (see attached the definition), every time I try to use it against a "big table" monetdb is hanging out. What I found is that instead of sending the value of each nrm_json (to the function) , monet is sending the full column!!!.. This means that if the table has 10MM rows, then python is receiving ALL the records instead of the nrm_json STRING: SELECT sb_glob.getjsonfield_python(nrm_json, 'conference_id') FROM nrm_20180107 LIMIT 3; CREATE FUNCTION sb_glob.getjsonfield_python(nrm_json STRING, nrm_field STRING) RETURNS STRING LANGUAGE PYTHON { import ast a=ast.literal_eval(str(nrm_json.encode('utf-8'))) if not a.has_key(nrm_field): a[nrm_field] = '' return str(a[nrm_field]) }; I hope you understand, and guide me in solving this problem Thks in advance Ariel

1 0

program contains errors occured
by Amit Ambekar 18 Jan '18

18 Jan '18

Hi, I have big table around 15 mln records. getting error as "program contains errors" when i tried to fetch data using select query

2 1

Python Functions
by Ariel Abadi 11 Jan '18

11 Jan '18

Hi! Maybe someone can help me. Im trying to connect to a MySQL table to get some data and use it within MonetDB. Eveything works fine, TILL... the query Im running on MySQL is bringing NO DATA. As an example I put it as LIMIT 0 so I dont receive any row. Eventhough I have considered that situation and have defined the array with null values, Im still getting an error. *Could not create a Numpy array from the return type.* Does anybody has an idea how to solve this issue? Thks in advance! Ariel ----------------------------------------------------- DROP FUNCTION connect_to_mysql() ; CREATE FUNCTION connect_to_mysql() RETURNS TABLE(agr_id integer, agr_name string ) LANGUAGE PYTHON { import datetime import mysql.connector cnx = mysql.connector.connect(user='XXXX', password='XXXX',host='192.168.15.5',database='XXXXX',use_pure=False) cursor = cnx.cursor() query = "SELECT agr_id, agr_name FROM AGR_Agreement LIMIT 0 " cursor.execute(query) allrows = cursor.fetchall() ncols = len(cursor.description) colnames = [cursor.description[i][0] for i in range(ncols)] result = dict() for col in range(ncols): if len(allrows) == 0: result[colnames[col]] = [None] for row in range(len(allrows)): if allrows[row][col] == None: valor = None else: valor = str(allrows[row][col]) try: result[colnames[col]].append(valor) except KeyError: result[colnames[col]] = [valor] cursor.close() cnx.close() return result }; sql>SELECT * FROM connect_to_mysql() ; Could not create a Numpy array from the return type.

1 0

DEC2016-SP2 and BINARY bulk load
by Lynn Carol Johnson 11 Jan '18

11 Jan '18

Hi all – I have always used the COPY BINARY INTO … commands to load my 2.0 Billion row genetic data into a monetdb table. With 135 columns, it has been blindingly fast. Last week I moved from the June2016-SP2 release to dec2016-SP2. My binary loads are taking WAY longer. I killed one after 3 hours (via “call sys.stop(pid)” so it could clean up properly). I then started the load again, thinking perhaps the problem was related to the new columns I was adding. I have since dropped the table and remade it using the same data and scripts that worked in just over 3 minutes in February on the jun2016-SP2 load. It is really chugging along – I’m up to 30 minutes and counting. I don’t have access to the sql log files, but the Merovingian.log shows nothing. I do notice that previously the binary files, once loaded, were removed from the loading directly. This does not happen now. Were these files previously “moved” and now they are copied? Has anyone see this performance issue with Dec2016-SP2 COPY BINARY INTO …. Commands? Thanks - Lynn

5 17

RE: MonetDB - Column Header Question
by Sharma, Sreejith 09 Jan '18

09 Jan '18

I see below issue is happening only if base column datatype is integer. All other types working fine. From: Sharma, Sreejith Sent: Saturday, January 06, 2018 8:24 PM To: users-list(a)monetdb.org Subject: MonetDB - Column Header Question All, Quick question - For a specific scenario I would like to understand below - If we don't specify column header in SQL or alias, how is MonetDB picks up the column header? And what is the datatype for it? I saw it's hge (scalar type) is it so? Also, I would like to know where in source code it's happening. Example: SELECT SUM(COLUMN_NAME) FROM TABLE_NAME; ------------------+ | L3 | ------------------+ |123456678 | ------------------+ Regards, Sreejith

2 3

Bulk data upload through JDBC
by suvendu chakra 09 Jan '18

09 Jan '18

Hello, I am following SQLCopInto.java to upload bulk data into MonetDB but seems to be not working. I am even using mclient command to upload data but the command expects the input file to be accessible by the DB server, is there any alternative to this. Please suggest a best way to do through JDBC Thanks

3 3

MonetDB - Column Header Question
by Sharma, Sreejith 06 Jan '18

06 Jan '18

All, Quick question - For a specific scenario I would like to understand below - If we don't specify column header in SQL or alias, how is MonetDB picks up the column header? And what is the datatype for it? I saw it's hge (scalar type) is it so? Also, I would like to know where in source code it's happening. Example: SELECT SUM(COLUMN_NAME) FROM TABLE_NAME; ------------------+ | L3 | ------------------+ |123456678 | ------------------+ Regards, Sreejith

1 0

SyntaxException:parseError:Xclose
by Dennis Fassl 06 Jan '18

06 Jan '18

Hi all, I am generating MAL queries in Java and I am using the JDBC interface to execute them. This works fine for small tables. But if the resultSet has more than 100 entries MonetDB is printing the first 100 rows and then gives me the following error: ... [ 19, "undergraduateDegreeFrom", 1509 ] [ 19, "memberOf", 613 ] [ 19, "takesCourse", 556 ] [ 19, "advisor", 600 ] [ 20, "undergraduateDegreeFrom", 1493 ] [ 20, "memberOf", 613 ] [ 20, "takesCourse", 529 ] [ 20, "takesCourse", 578 ] [ 20, "takesCourse", 557 ] [ 20, "advisor", 1315 ] MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = sql.resultSet(table_names, table_attrs, table_type, table_lens, table_scales, start_id, type, end_id); ERROR = !SyntaxException:parseError:Xclose 0 !SyntaxException:parseError: ^';' expected Could you help me to avoid this error? This is the full query: sql.init(); tbd := sql.mvc(); start_id:bat[:int] := sql.bind(tbd,"sys","relations","start_id",0:int); type:bat[:str] := sql.bind(tbd,"sys","relations","type",0:int); end_id:bat[:int] := sql.bind(tbd,"sys","relations","end_id",0:int); table_names := bat.new(nil:str); table_names := bat.append(table_names, "start_id"); table_names := bat.append(table_names, "type"); table_names := bat.append(table_names, "end_id"); table_attrs := bat.new(nil:str); table_attrs := bat.append(table_attrs, "str"); table_attrs := bat.append(table_attrs, "str"); table_attrs := bat.append(table_attrs, "str"); table_type := bat.new(:str); table_type := bat.append(table_type, "int"); table_type := bat.append(table_type, "varchar"); table_type := bat.append(table_type, "int"); table_lens := bat.new(nil:int); table_lens := bat.append(table_lens, 1); table_lens := bat.append(table_lens, 1); table_lens := bat.append(table_lens, 1); table_scales := bat.new(nil:int); table_scales := bat.append(table_scales, 0); table_scales := bat.append(table_scales, 0); table_scales := bat.append(table_scales, 0); sql.resultSet(table_names, table_attrs, table_type, table_lens, table_scales, start_id, type, end_id); I am using "MonetDB Database Server v1.7 (Dec2016-SP5)" on Ubuntu 14.04. What am I trying to do: The goal of the project is to use Cypher as an input language for MonetDB. I have a Cypher optimizer written in Java. The output of this optimizer should be used as input for MonetDB. In the end we want to compare the the runtime of a query in MonetDB and in Neo4J. Thanks and regards, Dennis

1 1

Re: MonetDB header Datatype Question
by Stefan Manegold 04 Jan '18

04 Jan '18

What's the (SQL-)type of your column COLUMN_NAME?Which version of MonetDB are you using?Which version of the MonetDB ODBC driver are you using?Which compiler did you use to compile MonetDB on which OS?What did configure say about what the largest supported integer type is? -------- Original message --------From: "Sharma, Sreejith" <Sreejith.Sharma(a)harman.com> Date: 12/31/17 18:33 (GMT+01:00) To: users-list(a)monetdb.org Subject: RE: MonetDB header Datatype Question All, Here is the specifics of error – typeODBC: SQL_UNKNOWN_TYPE mDataType: HG_NULL_VALUE(hgColumnType) precision: 0 scale: 0 nullable: true Column name: L6. Regards, Sreejith From: Sharma, Sreejith Sent: Sunday, December 31, 2017 11:21 AM To: users-list(a)monetdb.org Subject: MonetDB header Datatype Question All, I would like to know what is the MonetDB header datatype if no alias or column name provided? In my BI application, am getting an error while using SELECT SUM(DISTINCT COLUMN_NAME) FROM TABLE_NAME. I see it works in 11.27.9 build when installed in CentOS. However, when i download the source tarball and recompile the code, this fox seems to be lost. Is there anywhere this can be identified? Or fixed? Regards, Sreejith

3 5

Re: MonetDB header Datatype Question
by Stefan Manegold 04 Jan '18

04 Jan '18

Looks fine to me, but of course you're free to try a different compiler of your choice. Stefan -------- Original message --------From: "Sharma, Sreejith" <Sreejith.Sharma(a)harman.com> Date: 1/4/18 10:31 (GMT+01:00) To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: RE: MonetDB header Datatype Question Thanks Stefan. However am suspecting if it's to do with the compiler. Did you got a chance to look at the details I sent across for the questions. configure:5039: gcc --version >&5 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16) Regards, Sreejith -----Original Message----- From: users-list [mailto:users-list-bounces+sreejith.sharma=harman.com@monetdb.org] On Behalf Of Stefan Manegold Sent: Wednesday, January 03, 2018 2:11 PM To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: [EXTERNAL] Re: MonetDB header Datatype Question I was mislead by the "HG" prefix in the error message. I now realize that the error message does not seem to be a MonetDB error message. Hence, I'm afraid we cannot help much. You said is works/worked with a binary installation of Jul2017-SP2 (11.27.9), but does not with a self-compiled installation of the very same release. You'd need to double-check you compilation and installation of the self-compiled release. You can also try with the latest Jul2017-SP3 (11.27.11) release --- binary and/or self-compiled --- but as said, since the error message is not from MonetDB, you'd need to search for the problem (also) in the client software / application you're using. Best, Stefan ----- On Jan 1, 2018, at 5:55 PM, Sharma, Sreejith Sreejith.Sharma(a)harman.com wrote: > Hi Stefan, > > > > Here are the details you had asked for. Any help here soon is appreciated. > > > > What's the (SQL-)type of your column COLUMN_NAME? – > > INTEGER > > > > Which version of MonetDB are you using? – > > MonetDB 5 server v11.27.9 "Jul2017-SP2" (64-bit, 128-bit integers) > > > > Which version of the MonetDB ODBC driver are you using? > > ODBC Driver - /usr/lib64/libMonetODBC.so (11.27.9 and tried 11.27.5 > updated) > > > > Which compiler did you use to compile MonetDB on which OS? > > configure:5030: checking for C compiler version > > configure:5039: gcc --version >&5 > > gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16) > > > > #define PACKAGE_NAME "MonetDB" > >| #define PACKAGE_TARNAME "MonetDB" > >| #define PACKAGE_VERSION "11.27.9" > >| #define PACKAGE_STRING "MonetDB 11.27.9" > >| #define PACKAGE_BUGREPORT "https://clicktime.symantec.com/a/1/js6skxjfSiwdIJPQwPy1ginHSudH8-MwrUFzw00H…" > >| #define PACKAGE_URL "https://clicktime.symantec.com/a/1/ZNQS7LXKm8wqT3QEExvIAnZ3Vk2CRq8bspHohtAV…" > >| #define PACKAGE "MonetDB" > >| #define VERSION "11.27.9" > >| #define HOST "x86_64-pc-linux-gnu" > >| #define MONETDB_RELEASE "Jul2017-SP2" > > > > What did configure say about what the largest supported integer type is? > > hge_bits='128' > > largest integer type: ${hge_bits} bits" > > > > > > Regards, > > Sreejith > > > > > From: users-list > [mailto:users-list-bounces+sreejith.sharma=harman.com@monetdb.org] On > Behalf Of Stefan Manegold > Sent: Monday, January 01, 2018 7:05 AM > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: [EXTERNAL] Re: MonetDB header Datatype Question > > > > > > > > > What's the (SQL-)type of your column COLUMN_NAME? > > > Which version of MonetDB are you using? > > > Which version of the MonetDB ODBC driver are you using? > > > Which compiler did you use to compile MonetDB on which OS? > > > What did configure say about what the largest supported integer type is? > > > > > > > > > -------- Original message -------- > > > From: "Sharma, Sreejith" < Sreejith.Sharma(a)harman.com > > > > Date: 12/31/17 18:33 (GMT+01:00) > > > To: users-list(a)monetdb.org > > > Subject: RE: MonetDB header Datatype Question > > > > > > All, > > > > Here is the specifics of error – > > > > typeODBC: SQL_UNKNOWN_TYPE mDataType: HG_NULL_VALUE(hgColumnType) > precision: 0 > scale: 0 nullable: true Column name: L6. > > > > Regards, > > Sreejith > > > > > From: Sharma, Sreejith > Sent: Sunday, December 31, 2017 11:21 AM > To: users-list(a)monetdb.org > Subject: MonetDB header Datatype Question > > > > > All, > > > > > > I would like to know what is the MonetDB header datatype if no alias > or column name provided? In my BI application, am getting an error > while using SELECT SUM(DISTINCT COLUMN_NAME) FROM TABLE_NAME. I see it > works in 11.27.9 build when installed in CentOS. However, when i > download the source tarball and recompile the code, this fox seems to be lost. Is there anywhere this can be identified? > Or fixed? > > > Regards, > > > Sreejith > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://clicktime.symantec.com/a/1/EiR5ebgK8iUa4CLKJS5IhZJEZAKRFGZRDFU > FUCIYntY=?d=l99sUyvGli3efslX3N2ZiOXRT2oUDQGPF4w16I0hM9zUMoOkJQMri6AAcO > e4XlOqB4NFs5YhzH2NIIET0LyAMDvzRQkCYnLv2cWSJsLXuGRkmOTx5BxDA1nwzncmE0Ex > YksGCv1OTBY5h8YekFgUFxCvagkJjjUM6qSdc8J_3jJdMcNfFBT-6B5sCtZZWTBqGFRvVr > mq4XFYO_7DYsI9LdIJf7p8tqhBL-xSVWB3pCxqFx46Iyjz8nidH-BXBjg-pRI2Gk56GQ4p > kiPFSFlud_PVfn48XxozGVFit5p6x7F-AVEgKrooyJUyAchXTTYPKW5qN60JUu-5toDlGZ > U0tcE6B8a_-BAZmcVxJiZTehQx5mL1hAm0sBK2BdVeLD5BkXQ_JG8gnBigTg0N4E9I6b5C > geHSGmxUBIKMr_4WO21MsBtAqElF9oRpXg%3D%3D&u=https%3A%2F%2Fwww.monetdb.o > rg%2Fmailman%2Flistinfo%2Fusers-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org https://clicktime.symantec.com/a/1/EiR5ebgK8iUa4CLKJS5IhZJEZAKRFGZRDFUFUCIY… _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

1 0

MonetDB header Datatype Question
by Sharma, Sreejith 31 Dec '17

31 Dec '17

All, I would like to know what is the MonetDB header datatype if no alias or column name provided? In my BI application, am getting an error while using SELECT SUM(DISTINCT COLUMN_NAME) FROM TABLE_NAME. I see it works in 11.27.9 build when installed in CentOS. However, when i download the source tarball and recompile the code, this fox seems to be lost. Is there anywhere this can be identified? Or fixed? Regards, Sreejith

1 1

Monetdb date column issue
by Amit Ambekar 27 Dec '17

27 Dec '17

Hi, i am using MonetDB v11.23.7 (Jun2016-SP1) version. in that when is reboot my ubuntu machine, my date columns in all tables becoming null . is there any issue related to version? Warm Regards, *Amit Suryakant Ambekar*** */Technology Associate/* Office: Ist floor, Poddar Chambers, Mathruradas Mill Compound, Lower Parel, Mumbai 400013 | Landline: +91 22 43470408 Cell: +91 7741823310 | Email: amit(a)g-square.in | Website:www.g-square.in

1 0

SELECT: too many nested operators
by Sharma, Sreejith 19 Dec '17

19 Dec '17

All, I have a valid scenario where our application generates a SQL which is around 4.4MB and contains 75K of combination of AND and OR operators. This SQL fails with "SELECT: too many nested operators" error while executing this SQL. While I went through the code, I noticed any thread size above THREAD_STACK_SIZE - 16 * 1024 is getting aborted. Is there any possibility that we can change this? Any reason why it's set to 16MB? As server our server has enough memory and storage and we need to understand if this can be handled in code or via parameter as this is valid scenario. Also, we can't split the SQL too. Regards, Sreejith

4 4

too many nested operators - Query size 4MB
by Sharma, Sreejith 16 Dec '17

16 Dec '17

All, Have a quick question- I have been trying to perform a test on bigger SQL and I see SQL's are failing which are above 4 MB with error "too many nested operators". There are many AND and OR operators. Not sure if there is any limitations. Regards, Sreejith

1 0

Announcement: New Jul2017-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Dec '17

14 Dec '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP3 bugfix release (11.27.11) MonetDB Common * Reimplemented summing of a column of floating point (flt and dbl) values. The old code could give wildly inaccurate results when adding up lots and lots of values due to lack of precision. Try SELECT sum(c) FROM t; where t is 100,000,000 rows, c is of type REAL and all values are equal to 1.1. (The old code returned 33554432 instead of 1.1e8.) Bug Fixes * 3898: Deadlock on insertion * 6429: ROUND produces wrong data type * 6436: Query sequence with 2x ifthenelse() and next nullif() causes mserver5 Segmentation fault * 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped. * 6439: Invalid references to sys.columns.id from sys.statistics.column_id * 6442: SEGFAULT with COPY INTO BEST EFFORT and skipping input columns * 6443: complex(?) query forgets(?) column name * 6444: Using 'with' keyword with table returning function crashes monetdb * 6445: Sqlitelogictest crash in MySQL query * 6446: sql_parser.y bug? * 6448: 'insert into' with multiple rows containing subqueries crashes * 6449: Assertion error in rel_dce_refs (sqlsmith) * 6450: Assertion error in exp_bin (sqlsmith) * 6451: Assertion error in sql_ref_dec (sqlsmith) * 6453: Assertion error in rel_rename_exps (sqlsmith) * 6454: SQL lexical error * 6455: Assertion error in rel_apply_rewrite (sqlsmith) * 6456: NULL becomes 0 in outer join * 6459: Assertion error in exp_bin (sqlsmith) * 6462: large virtual memory spike on BLOB column select * 6465: appending to variables sized atom bats other than str bats with force flag may result in corrupted heap * 6467: date_to_str formatter is wrong * 6470: mitosis gets in the way of simple select * 6471: calls to sys.generate_series should auto-convert arguments * 6472: Assertion failure in rel_rename (Sqlsmith) * 6477: assertion eror rel_push_project_up (sqlsmith) * 6478: Crash with nested order by/ limit offset * 6479: Mserver receives an assertion error on a procedure call * 6480: Segfault in mvc_find_subexp (sqlsmith)

1 0

Excessive slowness in query
by Alfio Bettini 14 Dec '17

14 Dec '17

Hi all, we're testing MonetDB performances in order to understand if it could be possible to use it as data mart in our BI software. We're considering a set of queries executed in our test environment (CentOS 7, 32 GB RAM, 4 cpu). All the executed queries have quite good performances (between 10 and 45 seconds) except the following one that is taking several minutes in order to be executed: SELECT id, count(distinct value) FROM table GROUP BY id; The table contains 100 millions of data. It has about 130 columns (varchar, int, bigint, smallint, tinyint, timestamp, datetime). We have no BLOB/CLOB/Text types. Can you suggest us any optimizations in order to have this query run faster? Do you know some possible reasons why it's so slow? Kind Regards, Alfio Bettini The information contained in this message, together with any attachments, may be privileged and confidential and is intended for the use of the addressee(s) only. Any review, copying, use or distribution of this email by others (including any of its attachments) is strictly prohibited. If you are not the intended recipient, please destroy this email and its attachments and contact the sender immediately. This email was sent by EidosMedia SpA<http://www.eidosmedia.com>

1 0