- announce-list - monetdb.org

Announcement: New Oct2020-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 24 Nov '20

24 Nov '20

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct 2020-SP1 bugfix release (11.39.7) Bug Fixes * 6890: Add support of xz/lzma (de)compression on MS Windows * 6891: Add support of lz4 (de)compression on MS Windows * 6971: Parsing table returning function on remote server fails * 6981: Oct2020: PREPARE DDL statement silently fails * 6983: monetdb allows to use non-existing optimizer pipe * 6998: MAL profiler buffer limitations * 7001: crossproduct generated for a simple (semi-)join * 7003: Segfault on large chain of constant decimal multiplication * 7005: Dropping a STREAM TABLE does not remove the associated column info from sys._columns * 7010: deallocate <id> results in all prepared statements being deallocated (not error-related) * 7011: uuid() called only once when used in projection list

1 0

Announcement: New Oct2020 Feature release of MonetDB suite
by Sjoerd Mullender 19 Oct '20

19 Oct '20

The MonetDB team at MonetDB BV is pleased to announce the Oct2020 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct 2020 feature release (11.39.5) MonetDB5 Server * The settings for specifying how mserver5 should listen to "The Internet" have been overhauled. See the manual for details. In brief, mapi_autosense, mapi_ipv6 and mapi_open are gone. If mapi_listenaddr equals "localhost" or "all", we listen to both IPv4 and IPv6 (if available), if "127.0.0.1" or "0.0.0.0", we listen to IPv4 only, if "::1" or "::" we listen to IPv6 only. The first of each pair is loopback interface only, the second is all interfaces. If mapi_listenaddr is "none", then no IP port is opened, you need to use a UNIX domain socket. If mapi_port is 0, we let the operating system choose a free port (like mapi_autosense). Default behavior has not changed. Whole System * Finished a first version of the new monitoring function user_statistics(), which is only intended for the DBAs. For each database user who has logged in during the current mserver5 session, it returns "username": login name of the database user, "querycount": the number of queries this user has executed since his/her first login, "totalticks": the total execution time (in microsecond) of the queries ran by this user, "maxquery": the query with the longest execution time (if two queries have the same execution time, the newer overwrites the older), "maxticks": the execution time of the 'maxquery' (in microsecond), "started": the start timestamp of the 'maxquery', "finished": the finish timestamp of the 'maxquery'. * Removed support for LiDAR data, that is the SQL procedures sys.lidarattach, sys.lidarload, and sys.lidarexport. * The sys.queue() has been turned into a circular buffer to allow for inspection of both active, paused and recently executed queries. Client Package * mclient and msqldump now also look in $XDG_CONFIG_HOME for the monetdb configuration file. SQL Frontend * Extended the system monitor with a table-returning function user_statistics() which keeps some statistics for each SQL user, e.g. the user's query count, total time spent, and maximal query seen. * Removed '@' syntax used to refer into a variable in a query. It was a non-standard method, which was replaced by a schema addition to variables. Variables in the global scope now have schema. All default global variables are set under schema "sys". However variables inside PSM don't have a schema, because there are no transaction semantics inside PSM at the moment. * Removed declared variables and tables from the global scope. They were transaction agnostic and incompatible with the SQL standard, i.e. they are valid exclusively under PSM (e.g. functions, procedures and triggers). * Scoping semantics were added for both variables and tables. Variables with the same name at a query are now resolved under the following precedence rules: 1. Tables, Views and CTEs at the FROM clause. 2. Variable declared in the body of function/procedure, i.e. local variable. 3. Function/procedure parameter. 4. Variable from the global scope. Tables with the same name now have the following precedence rules at a SQL query: 1. Table declared in the body of function/procedure, ie local table. 2. Temporary table. 3. Table from the current session schema. This means the query: SELECT * FROM "keys"; will list keys from temporary tables instead of persisted ones, because "keys" table is available for both "sys" and "tmp" schemas. * The table returning function "var" was extended with more details about globally declared variables, namely their schema, type and current value. * Removed obsolete sys.getContent(url) function. * Removed obsolete json.output(json) function. * Removed compatibility between interval types and other numeric types in favor for a more strict SQL standard compliance. This means operations between temporal types and other numeric types such as INT and DECIMAL are no longer possible, instead use interval types. e.g. SELECT date '2020-01-01' + 1; now gives the error. Instead do: SELECT date '2020-01-01' + interval '1' day; if 1 was meant to be a day interval. Setting an interval variable such as the session's current timezone with a number e.g. SET current_timezone = 1; is no longer possible. Instead do SET current_timezone = interval '1' hour; Casting between interval and other numeric types is no longer possible as well, because they are not compatible. * Because of incompatibilities this change may create, if a user intends to convert a numeric value to an interval, the multiplication function can be used in the form: <numeric value> * interval '1' <interval length> e.g. 10 * interval '1' second = interval '10' second. As for the other way around, the 'EPOCH' option was added to the extract syntax. This option returns the number of milliseconds since the UNIX epoch 1970-01-01 00:00:00 UTC for date, timestamp and time values (it can be negative). Meanwhile, for day and second intervals, it returns the total number of milliseconds in the interval. As a side note, the 'EPOCH' option is not available for month intervals, because this conversion is not transparent for this type. * Made general logarithm function log(x,base) compliant with the SQL standard, by swapping the input parameters. Instead of log(x,base), now is log(base,x). Bug Fixes * 3553: All schema access to ubiquitous functions * 3815: Incorrect results when expression contains implicit float/integer conversions * 6415: Date arithmetic types are inconsistent * 6814: provide native implementations for scalar functions sys.degrees(rad) and sys.radians(deg) * 6843: function sys.getcontent(url) always returns "Feature not supported" * 6857: remove not implemented aggregate function json.output(js json) * 6870: Missing bulk operators * 6878: SQL Connection Error when running SELECT queries containing AND command * 6910: SQLancer query: 'bat.append' undefined * 6930: SQLancer crash on join with coalesce * 6931: Allow EDITOR to be used for the current command in mclient * 6935: Wrong result when dividing interval by literal float * 6937: Lost the microsecond precisions * 6938: Segmentation fault in MalOptimizer * 6939: Error in optimizer multiplex when selecting profiler.getlimit() or wlc.clock() or wlc.tick() or wlr.clock() or wlr.tick() * 6941: SELECT queries on remote table fail when using LIKE in WHERE conditions * 6943: JSON parser is too permissive * 6948: msqldump with Empty BLOBs cannot be imported * 6949: Loosing timing precision * 6950: redundant/replicated code line in gdk/gdk_hash.c * 6951: Use a different naming scheme for MAL blocks * 6954: FILTER functions no longer find their implementation * 6955: ROUND(DECIMAL, PRECISION) gives incorrect result with non-scalar precision parameter * 6960: implementation of log(arg1,arg2) function is not compliant with the SQL standard, arguments are switched * 6962: "SELECT * FROM ids LIMIT 1" produces: exp_bin: !ERROR: Could not find %173.id * 6964: Table returning function: Cannot access column descriptor * 6965: Crash when using distinct on the result of a table returning function * 6974: Oct2020-branch cannot attach and load FITS files * 6976: Oct2020: default dbfarm cannot be started * 6978: Oct2020: \d shows empty result in schema created by include sql script * 6979: timestamp add integer * 6980: Oct2020: wrong mel definition for str.epilogue

1 0

Announcement: New Jun2020-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Jul '20

27 Jul '20

The MonetDB team at MonetDB BV is pleased to announce the Jun2020-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jun 2020-SP1 bugfix release (11.37.11) MonetDB Common * Hash buckets come in variable widths. But if a BAT grows long enough so that the BAT indexes that are stored in the buckets don't fit anymore, the buckets need to be widened. This is now fixed. Bug Fixes * 6844: sys.getUser('https://me:pw@www.monetdb.org/Doc') does not return the user: me * 6845: the url sys.get...(url) functions do not allow null as a parameter * 6858: json.keyarray(json '{ "":0 }') fails with error: Could not allocate space * 6859: only first character of the separator string in json.text(js json, sep string) is used * 6873: sys.hot_snapshot() creates incomplete snapshots if the write-ahead log is very large * 6876: tar files created by sys.hot_snapshot() produce warnings on some implementations of tar * 6877: MonetDB produces malformed LZ4 files * 6880: Left fuzzy queries are much slower than other fuzzy queries. * 6882: cgroups limits no longer respected? * 6883: SQLancer crash on delete query * 6884: SQLancer generates query with unclear error message * 6885: SQLancer causes assertion error on UTF8_strlen * 6886: SQLancer alter table add unique gives strange error message * 6887: SQLancer crash on complex query * 6888: SQLancer crash on cross join on view * 6889: SQLancer crash on long query * 6892: SQLancer crash on query with HAVING * 6893: SQLancer inner join reporting GDK error * 6894: SQLancer crash on rtrim function * 6895: SQLancer causing 'algebra.select' undefined error * 6896: SQLancer algebra.select' undefined 2 * 6897: SQLancer distinct aggregate with error on group by constant * 6898: SQLancer crash on join query * 6899: SQLancer TLP query with wrong results * 6900: SQLancer generated SIGFPE * 6901: SQLancer TLP query with wrong results 2 * 6902: SQLancer query: batcalc.between undefined * 6903: SQLancer calc.abs undefined * 6904: SQLancer aggr.subavg undefined * 6905: SQLancer TLP query with wrong results 3 * 6906: SQLancer crash on complex join * 6907: SQLancer algebra.select undefined * 6908: SQLancer inputs not the same size * 6909: SQLancer query with wrong results * 6911: SQLancer query: 'calc.bit' undefined * 6917: Decimal parsing fails * 6918: SQLancer query compilation error * 6919: SQLancer insert function doesn't handle utf-8 strings * 6920: SQLancer project_bte: does not match always * 6922: Timestamp columns not migrated to new format * 6923: Imprints data files for timestamp BAT not migrated to the new format * 6924: SQLancer query copy on unique pair of columns fails and complex query with GDK error * 6925: Count string rows in union of string tables leaks (RSS) memory * 6926: SQLancer query with wrong results * 6927: SQLancer inputs not the same size * 6928: SQLancer crash on coalesce * 6929: SQLancer calc.date undefined * 6932: Syntax error while parsing JSON numbers with exponent * 6934: sys.isauuid() returns wrong answer for some invalid uuid strings

1 0

Announcement: New Jun2020 Feature release of MonetDB suite
by Sjoerd Mullender 03 Jun '20

03 Jun '20

The MonetDB team at MonetDB BV is pleased to announce the Jun2020 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jun 2020 feature release (11.37.7) MonetDB5 Server * The mserver5 option --verbose (-v) was removed. A similar effect can be had by issuing the query CALL logging.setcomplevel('SQL_TRANS', 'INFO'); as the monetdb user. * The functions algebra.intersect, algebra.semijoin, and algebra.crossproduct have an extra argument, bool max_one, which indicates that there must be no more than one match in the join. * The example module opt_sql_append is not installed in the binary packages anymore. * Added mserver5 option (--dbtrace=<path>) in order to be able to specify the output file any produced traces. * Removed function bat.setKey(). * algebra.rangejoin now has two extra arguments: anti:bit and symmetric:bit. * Added session identifier, number of workers and memory claim to the sysmon queue. * The worker (number of threads), memory (in MB) and optimizer pipeline limits can now be set per user session basis. The query and session timeouts are now set in seconds. * With required privileges an user can set resource limits for a session. * There are now versions of group.(sub)group(done) that produce a single output containing just the groups. * algebra.join and algebra.leftjoin now have forms which return a single column. The column that is returned is the left column of the two column version. * The MAL profiler now assigns the SQL TRACE output to the client record thereby avoiding the interaction with other queries, but loosing insight of competing queries. The stethoscope should be used for that. SELinux Support * There was a problem with the MonetDB SELinux support on Fedora 32. That is fixed in this release. In order to do a proper upgrade of the package if you have already installed MonetDB-selinux on Fedora 32, you may need to uninstall (dnf remove) the old package and then install the new. Whole System * A new system to deal with debug output has been implemented. There is now an option --dbtrace to mserver5 that takes a file argument to which debug output is written. The default value is the file mdbtrace.log inside the database directory. This option can also be set through the monetdb program. * The home directory of the automatically created monetdb user was changed from /var/MonetDB to /var/lib/monetdb (RPM based systems only). This home directory is (currently) not used for anything, though. * Python 2 support has been removed. There is now only support for using Python 3. * Removed support for bam and sam files. * Added mserver5 option (--set raw_strings=true|false) and monetdb database property (raw_strings=yes|no) to control interpretation of strings. Merovingian * On Fedora and RHEL systems (not RHEL 6), if monetdbd runs under systemd, when the package is updated, monetdbd (and hence any mserver5 process it runs) is restarted. * Added dbtrace mserver5 option to the daemon in order to set mserver5's output directory for the produced traces. Client Package * The monetdb-client-tools (Debian/Ubuntu) and MonetDB-client-tools (Fedora/RH) containing the stethoscope, tachograph, and tomograph has been removed. A completely new version of stethoscope will be released to replace the old version. * Removed the possibility of using the MD5 checksum for authentication purposes. It was never actively used but was there as an option. Now the option has been removed. * Add port and host as fields in the .monetdb file. * Added 'sessionid' column to system function sys.queue(), so each query gets tagged with the current session identifier * Allow monetdb user to control session and query time out and selectively stopping a client sessions with a soft termination request. ODBC Driver * The NUMERIC and FLOAT types are now handled fully. Before only DECIMAL, FLOAT, and DOUBLE were handled fully. * Some bugs were fixed in the passing back and forth between application and server of values of type GUID (UUID). MonetDB Common * The functions BATintersect, BATsemijoin, and BATsubcross have an extra argument, bool max_one, which indicates that there must be no more than one match in the join. * The "unique" property on BATs was removed. The property indicated that all values in a BAT *had* to be distinct, but this was not actually used. * A new type of candidate list has been introduced. Candidate lists are used internally to specify which rows of a column participate in an operation. Before, candidate lists always contained a list of candidate row IDs. The new candidate list type specifies a list of row IDs that should NOT be considered (negative candidates). * The maximum number of BATs in the system has been increased for 64 bit architectures. * The hash tables used internally by the system now uses a technique based on Linear Hashing which allows them to grow gracefully. This means that hash tables aren't removed and recreated nearly as often anymore. This also meant that the hash table had to be split into two files, which means that after an upgrade the hash tables have to be recreated. * Implemented a version of BATproject, called BATproject2, with two "right" arguments which conceptually follow each other. * Removed MT_mmap and MT_munmap from the list of exported functions. Use GDKmmap and GDKmunmap with the same parameters instead. * Changed the interface of the atom "fix" and "unfix" functions. They now return a value of type gdk_return to indicate success/failure. * Removed the tunique property. The tunique property indicated that all values in the column had to be distinct. It was removed because it wasn't used. * BATrangeselect now has two extra arguments: anti and symmetric (both bool). SQL Frontend * Updating the value of a sequence now requires privilege on its own schema. * The sys.querylog_enable(threshold integer) now actually enables the querylog and uses a threshold in milliseconds. * Removed UNION JOIN statements. They were dropped by the SQL:2003 standard, plus MonetDB implementation was not fully compliant. * The OFFSET value in the COPY INTO query now counts uninterpreted newlines. Before it counted "unquoted record separators" which meant that if you had a single quote on a line that you want to skip, you could not use the feature. * Added support for FROM RANGE MINVALUE TO RANGE MAXVALUE and FROM RANGE MINVALUE TO RANGE MAXVALUE WITH NULL VALUES cases in partitioned tables by range (before they weren't). * Removed support for Python 2. Python 2 itself is no longer supported. Use Python 3 instead. Functions that were declared as LANGUAGE PYTHON2 or LANGUAGE PYTHON2_MAP are changed to LANGUAGE PYTHON and LANGUAGE PYTHON_MAP respectively (without changing the Python code). * Implemented 'covar_pop' and 'covar_samp' aggregate functions, as well as their window function counterparts. Implemented 'stddev_samp', 'stddev_pop', 'var_samp', 'var_pop', 'corr' and 'group_concat' window function correspondents. * Extended SQL catalog with CREATE WINDOW syntax for user-defined SQL:2003 window functions. At the moment, window functions must be defined on the backend engine, i.e. on this case MAL. In the current implementation, the backend code generation creates two additional columns of type lng with the start and end offsets for each row. * Added prepared_statements_args view, which details the arguments for the prepared statements created in the current session. * Added sys.prepared_statements view, which lists the available prepared statements in the current session. * Added deallocate statements with the syntax 'DEALLOCATE [PREPARE] { number | ALL }', to close an existing prepared statement or all, through the SQL layer. Previously this feature was available via MAPI exclusively with the "release" command. * Added support for raw strings using the syntax r'' or R''. This means that C-like escapes will remain uninterpreted within those strings. For instance SELECT r'\"' returns a string of length two. The user needs to escape single quotes by doubling them: SELECT r''''. * Implemented ROLLUP, CUBE and GROUPING SETS from SQL:1999. They define grouping subsets used with the GROUP BY clause in order to compute partial groupings. Also, the GROUPING aggregate was added. This aggregate is a bitmask identifying the grouping columns not present in the generated grouping row when used with the operators described above. * Updated user session procedures by adding the possibility to set properties based on a session identifier. Optimizer pipeline: sys.setoptimizer(int, string) Number of worker threads: sys.setworkerlimit(int, int) Memory limits (in MB): sys.setmemorylimit(int, int) Query timeout (in ms): sys.setquerytimeout(int, int) Session timeout (in ms): sys.setsessiontimeout(int, int) The first argument corresponds to the id of the session to modify, and these procedures are bound to the monetdb user exclusively. The versions of the mentioned procedures with just the second argument were added as well, where the changes are reflected in the current user session, and therefore every user can call them. * The procedures sys.settimeout(bigint), sys.settimeout(bigint,bigint) and sys.session(bigint) are now deprecated. Instead use sys.setquerytimeout and sys.setsessiontimeout mentioned above. * Added SQL procedure sys.hot_snapshot() which can be used to write a snapshot of the database to a tar file. For example, sys.hot_snapshot('/tmp/snapshot.tar'). If compression support is compiled in, snapshots can also be compressed ('/tmp/snapshot.tar.gz'). The tar file expands to a single directory with the same name as the database that was snapshotted. This directory can be passed directly as the --dbpath argument of mserver5 or it can be copied into an existing dbfarm and started from monetdbd. Bug Fixes * 6298: unexpectedly slow execution of SELECT length(fieldname) FROM tablename LIMIT 1 queries * 6401: Suspected memory leak in mserver5 when creating/dropping tables * 6687: Count distinct very slow and use too much the hard drive * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6731: Add system view to allow querying of available prepared statements and their parameters * 6732: Add SQL command to close a specific prepared statement * 6750: Executing a query on a non-existing column on a remote table crashes the remote server * 6785: function sys.isaURL(url) should have been declared as sys.isaURL(string) * 6808: reveal the alarm.sleep procedure in SQL * 6813: function not_uniques(bigint) returns error when called * 6818: usage of multiple column expressions in where-clause (f(a), f(b)) in (select a, b) causes assertion failure on mserver5 * 6821: Failed to start monetdb with embedded python * 6828: Server crashes when executing a window query with ordering by EXTRACT date * 6846: Global temporary table not accessible in other connections / sessions * 6847: A simple way of speeding up impscheck for dense canditers * 6850: Idle timestamp not set * 6851: json parser doesn't parse integers correctly * 6863: thash files not released upon drop table * 6864: (I)LIKE with multiple % doen't find matches

1 0

Announcement: New Nov2019-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 21 Feb '20

21 Feb '20

The MonetDB team at MonetDB BV is pleased to announce the Nov2019-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Nov 2019-SP2 bugfix release (11.35.15) Bug Fixes * 6805: Using the cascade operator in a drop table statement ends in an exit from the Monetdb shell. * 6807: Median_avg and quantile_avg ignore NULL values * 6815: query with ifthenelse() crashes mserver5 * 6816: Monetdb Crashes on INSERT statement after ALTER statement in another connection * 6817: running analyze on a schema which contains a stream table stops with an error * 6819: functions do not persist

1 0

Announcement: New Nov2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Dec '19

20 Dec '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019-SP1 bugfix release (11.35.9) Bug Fixes * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6724: Prepare confuses types when more than one argument is used * 6726: Python aggregation does not create aggr_group when aggregating over all rows * 6765: GRANT SELECT privilege on a subset of table columns results in access denied error when selecting the same columns from the table * 6790: Count distinct giving wrong results * 6791: str_to_time('11:40', '%H:%M') creates wrong time value * 6792: JSON path compiler accepts invalid input * 6793: cast(interval second value to int or decimal) is wrong (by a factor of 1000), cast(interval month value to decimal or floating point) fails * 6794: external name fits.listdir not bound (sys.listdir) Fatal error during initialization: * 6796: Incorrect crash time reported by monetdb tool after crash of mserver5 * 6798: json.text off by one error * 6804: DNS resolution of 0.0.0.0 fails on recent Ubuntus

1 0

Announcement: New Nov2019 Feature release of MonetDB suite
by Sjoerd Mullender 28 Nov '19

28 Nov '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019 feature release (11.35.3) MonetDB5 Server * The server "console" has been removed, as has the --daemon option. The server now doesn't read from standard input anymore. The way to stop a server is by sending it a TERM signal (on Linux/Unix) or by sending it an interrupt signal (usually control-C -- on all systems). * Implemented a function bat.diffcand to calculate difference of two candidate lists. * The mtime module was completely rewritten, the atom types date, daytime, and timestamp were changed. Upgrade code for BATs containing these types has been implemented. The old daytime type used a 32 bit integer to record milliseconds since the start of the day. The new daytime type uses a 64 bit integer to record microseconds since the start of the day. The old date type recorded days since or before the year 1. The new daytime type records the day of the month and the number of months since the year -4712 separately in a single 32 bit integer of which only 26 bits are used. Dates now use the proleptic Gregorian calendar, meaning the normal Gregorian callendar has been extended backward, and before the year 1, we have the year 0. Both the old and new timestamp types are a combination of a daytime and a date value, but since those types have changed, the timestamp type has also changed. The new date type has a smaller range than the old. The new date range is from the year -4712 to the year 170049. During conversion of date and timestamp columns, the dates are clamped to this range. * The tzone and rule atom types have been removed. They were not used by any code, and they were defined in a non-portable way. * Added "mapi_ipv6" property to monet_options to force ipv6 address binding only. This property is inherited while forking from monetdbd if it is also set there. * Removed (bat)calc.between_symmetric and changed (bat)calc.between by adding a number of extra arguments, all of type :bit: symmetric, low inclusive, high inclusive, nils false. Merovingian * Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively. * Added ipv6 property to monetdbd properties to force IPv6 addresses binding only. By default this property is false to allow IPv4 addresses as well. MonetDB Common * BATcalcbetween and all its variants now have an extra bool parameter "anti" to invert the test. * All forms of BATcalcbetween and VARcalcbetween have extra bool arguments for low inclusive, high inclusive and nils false. The latter causes the result to be false instead of nil if the value being checked is nil. SQL Frontend * Removed functions json.text(string) returns string and json.text(int) returns string. Their MAL implementation didn't exist, so they were meaningless. * There are new aggregate functions sys.median_avg and sys.quantile_avg that return the interpolated value if the median/quantile doesn't fall exactly on a particular row. These functions always return a value of type DOUBLE and only work for numeric types (various width integers and floating point). * Added sys.deltas ("schema" string, "table" string, "column" string) returns table ("values" bigint) system function which returns a single column with 6 values: a flag indicating if the column's upper table is cleared or not, the count of the RDONLY, RD_INS and RD_UPD_ID deltas of the column itself, the number of deleted values of the column's table, as well as the level of the current transaction in the transaction level tree. It should be used for debugging purposes only. * Added "VALUES row_list" statement as a top SQL projection statement. * The implementation of in-expression now follows a join-based approach instead of using iterative union/selects. This greatly improves performance for large in-value-lists. Furthermore the old approach has large in-value-lists generate large MAL plans. This is now no longer the case. * Strings are now limited to 1GB, double-quoted tokens are limited to 2kB. These sizes are bytes of (UTF-8 encoded) input data. Bug Fixes * 3533: SQL aggregate functions avg(), sum() and median() return an error when used on a column with datatype interval second * 6134: Query produces error: HEAPalloc: Insufficient space for HEAP of 1168033427456 bytes. * 6613: LATERAL crash /.../rel_bin.c:1473: rel2bin_table: Assertion `0' failed. * 6683: Bug in subselect * 6686: Bug in subselect (count function) * 6688: Bug in subselect (or condition) * 6689: Trying to improve the performance of SQL queries with a large list of members in IN clause. * 6695: timestamp transformation * 6700: Monetdb Bugs in Subselect statements: * 6722: window functions issues * 6740: while upgrading the database from verison (MonetDB-11.27.13) to (MonetDB-11.33.3) we are unable to bring up the database * 6754: in mclient a strang msg is reported after issueing command: set schema sys; * 6755: Assertion failure in rel_bin.c * 6756: Error in optimizer garbageCollector on merge tables select * 6757: Double free or corruption (out) * 6758: SIGSEGV in __strcmp_sse2_unaligned() * 6759: Python JSON loader creates invalid data type for strings * 6761: Error: Program contains errors.:(NONE).multiplex * 6762: mserver5 crashes on (re-)start * 6764: mserver5 crashes with corruption, double free, invalid size or invalid pointer * 6766: Missing bulk implementation for get_value and next_value calls * 6769: ProfilerStart is not threadsafe * 6771: R-devel * 6772: TRACE includes information from all active sessions * 6773: json.filter returns corrupted string when selecting JSON null value * 6774: PROD aggregation gives wrong result * 6775: NOT IN with an AND containing an OR gives wrong result * 6776: Creating a table with a full outer join query gives type with wrong digits on the joined key. * 6779: Using Windows Messages translation for errno error codes. * 6780: Wrong value of the rank function * 6781: Insert after index creation crash * 6783: AVG changes scale of its results * 6784: function sys.isauuid(string) should return false if string value cannot be converted to a UUID

1 0

Announcement: New Apr2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Sep '19

02 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019-SP1 bugfix release (11.33.11) Build Environment * Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched. MonetDB Common * We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available. SQL Frontend * Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries. * Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR. Bug Fixes * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6697: Duplicate expressions not eliminated with long CASE statement * 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed * 6703: SQL optimizer enters loop and goes into stack overflow * 6706: prepare doesn't recognize merge statement * 6712: Where clause with cast ignores sub-select * 6713: COPY INTO FROM 'file.xz' does not work * 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit) * 6715: Assertion failure in rel_bin.c for MERGE INTO command * 6716: COPY INTO does not load UTF8 encoded text * 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash) * 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation] * 6721: Possibly incorrect call to pcre_exec * 6725: Prepare statement on non-existing function crashes * 6728: SELECT DISTINCT not removing duplicates * 6729: Creating a table with duplicate column name should not be possible * 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always" * 6736: Unexpected query result with merge tables and view after upgrade * 6738: issue with window functions and hugeint type coercion * 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi * 6744: CREATE SEQUENCE seq; fails * 6748: MonetDB "forgets" columns after ALTER TABLE statements * 6749: mserver5 restart aborts/segfaults after dropping column with constraint * 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL

1 0

Announcement: New Apr2019 Feature release of MonetDB suite
by Sjoerd Mullender 30 Apr '19

30 Apr '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019 feature release (11.33.3) MonetDB5 Server * Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first. * Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8. * Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible. * The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter. Build Environment * We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line. * We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old). SELinux Support * There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories. Merovingian * Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it. Client Package * The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q. Mapi Library * Removed function mapi_cache_shuffle. * Removed function mapi_stream_query. Stream Library * The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.). * Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h. * Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream). * Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped. * Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'. MonetDB Common * During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts. * Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed). * Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending). * Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter). * The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'. * Function PROPdestroy now takes a BAT* parameter. * A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value. SQL Frontend * Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments. * Implemented SQL2003 natural logarithm function: ln(num_expr). * Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument). * Implemented length(blob) and comparison between blobs. * Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base. * Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm). * Removed unused global SQL variable "history". * Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong. * Allow usage of WITH clauses in insert, update, delete and merge statements. * Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] } * Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident * Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5) * Improved and extended storagemodel functionality. Changes include: * Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema. * Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns. * In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns. * System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize(). * System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya. * System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate. * System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead. * Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names. * Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table. * Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns. * Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value). * The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;). * Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);). * Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation. * Implemented X'...' style binary string literals. * Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation. * Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release. * Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending). * Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements. * The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>] * Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified. * The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated. * Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition. Bug Fixes * 2403: stream: http read support in streams * 2416: GDK: file/dir creation mask * 2478: SQL: rename columns * 2496: SQL: implement greatest / least * 3384: Auxiliary window functions * 3416: log() SQL function only supports natural logarithm * 3448: Make lbatomic_ops optional if installed * 3520: Add support for scalar function LN(num) * 3530: sigabort on window aggr function * 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename * 3743: DELETE FROM merge table not supported. * 3832: Cannot use expressions in GROUP BY clause * 3946: Expose C log2 function to SQL * 4056: mapi_timeout is not implemented * 6181: comments cause error with return in function * 6326: Eliminate unused UNION tables * 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB * 6475: Remove unused SQL global variable "history" * 6588: Table aliasing are not supported in SQL update queries * 6591: Name mangling in the SQL catalog results in ambiguity * 6608: Sqlitelogictest error message in group by with coalesce * 6609: Rename Schema in MonetDB * 6636: sys.queue() extension: allow DBA to see all running queries * 6647: Add suport to Python 3 on Windows * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6671: Error when running user function on merge table with remote part * 6674: R UDF with Date type fails to convert RAPI * 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids) * 6678: Binding NULL parameter to parametrized query results in syntax error on execution * 6680: Copy cannot open CSV file if systemctl enabled * 6684: Inserting into a replica table crashes the server * 6685: adding a view to a merge table gives unexpected behaviour * 6690: Unable to fetch empty string with ODBC driver * 6696: Re-use of MAL explain plans with complex type coercions

1 0

monetdb.org system maintenance
by Sjoerd Mullender 19 Feb '19

19 Feb '19

Tomorrow, Febuary 20, we will be upgrading the hardware for the system on which all monetdb.org infrastructure runs. Expect some downtime. -- Sjoerd Mullender

1 0