- users-list - monetdb.org

Announcement: New Dec2023-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 18 Mar '24

18 Mar '24

The MonetDB Foundation is pleased to announce the Dec2023-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### - The copyright for the MonetDB software has been transferred to the newly established MonetDB Foundation, a not-for-profit foundation with the express goal of furthering the MonetDB database system. The license for the software does not change: MonetDB remains fully open source. ### Client Package - Fixed an issue where mclient wouldn't exit if the server it had connected to exited for whatever reason while the client was waiting for a query result. ### MonetDB Common - The internal hash function for floating point types has been changed. It is now no longer based on the bit representation, but on the value, meaning that +0 and -0 (yes, they both exist in floating point) now hash to the same value. - Fixed a regression where bats weren't always cleaned up when they weren't needed anymore. In particular, after a DELETE FROM table query without a WHERE clause (which deletes all rows from the table), the bats for the table get replaced by new ones, and the old, now unused, bats weren't removed from the database. ### Geom Module - We switched over to using the reentrant interface of the geos library. This fixed a number of bugs that would occur sporadically. ### SQL Frontend - performance improvement of 'startswith' and 'endswith' filter functions for join operators - The function json.isvalid(json) incorrectly returned true if the argument was null. It should return null. ### Bug Fixes - [6800](https://github.com/MonetDB/MonetDB/issues/6800): Please add information_schema (ANSI SQL norm) - [7152](https://github.com/MonetDB/MonetDB/issues/7152): Occasional dbfarm corruption upon database restart - [7390](https://github.com/MonetDB/MonetDB/issues/7390): Some MonetDB Server crashes found - [7412](https://github.com/MonetDB/MonetDB/issues/7412): MonetDB server crashes in `vscanf` - [7415](https://github.com/MonetDB/MonetDB/issues/7415): MonetDB server crashes in `HEAP_malloc` - [7416](https://github.com/MonetDB/MonetDB/issues/7416): MonetDB server crashes in `atom_get_int` - [7417](https://github.com/MonetDB/MonetDB/issues/7417): MonetDB server crashes in `trimchars`. - [7418](https://github.com/MonetDB/MonetDB/issues/7418): MonetDB server crashes in `bind_col_exp` - [7420](https://github.com/MonetDB/MonetDB/issues/7420): Performance issue with lower(string) - [7425](https://github.com/MonetDB/MonetDB/issues/7425): The last statement, execution error, is a false positive? - [7426](https://github.com/MonetDB/MonetDB/issues/7426): Unexpected result for INNER JOIN with IS NOT NULL - [7428](https://github.com/MonetDB/MonetDB/issues/7428): Unexpected result when using BETWEEN operator - [7429](https://github.com/MonetDB/MonetDB/issues/7429): Unexpected result when using `CASE WHEN` - [7430](https://github.com/MonetDB/MonetDB/issues/7430): Unexpected result when using `AND` and `IS NOT NULL` - [7431](https://github.com/MonetDB/MonetDB/issues/7431): [bug] Error code found, please confirm - [7432](https://github.com/MonetDB/MonetDB/issues/7432): MonetDB server crashes in `dameraulevenshtein` - [7433](https://github.com/MonetDB/MonetDB/issues/7433): MonetDB server crashes in `exp_atom` - [7434](https://github.com/MonetDB/MonetDB/issues/7434): MonetDB server crashes in `exp_bin` - [7435](https://github.com/MonetDB/MonetDB/issues/7435): MonetDB server crashes in `exp_copy` - [7436](https://github.com/MonetDB/MonetDB/issues/7436): MonetDB server crashes in `exp_ref` - [7437](https://github.com/MonetDB/MonetDB/issues/7437): MonetDB server crashes in `exp_values_set_supertype` - [7438](https://github.com/MonetDB/MonetDB/issues/7438): MonetDB server crashes in `exps_bind_column` - [7439](https://github.com/MonetDB/MonetDB/issues/7439): MonetDB server crashes in `exps_card` - [7440](https://github.com/MonetDB/MonetDB/issues/7440): MonetDB server crashes in `gc_col` - [7441](https://github.com/MonetDB/MonetDB/issues/7441): MonetDB server crashes in `is_column_unique` - [7442](https://github.com/MonetDB/MonetDB/issues/7442): MonetDB server crashes in `mat_join2` - [7443](https://github.com/MonetDB/MonetDB/issues/7443): MonetDB server crashes in `merge_table_prune_and_unionize` - [7444](https://github.com/MonetDB/MonetDB/issues/7444): [bug] the table cannot be created because the reserved word is incorrectly set - [7447](https://github.com/MonetDB/MonetDB/issues/7447): Unexpected result when using `BETWEEN` in `INNER JOIN` - [7448](https://github.com/MonetDB/MonetDB/issues/7448): Unexpected result when using `AND`/`OR` chain - [7450](https://github.com/MonetDB/MonetDB/issues/7450): Unexpected result when `CREATE VIEW` with `WHERE NULL` - [7451](https://github.com/MonetDB/MonetDB/issues/7451): Unexpected result when using `BETWEEN` and `CAST` - [7453](https://github.com/MonetDB/MonetDB/issues/7453): Cannot recover an msqldump - [7455](https://github.com/MonetDB/MonetDB/issues/7455): Unexpected result when using `BETWEEN` with `BOOLEAN` values - [7456](https://github.com/MonetDB/MonetDB/issues/7456): Crash when `INNER JOIN` with `VIEW` - [7457](https://github.com/MonetDB/MonetDB/issues/7457): Unexpected result when using `AND` with `INTEGER` - [7458](https://github.com/MonetDB/MonetDB/issues/7458): Unexpected result when using `SIGN` - [7461](https://github.com/MonetDB/MonetDB/issues/7461): Crash by potentially use of bad escape characters - [7462](https://github.com/MonetDB/MonetDB/issues/7462): Crash when using `BETWEEN AND` - [7465](https://github.com/MonetDB/MonetDB/issues/7465): Unexpected result when using `NULL` in `BETWEEN`

1 0

[MonetDB-users] Proofreading your professional documents
by Richard Hounslow 24 Feb '24

24 Feb '24

Dear Professor/Lecturer Please excuse us for contacting you directly. We are offering an efficient service to help you ensure your academic and professional work is written in perfect English. We will check the grammar and style of your work and return it to you to meet your requirements and deadlines. Proof-Reading-Service.com can provide you with a professional proofreading service at a very reasonable rate. All our proofreaders are highly qualified native English speakers. Many work as leading academics in their fields and all have extensive experience of proofreading to the highest standards. If you are interested in our service, please take a look at our website: www.proof-reading-service.com All you have to do is send us your document as a word attachment with the deadline and we will guarantee delivery of a perfectly written document to give you complete confidence when you submit your work. The fee is worked out on a flat rate (£6.95 per thousand words or 0.695 pence per word), so you know exactly how much the proofreading will cost in advance. Once we have received your document(s) we will confirm the word count, the price and the deadline. If you have not heard from us within 2 hours during normal business hours after you have sent your work please resend it from a different email account or send the file to proofreadings(a)gmail.com. We look forward to hearing from you! Yours faithfully Proof-Reading-Service.com

2 2

Problem to install new fresh database
by Ariel Abadi 30 Dec '23

30 Dec '23

Hi! Im trying to install the latest version of Monetdb, and seems the repo has an error. Can anyone help!? or had an idea how to solve this? [root@tc-mn monetdb]# yum update Loaded plugins: fastestmirror Determining fastest mirrors epel/x86_64/metalink | 32 kB 00:00:00 * epel: mirror.de.leaseweb.net base | 3.6 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 2.9 kB 00:00:00 monetdb | 1.5 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/5): epel/x86_64/group_gz | 100 kB 00:00:00 (2/5): epel/x86_64/updateinfo | 1.0 MB 00:00:00 (3/5): monetdb/7/x86_64/primary | 120 kB 00:00:00 (4/5): updates/7/x86_64/primary_db | 24 MB 00:00:00 (5/5): epel/x86_64/primary_db | 7.0 MB 00:00:01 One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Parsing primary.xml error: Start tag expected, '<' not found [root@tc-mn monetdb]#

2 1

Announcement: New Dec2023 Feature release of MonetDB suite
by Sjoerd Mullender 28 Dec '23

28 Dec '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Dec2023 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Dec2023 documentation can be found [here](https://www.monetdb.org/documentation-Dec2023/). ## Dec2023 Feature Release (11.49.1) ### - All binary packages are now signed with a new key with key fingerprint DBCE 5625 94D7 1959 7B54 CE85 3F1A D47F 5521 A603. - The ranges of merge partitions are now pushed down into the low level GDK operations, giving them a handle to sometimes execute more efficiently. - Removed the PYTHON MAP external language option, as after a fork the synchronization primitives could be in any state, leading to deadlocks. During the upgrade function definitions will fallback to the normal PYTHON language option. - Implemented direct masking for strimp construction. The strimps datastructure now keeps an array of 65K 64-bit integers that is zero everywhere except at the indexes that correspond to header pairs. The entry for the nth pair in order has the nth bit of the bitstring on. These can be used to quickly construct bitstrings. ### ODBC Driver - Corrected the output value of column CHAR_OCTET_LENGTH of ODBC functions SQLColumns() and SQLProcedureColumns(). ### MonetDB Common - Fixed a (rare) race condition between copying a bat (COLcopy) and updates happening in parallel to that same bat. This may only be an actual problem with string bats, and then only in very particular circumstances. - Introduced options wal_max_dropped, wal_max_file_age and wal_max_file_size that control the write-ahead log file rotation. - Removed function BATroles to set column names on BATs. - Removed the compiled-in limit on the number of threads that can be used. The number of threads are still limited, but the limit is dictated solely by the operating system and the availability of enough memory. - We now prevent accidental upgrades from a database without 128 bit integers to one with 128 bit integers (also known as HUGEINT) from happening. Upgrades will only be done if the server is started with the option --set allow_hge_upgrade=yes. ### Geom Module - Because recent changes to the geom module require the use of geos 3.10, the geom module is no longer available in older versions of Debian and Ubuntu. Specifically, Debian 10 and 11 (buster and bullseye) and Ubuntu 20.04 (Focal Fossa) are affected. There is no automatic upgrade available for databases that were geom enabled to databases that are not, so dump + restore is the only option (if no geom types are actually used). - Implements Rtree index in GDK layer based on librtree. The index is used in the implementation of the filter functions ST_Intersects and ST_Dwithin for geometric points. - Improves shapefile support by replacing functions SHPattach, SHPpartialimport, ahd SHPimport with SHPload. - Introduces functions ST_DistanceGeographic, ST_DwithinGeographic, ST_IntersectsGeographic, ST_CoversGeographic, ST_Collects with geodesic semantics. ST_Transform can be used to convert geodetic into geographic data using libPROJ. ### MonetDB5 Server - Removed MAL functions bat.reuse and bat.reuseMap. - The MAL functions io.import and io.export have been removed. - Removed MAL functions bat.getRole and bat.setColumn since the underlying function BATroles was removed. - Change how json is stored in the database: We now normalize json strings after parsing, removing whitespace and eliminating duplicate keys in objects. - The function json.filter now properly returns json scalars instead of wrapping them in an array. - Removed the MAL tokenizer module. It was never usable from SQL and in this form never would be. ### SQL Frontend - Introduction of table returning function `persist_unlogged(schema string, table string)` that attempts to persist data in disk if "schema"."table" is unlogged table in insert only mode. If persist attempt is successful, the count of the persisted rows is returned, otherwise the count is 0. - Added ISO/IEC 9075-11 SQL/Schemata (SQL:2011) with SQL system views: information_schema.schemata information_schema.tables information_schema.views information_schema.columns information_schema.character_sets information_schema.check_constraints information_schema.table_constraints information_schema.referential_constraints information_schema.routines information_schema.parameters information_schema.sequences For details see https://www.monetdb.org/documentation/user-guide/sql-catalog/information_sc… Most views have been extended (after the standard columns) with MonetDB specific information columns such as schema_id, table_id, column_id, etc. This simplifies filtering and joins with system tables/views in sys schema when needed. Note: MonetDB does NOT support catalog qualifiers in object names, so all the CATALOG columns in these information_schema views will always return NULL. - Added support for generated column syntax: GENERATED BY DEFAULT AS IDENTITY ... This allows the user to override the default generated sequence value during inserts. - Added SQL support for: <result offset clause> and <fetch first clause> in <query expression> ::= [ <with clause> ] <query expression body> [ <order by clause> ] [ <result offset clause> ] [ <fetch first clause> ] [ <sample clause> ] <result offset clause> ::= OFFSET <offset row count> [ {ROW|ROWS} ] <fetch first clause> ::= FETCH {FIRST|NEXT} <fetch first row count> {ROW|ROWS} ONLY ### Bug Fixes - [6933](https://github.com/MonetDB/MonetDB/issues/6933): Add support for scalar function IFNULL(expr1, expr2) - [7044](https://github.com/MonetDB/MonetDB/issues/7044): Improve error message regarding 3-level SQL names - [7261](https://github.com/MonetDB/MonetDB/issues/7261): Misleading error message - [7274](https://github.com/MonetDB/MonetDB/issues/7274): Aggregate function ST_Collect crashes mserver5 - [7376](https://github.com/MonetDB/MonetDB/issues/7376): Concurrency Issue: Second Python UDF Awaits Completion of First UDF - [7391](https://github.com/MonetDB/MonetDB/issues/7391): SQL 2023 : greatest/least functions with unlimited arguments (not only 2) - [7403](https://github.com/MonetDB/MonetDB/issues/7403): Join not recognized between two row_number() columns - [7413](https://github.com/MonetDB/MonetDB/issues/7413): MonetDB server crashes in `BATcalcbetween_intern`

1 0

Announcement: New Jun2023-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Nov '23

20 Nov '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB Common - Fixed a regression where after a while the write-ahead log files weren't being rotated, meaning from some point onwards, the newest file just kept on growing. - When saving the SQL catalog during a low-level commit, we should only save the part of the catalog that corresponds to the part of the write-ahead log that has been processed. What we did was save more, which resulted in the catalog containing references to tables and columns whose disk presence is otherwise only in the write-ahead log. - A bug was fixed where the administration of which bats were in use was interpreted incorrectly during startup, causing problems later. One symptom that has been observed was failure to startup with a message that the catalog tables could not be loaded. ### SQL Frontend - Added a missing interface function sys.timestamp_to_str with a TIMESTAMP (as opposed to TIMESTAMP WITH TIME ZONE) argument. The missing interface caused error messages being produced when the function was called with a TIMESTAMP argument, although it did give the correct result. ### Bug Fixes - [7300](https://github.com/MonetDB/MonetDB/issues/7300): Implement missing standard SQL DATE and TIMESTAMP functions - [7324](https://github.com/MonetDB/MonetDB/issues/7324): string_distance('method',str1, str2) as a generic distance function - [7409](https://github.com/MonetDB/MonetDB/issues/7409): Numpy table returning UDFs with variadic arguments - [7410](https://github.com/MonetDB/MonetDB/issues/7410): SIGSEGV cause database corruption

1 0

Hot Backup no longer working when restoring
by warren.master＠rohatyngroup.com 18 Nov '23

18 Nov '23

i have been using hot backups for a while now , they are fast and have been very reliable for creating new servers and restoreing existig ones today I tried a restore from our daily hot backup that used call sys.hot_snapshot('/home/monetdb/mdb_hotbackup_20231117-1355390956724.tar.lz4'); I use the following to set up a new server monetdbd create /home/dbFarm tar -I lz4 -C /home/dbFarm -axf mdb_hotbackup_20231117-1355390956724.tar.lz4 monetdb start trg log filles with thousands of the below message and the db crashes MSG trg[9744]: #main thread: cleanup_and_swap: WARNING: Failed to set bat(381716) transient Im using lastest build on all machines MonetDB 5 server v11.47.11 (Jun2023-SP2) 2023-11-18 04:58:15 MSG trg[9744]: # Serving database 'trg', using 8 threads 2023-11-18 04:58:15 MSG trg[9744]: # Compiled for x86_64-pc-linux-gnu/64bit with 128bit integers 2023-11-18 04:58:15 MSG trg[9744]: # Found 62.759 GiB available main-memory of which we use 51.149 GiB 2023-11-18 04:58:15 MSG trg[9744]: # Copyright (c) 1993 - July 2008 CWI. 2023-11-18 04:58:15 MSG trg[9744]: # Copyright (c) August 2008 - 2023 MonetDB B.V., all rights reserved

1 0

Announcement: New Jun2023-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Oct '23

03 Oct '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### - Fixed an installation issue on Debian and Ubuntu introduced in the last build. - Do a lot more error checking, mostly for allocation failures. More is still needed, though. - Improve performance of the ILIKE operator when the pattern contains only ASCII characters. In this case we do not need to treat any characters as UTF-8 and we can use much faster routines that perform byte comparisons. ### MonetDB Common - Fixed a number of data races (race conditions). - Fixed a reference counting problem when a BAT could nog be loaded, e.g. because of resource limitations. - Only check for virtual memory limits when creating or growing bats, not for general memory allocations. There is (still) too much code that doesn't properly handle failing allocations, so we need to avoid those as much as possible. This has mostly an effect if there are virtual memory size restrictions imposed by cgroups (memory.swap.max in cgroups v2, memory.memsw.limit_in_bytes in cgroups v1). - The low-level commit turned out to always commit every persistent bat in the system. There is no need for that, it should only commit bats that were changed. This has now been fixed. - Implemented timeout/exit checks in a bunch more operators. Long(er) running operators occasionally check whether they're taking too long (past a user-specified timeout) or whether the server is exiting. This is now done in more places. ### Bug Fixes - [7094](https://github.com/MonetDB/MonetDB/issues/7094): Drop remote tables in transactions and rollback - [7303](https://github.com/MonetDB/MonetDB/issues/7303): Improve the performance of multi-column filters - [7400](https://github.com/MonetDB/MonetDB/issues/7400): VM max memory is not check correctly for cgroups v2 - [7401](https://github.com/MonetDB/MonetDB/issues/7401): Column aliases used incorrectly in UNION subqueries - [7402](https://github.com/MonetDB/MonetDB/issues/7402): Privileges on merge table not propagated to partition tables

1 0

rdf/sparql support in monetdb
by nguyenm9＠aol.com 08 Aug '23

08 Aug '23

Hey folks, new to monetdb and was wondering what the current support for rdf/sparql would be in the db? I can see mention of rdf support from years past but nothing recent. is this option still available? thanks!

1 0

Announcement: New Jun2023-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Jul '23

25 Jul '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB5 Server - There is now a new option --set tablet_threads=N to limit the number of threads used for a COPY INTO from CSV file query. This option can also be set for a specific database using the monetdb command using the ncopyintothreads property. ### SQL Frontend - An upgrade that both creates a new .snapshot user and extends the sys.db_user_info table with (among others) a password column did these in such a way that the passord value for the new user was NULL. This is fixed by updating the password. ### Bug Fixes - [7388](https://github.com/MonetDB/MonetDB/issues/7388): Query results in large cross product - [7394](https://github.com/MonetDB/MonetDB/issues/7394): BBPextend: ERROR: trying to extend BAT pool beyond the limit (163840000)

1 0

Announcement: New Jun2023 Feature release of MonetDB suite
by Panagiotis Koutsourakis 29 Jun '23

29 Jun '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Jun2023 documentation can be found [here](https://www.monetdb.org/documentation-Jun2023/). ## Jun2023 Feature Release (11.47.3) ### Client Package - The COPY INTO from file ON CLIENT was extended to also look for a relative path name relative to the file from which the query was read. This is only possible if the name of the query file is known, so when it is specified on the command line or read using the interactive \< command. ### Mapi Library - Deprecate mapi_setfilecallback() in favor of mapi_setfilecallback2() which can handle binary downloads. For the time being, the old callback still works. ### ODBC Driver - Extended Windows MonetDB ODBC Data Source setup program with option to specify a logfile to enable tracing of ODBC Driver API calls. On other platforms users can edit the odbc.ini file and add a line: logfile=/home/username/odbctrace.log When a logfile is specified it will start logging the ODBC Driver API calls to the logfile after a new connection is made via SQLConnect() or SQLDriverConnect() or SQLBrowseConnect(). Note that enabling ODBC logging will slow down the performance of ODBC applications, so enable it only for analysing ODBC Driver problems. - Enhanced SQLTables() by adding support for table type names: 'BASE TABLE', 'GLOBAL TEMPORARY' and 'LOCAL TEMPORARY' in parameter TableType. These are synonyms of: 'TABLE', 'GLOBAL TEMPORARY TABLE' and 'LOCAL TEMPORARY TABLE'. ### MonetDB5 Server - The MAL debugger code has been removed. ### SQL Frontend - Add MAX_MEMORY and MAX_WORKERS options to the ALTER USER statement - Function 'similarity(x string, y string)' marked as deprecated and will be removed in the next release. - New functionality for string matching and similarity: startswith, endswith, contains, Levenshtein distance and Jaro-Winkler similarity. The functions startswith, endswith and contains have a version where a case insentive flag can be used. Also, there are new custom join functionality for startswith, endswith, contains, Levenshtein distance and Jaro-Winkler similarity. - Renamed previous Levenshtein distance to Damerau-Levenshtein distance. - New string function that transform from UTF-8 encoding to Ascii called asciify. - Session timeout feature improvement to start evaluating from the moment the procedure is called, instead of beginning of the session. - Queries stopped with the stop procedure are now marked as 'aborted' 'finished'. - The DEBUG statement has been removed. - SQL function sys.queue() overloaded with sys.queue(username string), SYSADMIN only, allowing to filter the global queue by username or use 'ALL' to retrieve the global queue. Calling the function without arguments returns the queue for the current user. - SQL procedures sys.pause(tag bigint), sys.resume(tag bigint), sys.stop(tag bigint) overloaded with sys.pause(tag bigint, username string), sys.resume(tag bigint, username string) and sys.stop(tag bigint, username string), SYSADMIN only, allowing to pause, resume and stop query executions by TAG, USERNAME. The call without arguments is a public procedure giving access to users to pause, resume and stop their own query executions. - Added support of ODBC escape sequences syntax to SQL layer. Now all clients (including ODBC/JDBC/pymonetdb) can use them without further processing. For details on ODBC escape sequences syntax see: https://learn.microsoft.com/en-us/sql/odbc/reference/appendixes/odbc-escape… - It is no longer allowed to create a merge table or remote table or replica table or unlogged table in schema "tmp". The tmp schema is reserved for temporary objects only, such as local/global temp tables. - System views sys.dependency_tables_on_functions and dependency_views_on_functions have been extended with column: function_id. - When loading data using COPY BINARY INTO, apply default values instead of just inserting NULLs. - When loading data using COPY BINARY INTO, validate DECIMAL(prec,scale) and VARCHAR(n) column width. - When loading data using COPY BINARY INTO, string used to have their line endings converted from CR LF to LF. Do not do this, it is the responsibility of the client. - Implemented dumping binary data using COPY SELECT ... INTO BINARY <file(s)>. - Removed code for Workload Capture and Replace, including system schemas "wlc" and "wlr" and the objects in those schemas. The code was experimental, and it didn't work out. A different approach will be taken. ### Merovingian - The monetdb get, inherit, lock and release commands are extended with ‘apply-to-all’ syntax similar to related functionality: Usage: monetdb set property=value [database ...] sets property to value for the given database(s), or all Usage: monetdb inherit property [database ...] unsets property, reverting to its inherited value from the default configuration for the given database(s), or all Usage: monetdb lock [-a] database [database...] Puts the given database in maintenance mode. Options: -a locks all known databases Usage: monetdb release [-a] database [database ...] Brings back a database from maintenance mode. Options: -a releases all known databases ### Bug Fixes - [7311](https://github.com/MonetDB/MonetDB/issues/7311): Missing `REGEXP_REPLACE` function. - [7344](https://github.com/MonetDB/MonetDB/issues/7344): Database upgrade failure due to user object dependency on system procedure - [7348](https://github.com/MonetDB/MonetDB/issues/7348): Subquery inside case always evaluated - [7378](https://github.com/MonetDB/MonetDB/issues/7378): MonetDB server crashes at sql_trans_copy_key - [7379](https://github.com/MonetDB/MonetDB/issues/7379): MonetDB server 11.46.0 crashes at cs_bind_ubat - [7380](https://github.com/MonetDB/MonetDB/issues/7380): MonetDB server 11.46.0 crashes at `BLOBcmp` - [7381](https://github.com/MonetDB/MonetDB/issues/7381): MonetDB server 11.46.0 crashes at `log_create_delta` - [7382](https://github.com/MonetDB/MonetDB/issues/7382): MonetDB server 11.46.0 crashes at `gc_col` - [7383](https://github.com/MonetDB/MonetDB/issues/7383): MonetDB server 11.46.0 crashes at `list_append` - [7384](https://github.com/MonetDB/MonetDB/issues/7384): MonetDB server 11.46.0 crashes at `__nss_database_lookup` - [7386](https://github.com/MonetDB/MonetDB/issues/7386): MonetDB server 11.46.0 crashes in `rel_deps` - [7387](https://github.com/MonetDB/MonetDB/issues/7387): MonetDB server 11.46.0 crashes in `rel_sequences`

1 0

Announcement: New Sep2022-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 08 Jun '23

08 Jun '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - If the number of rows in mclient is set to 0 (using either --rows=0 option or \r0 on the mclient command line), the internal pager is used and it then uses the height of the terminal window. ### MonetDB Common - Warnings and informational messages are now sent to stdout instead of stderr, which means that monetdbd will now log them with the tag MSG instead of ERR. - Fixed parsing of the BBP.dir file when BAT ids grow larger than 2**24 (i.e. 100000000 in octal). - Fixed yet another occurrence of a missing .tailN file. This one could happen if a string bat was appended to in stages so that between appends the column was committed. If an append caused both a realloc of the tail heap because it was getting longer and a realloc because it got wider, the file might get removed before the GDK level commit happened. - When processing the WAL, if a to-be-destroyed object cannot be found, don't stop, but keep processing the rest of the WAL. - A race condition was fixed where certain write-ahead log messages could get intermingled, resulting in a corrupted WAL file. - If opening of a file failed when it was supposed to get memory mapped, an incorrect value was returned to indicate the failure, causing crashes later on. This has been fixed. - A race condition was fixed that could result in a missing tail file for a string bat (i.e. a file with .tail1, .tail2, or .tail4 extension). - When saving a bat failed for some reason during a low-level commit, this was logged in the log file, but the error was then subsequently ignored, possibly leading to files that are too short or even missing. - The write-ahead log (WAL) is now rotated a bit more efficiently by doing multiple log files in one go (i.e. in one low-level transaction). ### MonetDB5 Server - Client connections are cleaned up better so that we get fewer instances of clients that cannot connect. ### SQL Frontend - Fixed a regression introduced in the previous build: when a table is dropped in the same transaction where it was created, the data wasn't removed from disk. - When creating a hot snapshot, allow other clients to proceed, even with updating queries. - Increased the size of a variable counting the number of changes made to the database (e.g. in case more than 2 billion rows are added to a table). - Improved cleanup after failures such as failed memory allocations. - An insert into a table from which a column was dropped in a parallel transaction was incorrectly not flagged as a transaction conflict. - Fix bug where boolean NULLs were not recognized by COPY BINARY INTO ### Bug Fixes - [7360](https://github.com/MonetDB/MonetDB/issues/7360): Aggregates returning string crash - [7361](https://github.com/MonetDB/MonetDB/issues/7361): Table data is lost on DB restart, but only when a table has 2147483647 or more rows.

1 0

Possible BUG??
by Ariel Abadi 08 Mar '23

08 Mar '23

Hi All! I used to have a query that was working in a prior version (11.39.13), but now that I have upgraded to the newest version (11.45.13) this same query is not working any more. The query is the following: SELECT nrm_tim_date AS nrm_tim_date, ( (SELECT SUM(nrm_rse_amount_usd * exc.exc_rate) from sb_glob.exc_exchange exc where exc.exc_currency = 'USD') ) AS net_profit, SUM(nrm_rse_amount_usd) AS nrm_rse_amount_usd FROM sb_traf.cubm_202303 AS nrm GROUP BY nrm_tim_date The error Im having is "SQL Error [42000]: SELECT: subquery uses ungrouped column "nrm.nrm_rse_amount_usd" from outer query" This exact same query does work on version 11.39.13. Can someone help me with this? Thks a lot in advance ARIEL ABADI Direct: +54 11 5279.2054 Mobile: +54 9 11 6050.0101 *Email*: aabadi(a)starbi.com *Web*: www.starbi.com

2 1

Announcement: New MonetDB JDBC driver and JdbcClient released
by Martin van Dinther 02 Mar '23

02 Mar '23

The MonetDB team at MonetDB Solutions BV is pleased to announce a bug fix release of the MonetDB JDBC driver (monetdb-jdbc-3.3.jre8.jar) and JdbcClient CLI program (jdbcclient.jre8.jar). Besides bug fixes the new release improves performance and compliance. For a list of all changes see: https://www.monetdb.org/downloads/Java/ChangeLog We recommend all users of MonetDB JDBC driver and JdbcClient program to download and use the latest jar files. The download location is: https://www.monetdb.org/downloads/Java/ For information and tips on using the MonetDB JDBC driver, read: https://www.monetdb.org/downloads/Java/release.txt For information on using the MonetDB JDBC driver to execute COPY INTO <table> FROM <file> ON CLIENT ... and/or COPY <SELECT query> INTO <file> ON CLIENT ... commands for fast client side CSV file data import and export, read: https://www.monetdb.org/downloads/Java/onclient.txt Reminder: since release monetdb-jdbc-3.0.jre8.jar the MonetDB JDBC Driver class name has been changed to: org.monetdb.jdbc.MonetDriver It used to be nl.cwi.monetdb.jdbc.MonetDriver which is marked as deprecated and will be removed in a future release. Please use the new driver class name asap in your configuration files/scripts and/or Java code.

1 0

Announcement: New Sep2022-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Jan '23

31 Jan '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### ODBC Driver - A crash in the ODBC driver was fixed when certain unsupported functions where used in a {fn ...} escape. - Prepare of a query where the sum of the number of parameters (question marks in the query) and the number of output columns is larger than 100 could fail with an unexpected error. This has been fixed. ### MonetDB Common - Fixed a race condition that could lead to a bat being added to the SQL catalog but nog being made persistent, causing a subsequent restart of the system to fail (and crash). - Fixed a race condition where a hash could have been created on a bat using the old bat count while in another thread the bat count got updated. This would make the hash be based on too small a size, causing failures later on. - When extending a bat failed, the capacity had been updated already and was therefore too large. This could then later cause a crash. This has been fixed by only updating the capacity if the extend succeeded. ### SQL Frontend - Fixed a regression where when there are multiple concurrent transactions, the dependencies weren't checked properly. - Added some error checking to prevent crashes. Errors would mainly occur under memory pressure. - Fixed cleanup after a failed allocation where the data being cleaned up was unitialized but still used as pointers to memory that also had to be freed. - Fixed a double cleanup after a failed allocation in COPY INTO. The double cleanup could cause a crash due to a race condition it enabled. ### Bug Fixes - [7343](https://github.com/MonetDB/MonetDB/issues/7343): GDKmmap requesting 0 virtual memory - [7347](https://github.com/MonetDB/MonetDB/issues/7347): A bug where an exception occurs even though it is a query with normal syntax (Merge Table)

1 0

Announcement: New Sep2022-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 09 Dec '22

09 Dec '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - Also dump the new options of CREATE USER. ### ODBC Driver - Fixed issue with generated raw strings prefix when ODBC driver is used against a server older than Jun2020 (11.37). ### MonetDB Common - On Windows, use the wide-character interface of system calls when dealing with the environment (i.e. file names and getenv()). - Memory leaks have been fixed. - Improved maintenance of the estimated number of distinct values in BATs. The estimate helps in deciding which low-level algorithm to use. - Offset heaps (.tailN files) were growing too fast and unnecessarily under certain conditions. This has been fixed. Also, when such too large files are now loaded into the system, it is recognized they are too large and they are truncated to a more reasonable size. ### MonetDB5 Server - Fixed a crash when the server runs out of client contexts (i.e. more concurrent clients than the server is configured to handle). - A race condition in the SHA hash code was fixed which resulted in occasional failed connection attempts when they occurred concurrently. - Fix a bug where the MAL optimizer would use the starttime of the previous query to determine whether a query timeout occurred. ### SQL Frontend - Improved the handling of the "idle" value in the sys.sessions function and view. ### Merovingian - Stop logging references to monetdbd's logfile in said logfile. ### Bug Fixes - [7330](https://github.com/MonetDB/MonetDB/issues/7330): Creating temporary table fails after reconnect - [7333](https://github.com/MonetDB/MonetDB/issues/7333): DLLs fail to load on Windows with accented characters in path - [7336](https://github.com/MonetDB/MonetDB/issues/7336): Selecting from a literal-value table returns wrong values - [7339](https://github.com/MonetDB/MonetDB/issues/7339): MonetDB corrupted state after SIGKILL - [7342](https://github.com/MonetDB/MonetDB/issues/7342): column which datatype is double couldn't group or aggregation in version 11.45.7

1 0

Re: JSON column disk space
by Panagiotis Koutsourakis 07 Oct '22

07 Oct '22

Hello Alberto, I am looking into this issue and I would like to give you a small update and ask a couple of questions. Some background first. JSON objects are stored in MonetDB as strings. This creates overhead from the object keys and from numeric values that are now stored as strings. If I understand your issue correctly you don't have the chance to commit the reconstructed JSON column. Is this correct? From some tests I performed, I noticed that even before committing, disk usage goes up slowly. This is probably unrelated to JSON itself although I have not had a chance to verify this yet. So I think the most relevant question at this stage is how long are the keys in the JSON objects? I think that these are the most likely cause for the large overhead you are observing. Best regards, Panos. On 2022-09-28 21:17, Alberto Ferrari wrote: > Niels, > It's hard to me to create a demo script, but at leats I'd want to know > if it's a known "issue" (maybe it's not an issue, just as Monet works > internally) > > I can tell you that a table with 100 columns (ints, varchars, dates, > etc) with 30 million records, may occupy 30 Gb of disk. > When I merge some of these columns into one new json column -(i.e. 5 > single columns into one object like {col1:value, col2:value...}- and > dropped the 5 old columns (already merged in json), the table now > grows upto 100 Gb of disk. > > Thanks > > > El jue, 22 sept 2022 a la(s) 10:35, Niels Nes > (niels.nes(a)monetdbsolutions.com) escribió: >> >> >> Alberto >> >> Do you have some script which replicates this issue? >> And could you post the issue on the github issues? >> >> Niels >> >> On Thu, Sep 22, 2022 at 07:57:07AM -0300, Alberto Ferrari wrote: >> > Hi all: >> > >> > Monet version 11.35.19 (Nov2019-SP3) >> > >> > We realized that json columns in any table produces bigger stored >> > disk, no matter how many rows has big data. >> > It seems like depends on rows with different values in same json >> > column: if many rows has different values, then the space on disk >> > increases exponentially. So we can't use jaon columns in big tables, >> > since we run out of space disk in short time. >> > (also tested with 11.43.5 -Jan2022, same issue) >> > >> > Is there any solution to this problem? >> > Tanks! >> > Alberto >> > _______________________________________________ >> > users-list mailing list -- users-list(a)monetdb.org >> > To unsubscribe send an email to users-list-leave(a)monetdb.org >> >> _______________________________________________ >> users-list mailing list -- users-list(a)monetdb.org >> To unsubscribe send an email to users-list-leave(a)monetdb.org > _______________________________________________ > users-list mailing list -- users-list(a)monetdb.org > To unsubscribe send an email to users-list-leave(a)monetdb.org

4 4

JSON column disk space
by Alberto Ferrari 28 Sep '22

28 Sep '22

Hi all: Monet version 11.35.19 (Nov2019-SP3) We realized that json columns in any table produces bigger stored disk, no matter how many rows has big data. It seems like depends on rows with different values in same json column: if many rows has different values, then the space on disk increases exponentially. So we can't use jaon columns in big tables, since we run out of space disk in short time. (also tested with 11.43.5 -Jan2022, same issue) Is there any solution to this problem? Tanks! Alberto

2 4

Announcement: New Sep2022 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '22

26 Sep '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Sep2022 documentation can be found [here](https://www.monetdb.org/documentation-Sep2022/). ## Sep2022 Feature Release (11.45.7) ### Client Package - Dumping of function GRANTs was improved by adding the types of the function (and procedure) arguments. ### ODBC Driver - Improved SQLSetPos(). It now allows RowNumber to be 0 when Operation is SQL_POSITION. - Improved ODBC Error/Warning messages. They now include the name of the Data Source as required by the ODBC specification: [MonetDB][ODBC driver VERSION][data-source-name] data-source-supplied-text - Improved MonetDB ODBC Data Source Configuration dialog on MS Windows by hiding the typed in password text. It now shows dots for the characters. This fixes request https://github.com/MonetDB/MonetDB/issues/7314 - Changed output of TABLE_CAT or PROCEDURE_CAT result columns as returned by ODBC functions: SQLTables(), SQLColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). They used to return the static database name but now they will return NULL as MonetDB does not support CATALOG objects or qualifiers. - Removed the possibility to retrieve or set the CURRENT_CATALOG via SQLGetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) and SQLSetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) as MonetDB does not support CATALOG objects (no SQL support for: CREATE CATALOG abc or SET CATALOG abc) and therefore there is no CURRENT_CATALOG. - Corrected ODBC functions SQLTablePrivileges() and SQLColumnPrivileges() for local temporary tables located in schema tmp. They did not return any rows when the temporary table had privileges set. Now they do return rows as expected. - Improved SQLProcedures() and SQLProcedureColumns(). They now list information also for all built-in system procedures and functions, not only those created via SQL. Also corrected the value of ORDINAL_POSITION for scalar function arguments. It would start at 2 instead of 1. - Extended output of SQLProcedures() and SQLProcedureColumns() resultsets with an extra column SPECIFIC_NAME. This column contains the name which uniquely identifies this procedure or function within its schema. As MonetDB supports overloaded procedures and functions, the combination of PROCEDURE_SCHEM and PROCEDURE_NAME is not enough to uniquely identify a procedure or function. This extra column allows you to correctly match the corresponding rows returned by SQLProcedureColumns() with the specific rows of SQLProcedures(). This extra column SPECIFIC_NAME is implemented similar to the JDBC DatabaseMetaData methods getProcedures() and getProcedureColumns(). - For SQLForeignKeys() corrected the output of columns UPDATE_RULE and DELETE_RULE. These columns used to always return 3 (= SQL_NO_ACTION) but now they will report the action codes as specified in the FOREIGN KEY CONSTRAINT construction. - Fixed issue in function SQLProcedureColumns(). When the argument ColumnName was not empty it generated an invalid SQL query which failed with error: SELECT: no such column 'c.name'. This has been resolved. - Corrected implementation of SQLGetInfo(hdbc, SQL_MAX_DRIVER_CONNECTIONS, ...). It used to always return 64. Now it returns the value from the MonetDB server. - Changed the column names case of the result sets as returned by SQLTables(), SQLColumns(), SQLSpecialColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). The column names where all in lowercase but the ODBC specification defines them in uppercase, so changed them to uppercase. - Corrected SQLSpecialColumns(..., SQL_BEST_ROWID, ...). Previously it only returned rows when the table had a primary key. Now it also returns rows when a table has a unique constraint. - Corrected SQLStatistics(..., SQL_INDEX_ALL, ...). Previously it only returned rows when the table had a primary or unique key. Now it also returns rows for indices which are not based on a key. - Corrected SQLStatistics(..., SQL_ENSURE). It now returns CARDINALITY information for columns based on a primary/unique key. Previously it always returned NULL for the CARDINALITY result column. - Corrected ODBC functions SQLPrimaryKeys(), SQLSpecialColumns() and SQLStatistics() for local temporary tables located in schema tmp. They did not return any rows when the temp table had a primary or unique key or index. Now they do return rows as expected. ### MonetDB Common - Fixed a bug in ORDER BY with both NULLS LAST and LIMIT when the ordering was on an interger or integer-like column and NULL values are present. - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). - The abort_on_error parameter of all GDK-level functions has been removed. Errors (e.g. overflow) now also results in an error. - Implemented BC/AD (and BCE/CE) suffixes when parsing dates. - The interface for using strimps has not changed (create an imprint index on a column of a read only table), but now construction happens at the beginning of the first query that uses the strimp and is performed in a multithreaded manner. - Implemented the use of strimps for NOT LIKE queries. The idea is to run the corresponding LIKE query using strimps and take the complement of the result. We keep around NULL values both during strimp filtering and during the pcre part of the LIKE query so that they get discarded automatically when we take the complement. - The function BBPkeepref now gets a BAT pointer as argument instead of a bat id. - Get rid of macro Tsize, use ->twidth instead. - Get rid of macro BUNlast, just use BATcount instead. - The BLOB type has been moved into the GDK layer. - When adding or subtracting months from a date or timestamp value, clamp the result to the calculated month instead of wrapping to the beginning of the next month. See bug [7227](https://https://github.com/MonetDB/MonetDB/issues/7227). - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). ### MonetDB5 Server - Disabled volcano pipeline due to known issues. ### SQL Frontend - The function sys.tracelog is now executable by anyone. The function (and view of the same name) returns the tracing information of a query that was prepended with the TRACE keyword. - Fixed a bug in COPY BINARY INTO where the input wasn't checked thoroughly enough. - Extended the built-in profiler to emit non-MAL events related to query compilation, optimization, transactions and client connections. To minimize, simplify and optimize the process of generating and processing profiler output, only the events marking the end of an operation are emitted in most cases and the emitted json messages themselves are trimmed down to their essential fields. Furthermore, the MAL instruction profiler.openstream now requires an integer as a single argument, "0" for default behaviour or "4" to turn on the profiler in minimal mode, which causes it to only emit general events and excludes individual MAL instruction execution events from the profiler streams. The MAL instruction profiler.openstream with zero arguments is deprecated. - Extended system view sys.users with columns: schema_path, max_memory, max_workers, optimizer and default_role. Extended system table sys.db_user_info with columns: schema_path, max_memory, max_workers, optimizer, default_role and password. The password is encrypted. This table can only be queried when the user has been granted the select privilege from monetdb. - CREATE USER statement has been extended to take more optional arguments like MAX_MEMORY, MAX_WORKERS, OPTIMIZER, DEFAULT ROLE. ALTER USER statement has also been extended with DEFAULT ROLE. - A new function sys.url_extract_host(string, bool) which returns the host name from the given URL has been implemented. The bool argument, if true, causes the www. prefix to be removed. - The user authentication tables are now administered by the SQL layer instead of the MAL layer. This means that changing (adding, dropping, altering) user and role information is now transaction-safe. - Added the UNLOGGED TABLE feature. - Removed functions sys.index and sys.strings. ### Merovingian - Added "loglevel" property for monetdbd logging (to merovingian.log). The loglevel can be set to: error or warning or information or debug. The loglevel property can be changed dynamically via command: monetdbd set loglevel=warning /path/to/dbfarm Default the loglevel is set to: information When loglevel is error, only errors are logged. When loglevel is warning, errors and warnings are logged. When loglevel is information, errors and warnings and information messages are logged. When loglevel is debug, all messages are logged. - There is now a loadmodules property that can be used to add --loadmodule arguments to the mserver5 command line. See the monetdb and mserver5 manual pages. - In certain cases (when an mserver5 process exits right after producing a message) the log message was logged over and over again, causing monetdbd to use 100% CPU. This has been fixed. ### Bug Fixes - [6982](https://github.com/MonetDB/MonetDB/issues/6982): Wrong constraint name in error message of constraint violation - [7209](https://github.com/MonetDB/MonetDB/issues/7209): Configuration option for merovingian.log - [7225](https://github.com/MonetDB/MonetDB/issues/7225): Invalid memory access when extending a BAT during appends - [7227](https://github.com/MonetDB/MonetDB/issues/7227): Date calculations, bug or feature - [7273](https://github.com/MonetDB/MonetDB/issues/7273): Concurrent reads and writes causes "BATproject2: does not match always" error - [7282](https://github.com/MonetDB/MonetDB/issues/7282): call sys.dump_table_data(); fails - [7285](https://github.com/MonetDB/MonetDB/issues/7285): C-UDFs: aggr_group.count has wrong value (number of input rows instead of number of groups). - [7296](https://github.com/MonetDB/MonetDB/issues/7296): Implictly cast a timestamp string to DATE when appropriate - [7297](https://github.com/MonetDB/MonetDB/issues/7297): Parsing partial dates behaves unpredictable - [7306](https://github.com/MonetDB/MonetDB/issues/7306): ODBC Driver Assertion `stmt->Dbc->FirstStmt' Failed - [7314](https://github.com/MonetDB/MonetDB/issues/7314): ODBC Driver : please mask/hide password - [7318](https://github.com/MonetDB/MonetDB/issues/7318): distinct in a subquery not working properly

1 0

Announcement: New Sep2022 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '22

26 Sep '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Sep2022 documentation can be found [here](https://www.monetdb.org/documentation-Sep2022/). ## Sep2022 Feature Release (11.45.7) ### Client Package - Dumping of function GRANTs was improved by adding the types of the function (and procedure) arguments. ### ODBC Driver - Improved SQLSetPos(). It now allows RowNumber to be 0 when Operation is SQL_POSITION. - Improved ODBC Error/Warning messages. They now include the name of the Data Source as required by the ODBC specification: [MonetDB][ODBC driver VERSION][data-source-name] data-source-supplied-text - Improved MonetDB ODBC Data Source Configuration dialog on MS Windows by hiding the typed in password text. It now shows dots for the characters. This fixes request https://github.com/MonetDB/MonetDB/issues/7314 - Changed output of TABLE_CAT or PROCEDURE_CAT result columns as returned by ODBC functions: SQLTables(), SQLColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). They used to return the static database name but now they will return NULL as MonetDB does not support CATALOG objects or qualifiers. - Removed the possibility to retrieve or set the CURRENT_CATALOG via SQLGetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) and SQLSetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) as MonetDB does not support CATALOG objects (no SQL support for: CREATE CATALOG abc or SET CATALOG abc) and therefore there is no CURRENT_CATALOG. - Corrected ODBC functions SQLTablePrivileges() and SQLColumnPrivileges() for local temporary tables located in schema tmp. They did not return any rows when the temporary table had privileges set. Now they do return rows as expected. - Improved SQLProcedures() and SQLProcedureColumns(). They now list information also for all built-in system procedures and functions, not only those created via SQL. Also corrected the value of ORDINAL_POSITION for scalar function arguments. It would start at 2 instead of 1. - Extended output of SQLProcedures() and SQLProcedureColumns() resultsets with an extra column SPECIFIC_NAME. This column contains the name which uniquely identifies this procedure or function within its schema. As MonetDB supports overloaded procedures and functions, the combination of PROCEDURE_SCHEM and PROCEDURE_NAME is not enough to uniquely identify a procedure or function. This extra column allows you to correctly match the corresponding rows returned by SQLProcedureColumns() with the specific rows of SQLProcedures(). This extra column SPECIFIC_NAME is implemented similar to the JDBC DatabaseMetaData methods getProcedures() and getProcedureColumns(). - For SQLForeignKeys() corrected the output of columns UPDATE_RULE and DELETE_RULE. These columns used to always return 3 (= SQL_NO_ACTION) but now they will report the action codes as specified in the FOREIGN KEY CONSTRAINT construction. - Fixed issue in function SQLProcedureColumns(). When the argument ColumnName was not empty it generated an invalid SQL query which failed with error: SELECT: no such column 'c.name'. This has been resolved. - Corrected implementation of SQLGetInfo(hdbc, SQL_MAX_DRIVER_CONNECTIONS, ...). It used to always return 64. Now it returns the value from the MonetDB server. - Changed the column names case of the result sets as returned by SQLTables(), SQLColumns(), SQLSpecialColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). The column names where all in lowercase but the ODBC specification defines them in uppercase, so changed them to uppercase. - Corrected SQLSpecialColumns(..., SQL_BEST_ROWID, ...). Previously it only returned rows when the table had a primary key. Now it also returns rows when a table has a unique constraint. - Corrected SQLStatistics(..., SQL_INDEX_ALL, ...). Previously it only returned rows when the table had a primary or unique key. Now it also returns rows for indices which are not based on a key. - Corrected SQLStatistics(..., SQL_ENSURE). It now returns CARDINALITY information for columns based on a primary/unique key. Previously it always returned NULL for the CARDINALITY result column. - Corrected ODBC functions SQLPrimaryKeys(), SQLSpecialColumns() and SQLStatistics() for local temporary tables located in schema tmp. They did not return any rows when the temp table had a primary or unique key or index. Now they do return rows as expected. ### MonetDB Common - Fixed a bug in ORDER BY with both NULLS LAST and LIMIT when the ordering was on an interger or integer-like column and NULL values are present. - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). - The abort_on_error parameter of all GDK-level functions has been removed. Errors (e.g. overflow) now also results in an error. - Implemented BC/AD (and BCE/CE) suffixes when parsing dates. - The interface for using strimps has not changed (create an imprint index on a column of a read only table), but now construction happens at the beginning of the first query that uses the strimp and is performed in a multithreaded manner. - Implemented the use of strimps for NOT LIKE queries. The idea is to run the corresponding LIKE query using strimps and take the complement of the result. We keep around NULL values both during strimp filtering and during the pcre part of the LIKE query so that they get discarded automatically when we take the complement. - The function BBPkeepref now gets a BAT pointer as argument instead of a bat id. - Get rid of macro Tsize, use ->twidth instead. - Get rid of macro BUNlast, just use BATcount instead. - The BLOB type has been moved into the GDK layer. - When adding or subtracting months from a date or timestamp value, clamp the result to the calculated month instead of wrapping to the beginning of the next month. See bug [7227](https://https://github.com/MonetDB/MonetDB/issues/7227). - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). ### MonetDB5 Server - Disabled volcano pipeline due to known issues. ### SQL Frontend - The function sys.tracelog is now executable by anyone. The function (and view of the same name) returns the tracing information of a query that was prepended with the TRACE keyword. - Fixed a bug in COPY BINARY INTO where the input wasn't checked thoroughly enough. - Extended the built-in profiler to emit non-MAL events related to query compilation, optimization, transactions and client connections. To minimize, simplify and optimize the process of generating and processing profiler output, only the events marking the end of an operation are emitted in most cases and the emitted json messages themselves are trimmed down to their essential fields. Furthermore, the MAL instruction profiler.openstream now requires an integer as a single argument, "0" for default behaviour or "4" to turn on the profiler in minimal mode, which causes it to only emit general events and excludes individual MAL instruction execution events from the profiler streams. The MAL instruction profiler.openstream with zero arguments is deprecated. - Extended system view sys.users with columns: schema_path, max_memory, max_workers, optimizer and default_role. Extended system table sys.db_user_info with columns: schema_path, max_memory, max_workers, optimizer, default_role and password. The password is encrypted. This table can only be queried when the user has been granted the select privilege from monetdb. - CREATE USER statement has been extended to take more optional arguments like MAX_MEMORY, MAX_WORKERS, OPTIMIZER, DEFAULT ROLE. ALTER USER statement has also been extended with DEFAULT ROLE. - A new function sys.url_extract_host(string, bool) which returns the host name from the given URL has been implemented. The bool argument, if true, causes the www. prefix to be removed. - The user authentication tables are now administered by the SQL layer instead of the MAL layer. This means that changing (adding, dropping, altering) user and role information is now transaction-safe. - Added the UNLOGGED TABLE feature. - Removed functions sys.index and sys.strings. ### Merovingian - Added "loglevel" property for monetdbd logging (to merovingian.log). The loglevel can be set to: error or warning or information or debug. The loglevel property can be changed dynamically via command: monetdbd set loglevel=warning /path/to/dbfarm Default the loglevel is set to: information When loglevel is error, only errors are logged. When loglevel is warning, errors and warnings are logged. When loglevel is information, errors and warnings and information messages are logged. When loglevel is debug, all messages are logged. - There is now a loadmodules property that can be used to add --loadmodule arguments to the mserver5 command line. See the monetdb and mserver5 manual pages. - In certain cases (when an mserver5 process exits right after producing a message) the log message was logged over and over again, causing monetdbd to use 100% CPU. This has been fixed. ### Bug Fixes - [6982](https://github.com/MonetDB/MonetDB/issues/6982): Wrong constraint name in error message of constraint violation - [7209](https://github.com/MonetDB/MonetDB/issues/7209): Configuration option for merovingian.log - [7225](https://github.com/MonetDB/MonetDB/issues/7225): Invalid memory access when extending a BAT during appends - [7227](https://github.com/MonetDB/MonetDB/issues/7227): Date calculations, bug or feature - [7273](https://github.com/MonetDB/MonetDB/issues/7273): Concurrent reads and writes causes "BATproject2: does not match always" error - [7282](https://github.com/MonetDB/MonetDB/issues/7282): call sys.dump_table_data(); fails - [7285](https://github.com/MonetDB/MonetDB/issues/7285): C-UDFs: aggr_group.count has wrong value (number of input rows instead of number of groups). - [7296](https://github.com/MonetDB/MonetDB/issues/7296): Implictly cast a timestamp string to DATE when appropriate - [7297](https://github.com/MonetDB/MonetDB/issues/7297): Parsing partial dates behaves unpredictable - [7306](https://github.com/MonetDB/MonetDB/issues/7306): ODBC Driver Assertion `stmt->Dbc->FirstStmt' Failed - [7314](https://github.com/MonetDB/MonetDB/issues/7314): ODBC Driver : please mask/hide password - [7318](https://github.com/MonetDB/MonetDB/issues/7318): distinct in a subquery not working properly

1 0

insert pandas df to monetdb
by heikal79＠live.com 06 Sep '22

06 Sep '22

Hi is it possible to insert pandas df to montedb?. I have used both sqlalchemy and df.to_sql( name='invoiceline', con=engine, schema='tst', if_exists='append',index=False, index_label=None) and it returns text() got an unexpected keyword argument 'bindparams'. Also used pymonetdb and cursor.executemany('INSERT INTO tst.invoiceline VALUES (%s)', [[x] for x, row in df.iterrows()]) and returned 21S01!INSERT INTO: number of values doesn't match number of columns of table 'invoiceline'. My table contins 5 columns. BR

3 2

Change server configuration on Windows
by heikal79＠live.com 01 Sep '22

01 Sep '22

Hi Is it possible to change the server configuration on Windows including datafarm location, port,... Regards

2 2

Recovering database after BBPcheckbats error
by imad hajj chahine 24 Aug '22

24 Aug '22

Hi, I am using version Jan2022-SP2, and after a server restart I had the BBPcheckbats error. I upgraded to the latest version Jan2022-SP4 as it has the fix for this reported bug, and I am still unable to start the database. After manually editing the BBP.dir file and removing the entries on which the database failed to load with messages similar to (BBPcheckbats file /bat/01/67/16774.tail1 too small (expected 143640, actual 0), I managed to make the database start and connect to it and query the tables successfully. Is this the right way to recover from this? what I am risking losing by deleting these entries, should I have corrupted tables that are affected by these entries? What is exactly the role of BBP.dir? any documentation describing the use of this file. Appreciate your help. Thank you.

2 3

Copy into from stdin hanging after upgrade to 11.43.21
by imad hajj chahine 23 Aug '22

23 Aug '22

Hi, I have a bunch of dump sql files generated using msqldump command, that I was loading without any error to the version 11.43.15. After upgrading to version 11.43.21 to avoid the BBPCheckbats bug, loading the dump started to hang the server. Downgrading back to 11.43.15 fixed the issue. the command I am using to load the dump: mclient -d dbname -f sql < "pathtofile/dump.sql" sample dump file: COPY 1 RECORDS INTO tablename FROM stdin USING DELIMITERS '\t','\n','"'; Thank you.

1 0

Announcement: New Jan2022-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 22 Aug '22

22 Aug '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jan2022-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/jan2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - Implemented dump of global grants, that is to say, grants for COPY INTO and COPY FROM which grant permission to users to write to or read from files on the server (COPY INTO queries without the ON CLIENT option). - Fixed a bug where when the semicolon at the end of a COPY INTO query that reads from STDIN is at exactly a 10240 byte boundary in a file, the data isn't read as input for the COPY INTO but instead as a new SQL query. ### MonetDB Common - A bug was fixed when upgrading a database from the Oct2020 releases (11.39.X) or older when the write-ahead log (WAL) was not empty and contained instructions to create new tables. - When destroying a bat, make sure there are no files left over in the BACKUP directory since they can cause problems when the bat id gets reused. - Fixed an off-by-one error in the logger which caused older log files to stick around longer in the write-ahead log than necessary. - When an empty BAT is committed, skip writing (and synchronizing to disk) the heap (tail and theap) files and write 0 for their sizes to the BBP.dir file. When reading the BBP.dir file, if an empty BAT is encountered, set the sizes of those files to 0. This fixes potential issues during startup of the server (BBPcheckbats reporting errors). - Make sure heap files of transient bats get deleted when the bat is destroyed. If the bat was a partial view (sharing the vheap but not the tail), the tail file wasn't deleted. - Various changes were made to satisfy newer compilers. - The batDirtydesc and batDirtyflushed Boolean values have been deprecated and are no longer used. They were both holdovers from long ago. - Various race conditions (data races) have been fixed. - All accesses to the BACKUP directory need to be protected by the same lock. The lock already existed (GDKtmLock), but wasn't used consistently. This is now fixed. Hopefully this makes the hot snapshot code more reliable. ### MonetDB5 Server - Various race conditions (data races) have been fixed. ### Merovingian - When multiple identical messages are written to the log, write the first one, and combine subsequent ones in a single message. - Fixed a leak where the log file wasn't closed when it was reopened after a log rotation (SIGHUP signal). - Try to deal more gracefully with "inherited" mserver5 processes. This includes not complaining about an "impossible state", and allowing such processes to be stopped by the monetdbd process. - When a transient failure occurs during processing of a new connection to the monetdbd server, sleep for half a second so that if the transient failure occurs again, the log file doesn't get swamped with error messages. ### Bug Fixes - [7040](https://github.com/MonetDB/MonetDB/issues/7040): Memory leak detected for MAPI interface - [7298](https://github.com/MonetDB/MonetDB/issues/7298): Irresponsive database server after reading incomplete SQL script. - [7308](https://github.com/MonetDB/MonetDB/issues/7308): Race condition in MVCC transaction management

1 0

Re: Define a cast function from day_interval to int
by yunus 08 Aug '22

08 Aug '22

Hello Imad, Without going into too much detail, you might want to start having a look at the MonetDB-extend <https://www.monetdb.org/hg/MonetDB-extend> repository. Also keep in mind, that the use of MAL scripts for declaring C functions in MAL is deprecated in favor of the command and pattern preprocessor macro's. Regards, Yunus

1 0